Slow upload with big files (multipart)

[jkermes]

Hello! Multipart file uploads are slower than expected using httpx, especially on big files.

OS platform: x86_64 GNU/Linux
Python version: 3.9.7
Dependencies:

fastapi==0.70.0
uvicorn==0.15.0
requests==2.26.0
httpx==0.20.0

Generate a test file
dd if=/dev/zero of=bigfile bs=1000M count=1

Server code (api.py)

from fastapi import FastAPI, File, UploadFile

app = FastAPI()


@app.post("/upload/")
async def create_upload_file(file: UploadFile = File(...)):
    return {"filename": file.filename}

Run the server with uvicorn api:app

Test code

import time

import httpx
import requests

url = "http://127.0.0.1:8000/uploadfile/"

with open('bigfile', 'rb') as file:
    # using requests
    t0 = time.time()
    response = requests.post(url=url, files={'file': file})
    elapsed_time = round(time.time() - t0, 2)
    print(f"Upload duration using requests: {elapsed_time} s")

    file.seek(0)

    # using httpx
    t0 = time.time()
    response = httpx.post(url=url, files={'file': file})
    elapsed_time = round(time.time() - t0, 2)
    print(f"Upload duration using httpx: {elapsed_time} s")

Results:

Upload duration using requests: 1.27 s
Upload duration using httpx: 15.9 s

[euri10]

I faced the same in a cli I wrote to upload files to a fastapi backend and switched to requests for that purpose,

didnt take time to see the root of this but I can confirm the issue @tomchristie

[lovelydinosaur]

Thanks folks. First pass on resolving this is #1948. I'm going to do a bit more digging tho and see if there's further scope for improvement.

[euri10]

can confirm this accelerated stuff big time, thanks @tomchristie