Skip to content

Uploading Multipart Files with httpx and aiohttp

Uploading a file as multipart/form-data — the format of an HTML form with a file input — is a few lines in either major async client. The detail that matters is what you hand the client: an open file handle lets it stream the body in chunks, while bytes force the whole file into memory first. Measured with a 100 MB file posted to a local Starlette endpoint (client and server in one process): httpx with a file handle peaked at 59 MB of resident memory and aiohttp at 52 MB; httpx given the file's bytes peaked at 160 MB. All three uploads took about half a second. This guide uploads files with both clients, adds fields and multiple files, streams large and generated content, and handles retries.

Prerequisites

1. Upload a file with httpx

Pass files= with a tuple of filename, file object and content type; form fields go in data=:

import httpx


async def upload(client: httpx.AsyncClient, path: str) -> dict:
    with open(path, "rb") as fh:
        r = await client.post(
            "https://api.example.com/upload",
            files={"file": ("report.pdf", fh, "application/pdf")},
            data={"note": "quarterly"},
        )
    r.raise_for_status()
    return r.json()

httpx builds the multipart body lazily and reads the file in chunks while sending, so memory stays near one chunk regardless of file size — measured at 59 MB peak for a 100 MB upload, including the in-process server that received it. The file object here is an ordinary synchronous file: httpx reads it in small blocking reads on the event loop thread, which is fine for local disk but not for slow network filesystems. Set the content type explicitly; the server may route or validate on it.

Verify: upload a file several times larger than you would want in memory and watch the process's resident memory stay flat.

Peak process memory uploading a 100 MB file 3 horizontal bars comparing aiohttp, file handle with the others. Peak process memory uploading a 100 MB file aiohttp, file handle 52 MB httpx, file handle 59 MB httpx, bytes 160 MB Peak RSS of one process running both client and Starlette server; upload took 0.49-0.67 s. Hand the client a file, not its contents.

2. Upload a file with aiohttp

aiohttp builds multipart bodies with FormData. Add fields and files explicitly:

import aiohttp


async def upload(session: aiohttp.ClientSession, path: str) -> dict:
    form = aiohttp.FormData()
    form.add_field("note", "quarterly")
    with open(path, "rb") as fh:
        form.add_field("file", fh, filename="report.pdf", content_type="application/pdf")
        async with session.post("https://api.example.com/upload", data=form) as r:
            r.raise_for_status()
            return await r.json()

aiohttp reads file objects in chunks in a thread pool, so the event loop is not blocked by disk reads; it peaked at 52 MB in the same test. Keep the file open until the request completes — the with block must enclose the post, because the body is read while sending, not when add_field is called. Passing data={"file": fh} also works and creates the form implicitly, but FormData makes filenames and content types explicit.

Verify: the server receives the field and the file with the right filename and content type.

3. Send several files and repeated fields

Multipart allows several parts with the same name, which servers read as a list. httpx takes a list of tuples instead of a dict for that; aiohttp just adds the field twice:

# httpx: a list allows repeated names
files = [
    ("attachments", ("a.png", open("a.png", "rb"), "image/png")),
    ("attachments", ("b.png", open("b.png", "rb"), "image/png")),
]
try:
    r = await client.post(url, files=files, data={"ticket": "4711"})
finally:
    for _, (_, fh, _) in files:
        fh.close()


# aiohttp: add_field repeatedly
form = aiohttp.FormData()
for name in ("a.png", "b.png"):
    form.add_field("attachments", open(name, "rb"), filename=name, content_type="image/png")

Close every file you open, including on errors; a contextlib.ExitStack makes that tidy when the number of files varies. On the server, Starlette's form.getlist("attachments") returns both parts.

Verify: the server receives every file under the repeated name, and no file descriptors remain open after the request (ls /proc/<pid>/fd).

The same upload in httpx and aiohttp A grid of 4 rows by 3 columns. The same upload in httpx and aiohttp task httpx aiohttp file part files={"f": (name, fh, type)} form.add_field("f", fh, filename=, content_type=) text field data={"k": "v"} form.add_field("k", "v") repeated name list of (name, tuple) add_field twice file reads sync reads on the loop in a thread Both stream file handles; they differ in API shape and where reads happen.

4. Stream generated content without a file

When the content is produced on the fly — a CSV generated from a query, an archive built in memory — write it to a temporary file and upload that, or send it as a raw streaming body if the API accepts one. Multipart parts in httpx need a sized, readable object, so the temporary file is the dependable route:

import tempfile


async def upload_export(client: httpx.AsyncClient, rows) -> None:
    with tempfile.TemporaryFile() as tmp:
        async for row in rows:
            tmp.write(row.to_csv_line().encode())
        tmp.seek(0)
        r = await client.post(url, files={"file": ("export.csv", tmp, "text/csv")})
        r.raise_for_status()


# APIs that accept a raw body can take an async generator directly
async def body():
    async for row in rows:
        yield row.to_csv_line().encode()

await client.put(url, content=body(), headers={"content-type": "text/csv"})

The raw-body form uses chunked transfer encoding and never holds more than one row. Object stores use raw bodies, not multipart, for exactly this reason; for very large objects they use multipart uploads — a different mechanism — as in uploading large files to S3 with async multipart.

Verify: exporting a large generated file keeps memory flat, and the server receives the complete content.

5. Make uploads retryable and time-bounded

An upload is a long request, and a retry must start the body from the beginning. A file handle that has been read to the end will send nothing the second time:

async def upload_with_retry(client: httpx.AsyncClient, path: str, attempts: int = 3) -> dict:
    for attempt in range(attempts):
        try:
            with open(path, "rb") as fh:                         # fresh handle per attempt
                r = await client.post(
                    url,
                    files={"file": (os.path.basename(path), fh, "application/octet-stream")},
                    timeout=httpx.Timeout(10.0, write=120.0),    # long write for big files
                )
            r.raise_for_status()
            return r.json()
        except (httpx.TransportError, httpx.HTTPStatusError) as exc:
            if isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code < 500:
                raise                                             # 4xx: retrying will not help
            await asyncio.sleep(2 ** attempt)
    raise RuntimeError("upload failed after retries")

Open the file inside the loop so each attempt sends the full content. Raise the write timeout for large files — httpx's default of 5 s applies to each write operation, and a slow uplink can exceed it on large chunks. And make the receiving endpoint idempotent, for example by an upload id, so a retry after a lost response does not create a duplicate.

Verify: inject a connection reset mid-upload; the retry sends the whole file and the server stores exactly one copy.

How should this content be uploaded? A decision on Where does the content come from with 3 outcomes. How should this content be uploaded? Where does the content come from? a file on disk multipart with file handle flat memory generated on the fly temp file, or raw streaming body never a big bytes huge object for a store store multipart upload API parts in parallel The client streams whatever you give it; give it something streamable.

Verification

Uploads are implemented well when:

  • Files are passed as handles, so memory does not grow with file size.
  • Field names, filenames and content types arrive as the server expects.
  • Every opened file is closed, including on errors.
  • Retries reopen the file, and timeouts fit the upload size.

Diagnostic Hook: log upload size, duration and peak memory per upload job. Memory that tracks file size means bytes are being passed somewhere instead of a handle; uploads that fail at a consistent size or duration point at a write timeout or a server body limit.

Pitfalls & edge cases

  • Reading the file into bytes first. Measured at 160 MB peak for a 100 MB upload.
  • Closing the file before the request is sent. The body is read during sending.
  • Retrying with an exhausted file handle. Reopen it for each attempt.
  • Default write timeouts on slow links. Raise them for large uploads.

Frequently Asked Questions

How do I upload a file with httpx AsyncClient?

Pass files={"field": (filename, open_file, content_type)} to client.post, with form fields in data=. Passing an open file handle lets httpx stream it instead of loading it into memory.

How do I upload a file with aiohttp?

Build an aiohttp.FormData, add the file with add_field(name, file_handle, filename=..., content_type=...), and pass it as data= to session.post, keeping the file open until the request completes.

Does httpx load the whole file into memory for multipart uploads?

Not when given a file handle: in testing, a 100 MB upload peaked at 59 MB for the whole process. Passing the file's bytes instead peaked at 160 MB.

How do I send several files under the same field name?

With httpx, pass files as a list of (name, (filename, handle, type)) tuples; with aiohttp, call add_field with the same name once per file.