Skip to content

Streaming Responses with Starlette and FastAPI

A handler that builds its whole response in memory has a memory ceiling equal to the largest response times the number of concurrent requests. For a JSON document that is fine; for a CSV export, a file download, a log tail or a model's token stream it is the reason a service dies under a load test that looked harmless. StreamingResponse takes an async generator instead of a body, and the framework writes each chunk as it is produced. Measured on the same 13 MB response, the buffered version peaked at 14.6 MiB and the streamed version at 0.9 MiB — sixteen times less, and the memory now scales with the chunk size rather than the payload.

Prerequisites

Peak memory for the same 13 MB response 2 bars comparing buffered Response with the others. Peak memory for the same 13 MB response buffered Response 14.6 MiB StreamingResponse 0.9 MiB tracemalloc peak on the receiving side; the server pays the same difference. Streaming keeps memory proportional to the chunk size, not to the response size.

1. Return a generator instead of a body

The handler returns as soon as the response object is constructed; the generator runs afterwards, as the server writes:

from starlette.responses import StreamingResponse


async def export(request):
    async def rows():
        async with request.state.pool.acquire() as conn, conn.transaction():
            yield b"id,email,created_at\n"
            async for record in conn.cursor("SELECT id, email, created_at FROM accounts"):
                yield f"{record['id']},{record['email']},{record['created_at']}\n".encode()

    return StreamingResponse(rows(), media_type="text/csv")

Two properties make this work. The generator yields bytes — a str is encoded per chunk, which is wasted work on a hot path. And the data source is itself streamed: pairing StreamingResponse with a server-side cursor keeps both halves bounded, while await conn.fetch(...) inside the generator would reintroduce the memory it was meant to avoid.

Chunk size is worth a thought. Yielding per row makes one ASGI message per row; for a million-row export that overhead is measurable. Accumulate into a buffer and yield every 64 KB or so.

Verify: peak memory during the response is a small multiple of the chunk size, whatever the total.

2. Know what changes in the response headers

Streaming changes how the response is framed:

/buffered  content-length=13107200  transfer-encoding=None
/streamed  content-length=None      transfer-encoding=chunked

Without a known length, the server uses chunked transfer encoding. Clients can still show progress if you set the length yourself — StreamingResponse(gen(), headers={"content-length": str(size)}) — which is worth doing for file downloads where the size is known in advance.

The bigger consequence is that the status code is committed as soon as the first chunk is sent. An error raised in the middle of a generator cannot become a 500: the client has already received 200 OK and some bytes, and the connection simply ends early. So validation, authorisation and anything else that can fail must happen before the first yield:

async def export(request):
    if not await authorised(request):
        return JSONResponse({"error": "forbidden"}, status_code=403)   # still changeable

    async def rows():
        yield header_line                                              # from here, 200 is fixed
        ...

For formats that can express it, a trailer record — a final line saying # complete or a JSON envelope with a "status" field — lets a client distinguish a truncated stream from a finished one.

Verify: the streamed route's response has no Content-Length and the body arrives incrementally.

What the response headers tell the client A grid of 4 rows by 2 columns. What the response headers tell the client response framing client can Response(body) Content-Length show a progress bar StreamingResponse Transfer-Encoding: chunked start rendering immediately StreamingResponse + length Content-Length you set both, if you know the size SSE or long poll chunked, never ending consume events as they arrive Measured: the buffered route sent content-length 13107200, the streamed route chunked.

3. Clean up when the client disappears

A client that closes the connection mid-stream causes a CancelledError at the generator's current yield. Verified with a client that read three chunks and left: the generator was cancelled and its finally ran. That is the hook for releasing whatever the stream holds:

    async def rows():
        conn = await pool.acquire()
        try:
            async for record in conn.cursor(QUERY):
                yield encode(record)
        except asyncio.CancelledError:
            log.info("export abandoned by client")
            raise                                      # always re-raise
        finally:
            await pool.release(conn)                   # runs on disconnect too

Without that finally, every abandoned download leaks a database connection — and abandoned downloads are common, because users close tabs. Note the raise: swallowing the cancellation here breaks shutdown, as described in cancellation patterns.

For cleanup that should run after the response finishes normally, Starlette's BackgroundTask is the cleaner hook, and it runs after the last chunk is written:

return StreamingResponse(rows(), background=BackgroundTask(remove_temp_file, path))

Verified: the background task ran exactly once, after the stream completed.

Verify: disconnect a client mid-download and confirm the connection count returns to baseline.

The life of a streaming response 5 stages from handler returns to finally runs. The life of a streaming response handler returns headers sent at once generator yields one chunk at a time written out drain when full disconnect cancels raised at the yield finally runs then background task Verified: a client that left after 3 chunks cancelled the generator and ran its finally.

4. Understand the back-pressure you actually get

The generator does not run ahead without limit — the server awaits the socket write when buffers fill — but the slack is larger than people expect. With a deliberately slow reader, the client had read 30 chunks while the server had already produced 88:

client read 30 chunks in 0.32s; server had produced 88 chunks by then

That is roughly 3.8 MB of buffering across the server's write buffer, the kernel socket buffers and the client's receive buffer. For a CSV export it is irrelevant; for a stream where each chunk is expensive to produce, or where producing ahead has side effects, it means "the client is 58 chunks behind" is a state your code should tolerate.

Where the gap matters, add your own limit — produce into a bounded asyncio.Queue and have the generator drain it, so production stops when the queue fills. That is the same bounded hand-off used in SSE, and it makes the back-pressure explicit rather than dependent on buffer sizes you do not control.

Verify: with a slow client, the producer's progress stops advancing rather than growing without bound.

5. Never block the loop inside the generator

The generator runs on the event loop between writes, so a synchronous call inside it blocks every other request in the worker:

    async def rows():
        for path in paths:
            data = await asyncio.to_thread(path.read_bytes)   # not path.read_bytes()
            yield data

This is the most common streaming bug in practice: a file export that reads with plain open() works perfectly in development with one user and collapses the worker's latency under concurrency. The same applies to CPU work — compressing, rendering or serialising a large chunk belongs in a thread, as covered in CPU-bound task offloading.

For plain file downloads, skip the generator entirely: FileResponse streams the file and can use sendfile where available, which moves the bytes without them passing through Python at all.

Verify: loop lag stays flat while a large export is running under concurrent traffic.

Should this response stream? A decision on How big, and how soon with 3 outcomes. Should this response stream? How big, and how soon? large or unbounded stream it exports, files, feeds small and complete buffer it JSON under a megabyte slow to produce, small stream for time to first byte progress, token streams Streaming trades the ability to change your mind: once the first chunk is sent, the status code is fixed.

Verification

A streaming endpoint is correct when:

  • Memory is flat: the peak tracks chunk size, not response size.
  • The data source streams too, rather than being fetched whole inside the generator.
  • Failures happen before the first yield, so status codes remain changeable.
  • Disconnects release resources, proven by connection counts returning to baseline.
  • Nothing in the generator blocks the loop.
  • Completion is distinguishable from truncation by the client.

Pitfalls & edge cases

  • Yielding str. Each chunk is encoded; yield bytes on hot paths.
  • Buffering inside the generator. await conn.fetch() or read() on the whole file defeats the purpose entirely.
  • Raising after the first chunk. The client has a 200 and a truncated body; there is no way to signal an error in the status.
  • Compression middleware. Gzip middleware may buffer the whole response, undoing the memory saving — exclude streaming routes.
  • Proxy buffering. nginx buffers responses by default; X-Accel-Buffering: no or proxy_buffering off is needed for real-time streams.
  • Timeouts on long downloads. Proxy and load-balancer read timeouts abort long streams; either raise them or send data often enough to keep the connection active.

Frequently Asked Questions

How do I stream a large response in FastAPI or Starlette?

Return StreamingResponse(async_generator(), media_type=...). The handler returns immediately and the generator is consumed as the server writes. Measured on a 13 MB body, streaming peaked at 0.9 MiB against 14.6 MiB for the buffered equivalent.

How do I know when a client disconnects from a streaming response?

The generator is cancelled at its current yield, so a CancelledError is raised there. Put resource release in a finally block and re-raise the cancellation. Verified: a client that read three chunks and left cancelled the generator and ran its cleanup.

Can I return an error status from inside a streaming generator?

No. The status code and headers are sent before the first chunk, so once streaming starts the response is committed to 200. Do authorisation and validation before the first yield, and give the format a way to mark completion so clients can detect truncation.

Does StreamingResponse apply back-pressure to the generator?

Partially. The server awaits socket writes when buffers fill, but several megabytes can be in flight: measured, the server had produced 88 chunks while the client had read 30. If your chunks are expensive, add an explicit bounded queue between the producer and the generator.

Should I use StreamingResponse for file downloads?

Use FileResponse, which streams the file and can use sendfile so the bytes never pass through Python. Reserve StreamingResponse for content you generate — exports, reports, event streams — and keep any file reads inside it in a thread.