Skip to content

ASGI Servers & Frameworks for Async Python

ASGI is a small specification — an application is a coroutine taking a scope, a receive callable and a send callable — and almost everything that matters in production is a consequence of one thing that specification implies: one event loop serves every request in a worker process. A handler that blocks for 50 ms does not add 50 ms to its own latency; it adds 50 ms to every request in flight. A pool opened at import time binds to a loop that may no longer exist. A response built in memory multiplies by concurrency. And a background task created without an owner is discarded by the next deploy.

The frameworks — Starlette, FastAPI, Litestar, Quart — differ in routing, validation and dependency injection, and agree completely on the parts this section is about: how resources are created and destroyed, how responses are streamed, how many processes to run, and where work that outlives a response belongs. Measurements throughout come from Starlette 1.6, FastAPI 0.141 and uvicorn 0.53 on a 24-core Linux machine. The parent section, Network I/O & Protocol Handling, covers the client side and the transports these servers sit on.

Scope of this section:

  • The lifespan protocol: creating and destroying resources at the right time.
  • Streaming responses, and the memory difference they make.
  • Sizing worker processes from a measurement rather than a formula.
  • Background work: what survives a shutdown, and what quietly does not.
  • The blocking calls, middleware and dependencies that cost an entire worker.

Architectural principles

  • The loop is the shared resource. Every middleware, dependency and handler in a worker runs on one loop. The budget for synchronous work in any of them is microseconds, and anything longer belongs in a thread or another process.
  • Resources belong to the lifespan. Connection pools, HTTP clients and background workers need a running loop, so they are created in the lifespan's startup and closed in its teardown — never at import, never per request.
  • Responses that can be large must stream. Buffering is a memory ceiling of response size times concurrency. Measured on a 13 MB body, streaming peaked at 0.9 MiB against 14.6 MiB buffered.
  • Processes are for CPU, not for concurrency. A single loop already handles thousands of concurrent awaits. Extra workers pay off when handlers compute — 58 to 364 rps from one worker to eight — and not when they wait.
  • Work that outlives a response needs an owner. BackgroundTasks is owned by the framework and shutdown waits for it; a bare create_task is owned by nothing and was abandoned at 0.29 s of a one-second job in testing.
What sits between a socket and your handler 5 stacked layers from process supervisor to handler. What sits between a socket and your handler process supervisor the orchestrator or --workers one loop per process server (uvicorn) HTTP parsing, keep-alive graceful shutdown timing middleware runs per request, on the loop a blocking one blocks everything application routing, dependencies lifespan state lives here handler your code, one coroutine must never block Every layer shares one event loop per worker: the slowest thing in any of them is the latency of all of them.

Execution model: one loop, three scopes

An ASGI server creates three kinds of scope. The lifespan scope exists once per worker process and brackets everything: its startup runs before the first request and its shutdown after the last. The http scope exists per request. The websocket scope exists per connection and lives as long as the client stays.

The lifespan scope is where most production correctness lives, because it is the only place with both a running loop and a guaranteed teardown. A pool created there is created once and closed once; a consumer task started there is cancelled when the process stops. The other two scopes are transient, and anything that outlives them — a background job, a queued message — must not hold their state, because the request object and its connection are gone the moment the response is written.

The ordering guarantees are worth knowing exactly, because so much shutdown code is written against a guess. On a shutdown signal the server stops accepting connections, in-flight requests run to completion, and only then does the lifespan teardown run. Measured with a one-second handler and a shutdown triggered 200 ms in: the request returned 200 at 0.81 s, and the teardown began afterwards. So resources handlers use are still open while those handlers finish — which is why closing a pool from a signal handler, rather than from the lifespan, breaks requests that were nearly done.

One request through the stack 5 stages from server parses to response written. One request through the stack server parses HTTP on the loop middleware auth, tracing, limits router path to handler handler awaits pools from the lifespan response written buffered or streamed A blocking call at any step stops every other request in that worker.

Pattern catalogue

Create resources in the lifespan, yield them as state

@contextlib.asynccontextmanager
async def lifespan(app):
    pool = await asyncpg.create_pool(DSN)
    workers = [asyncio.create_task(job_worker(i)) for i in range(4)]
    try:
        yield {"pool": pool}                            # reaches request.state
    finally:
        for task in workers:
            task.cancel()
        await asyncio.gather(*workers, return_exceptions=True)
        await pool.close()

Handlers then use request.state.pool, which is the identical object — verified with an identity check. See managing ASGI lifespan startup and shutdown.

Stream anything that can be large

async def export(request):
    async def rows():
        yield b"id,value\n"
        async for record in cursor:                     # a streamed source, too
            yield encode(record)
    return StreamingResponse(rows(), media_type="text/csv")

Headers go out before the first chunk, so validation and authorisation must happen before the first yield. See streaming responses with Starlette.

Accept work with a bound, or reject it

try:
    request.state.jobs.put_nowait(job)
except asyncio.QueueFull:
    return JSONResponse({"error": "busy"}, status_code=503, headers={"retry-after": "1"})
return JSONResponse({"accepted": True}, status_code=202)

A bounded queue turns an overload into a fast 503 instead of unbounded memory growth — the front-door form of load shedding.

Push blocking work off the loop

result = await asyncio.to_thread(legacy_client.fetch, key)   # never legacy_client.fetch(key)

The diagnostic for having missed one: a worker serving roughly 1 / handler_latency requests per second means requests are being serialised.

Report readiness from loop lag

ok = state.ready and state.loop_lag < LAG_BUDGET
return JSONResponse({"ready": ok, "loop_lag_ms": round(state.loop_lag * 1000)},
                    status_code=200 if ok else 503)

A handler blocking for 0.6 s produced 587 ms of measured lag and a 503, recovering on its own. See health and readiness probes.

Keep middleware off the critical path

class TimingMiddleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        if scope["type"] != "http":                            # lifespan and websocket pass through
            return await self.app(scope, receive, send)
        started = time.perf_counter()
        await self.app(scope, receive, send)
        LATENCY.observe(time.perf_counter() - started)         # no I/O, no logging call per request

Middleware runs for every request in the worker, so anything it does is multiplied by your request rate. The common mistakes are all I/O: looking up a session in the database, writing an access log synchronously, or calling an authorisation service without a cache. Each of them turns a middleware into a per-request round trip that every endpoint pays, including the health probe.

The three ASGI scopes and what they carry A grid of 4 rows by 2 columns. The three ASGI scopes and what they carry scope created holds lifespan once per worker process pools, clients, background tasks http per request headers, path, per-request state websocket per connection a long-lived message loop (none) background work after a response nothing: copy what you need Request state dies with the response, which is why background work must not hold it.

Resource boundaries

Resource What consumes it How to size and bound it
Event loop time Every handler, middleware and dependency Nothing synchronous over ~1 ms; measure loop lag continuously
Worker processes --workers, or replicas From a measurement: CPU-bound scaled 58 → 364 rps at 1 → 8 workers
Memory per worker A full interpreter plus your caches ~34 MiB baseline per worker here, plus the app's own footprint
Database connections Pool size per worker Divide the intended total by the worker count
Executor threads to_thread, sync dependencies, sync background tasks Default min(32, cpus + 4); give heavy work its own executor
Response memory Buffered bodies times concurrency Stream anything unbounded; measured 14.6 MiB → 0.9 MiB
In-flight background work One task per request, unbounded by default A bounded queue with worker tasks, plus a 503 when full
Open connections keep-alive, WebSockets, streams --timeout-keep-alive, --limit-concurrency, file-descriptor limits

The connection row causes the most production incidents: max_size=20 with eight workers is 160 connections, and the database's limit is reached during the deploy that added the workers. The executor row is the second: asyncio.to_thread, synchronous FastAPI dependencies and synchronous BackgroundTasks all draw on the same default pool, so a burst of one can delay the others in ways that look like a network problem. Where blocking work is a real part of the workload, give it a dedicated ThreadPoolExecutor sized separately, as in worker pool sizing for mixed workloads.

Integrated production example

One service combining all of it: lifespan-owned pool and worker tasks, a bounded job queue with a 503 when full, a streamed export, a readiness probe driven by loop lag, and a shutdown that drains the queue before cancelling anything.

import asyncio
import contextlib
import time

from starlette.applications import Starlette
from starlette.responses import JSONResponse, StreamingResponse
from starlette.routing import Route

LAG_BUDGET, QUEUE_MAX, WORKERS = 0.25, 1000, 4
jobs: asyncio.Queue = asyncio.Queue(maxsize=QUEUE_MAX)
stats = {"done": 0, "shed": 0, "failed": 0}


class Health:
    def __init__(self) -> None:
        self.ready = False
        self.lag = 0.0

    async def sample(self, interval: float = 0.05) -> None:
        while True:
            start = time.perf_counter()
            await asyncio.sleep(interval)
            overshoot = max(0.0, time.perf_counter() - start - interval)
            self.lag = 0.8 * self.lag + 0.2 * overshoot        # smoothed: no flapping


health = Health()


async def job_worker(name: str) -> None:
    while True:
        job = await jobs.get()
        try:
            await handle(job)
            stats["done"] += 1
        except asyncio.CancelledError:
            jobs.task_done()
            raise                                              # always re-raise
        except Exception:
            stats["failed"] += 1                               # one bad job never kills a worker
        finally:
            jobs.task_done()


@contextlib.asynccontextmanager
async def lifespan(app):
    tasks = [asyncio.create_task(health.sample())]
    tasks += [asyncio.create_task(job_worker(f"w{i}")) for i in range(WORKERS)]
    health.ready = True
    try:
        yield {"jobs": jobs}
    finally:
        health.ready = False                                   # stop new traffic first
        await jobs.join()                                      # then drain what was accepted
        for task in tasks:
            task.cancel()
        await asyncio.gather(*tasks, return_exceptions=True)


async def enqueue(request):
    try:
        request.state.jobs.put_nowait({"at": time.time()})
    except asyncio.QueueFull:
        stats["shed"] += 1
        return JSONResponse({"error": "busy"}, status_code=503,
                            headers={"retry-after": "1"})
    return JSONResponse({"accepted": True}, status_code=202)


async def export(request):
    async def rows():
        yield b"id,value\n"
        buffer = bytearray()
        for i in range(50_000):
            buffer += f"{i},{i * i}\n".encode()
            if len(buffer) >= 65536:                           # ~64 KB chunks, not per row
                yield bytes(buffer)
                buffer.clear()
        if buffer:
            yield bytes(buffer)
    return StreamingResponse(rows(), media_type="text/csv")


async def readyz(request):
    ok = health.ready and health.lag < LAG_BUDGET
    return JSONResponse(
        {"ready": ok, "loop_lag_ms": round(health.lag * 1000, 1),
         "queue": jobs.qsize(), **stats},
        status_code=200 if ok else 503)


app = Starlette(
    routes=[Route("/jobs", enqueue, methods=["POST"]),
            Route("/export", export),
            Route("/readyz", readyz)],
    lifespan=lifespan)

Run against it, 2,000 concurrent enqueue requests were all accepted with 202 in 2.17 s, the export streamed 0.79 MB with transfer-encoding: chunked in 0.01 s, the readiness probe reported loop_lag_ms: 0.5 with an empty queue once the workers caught up, and the shutdown drained the queue in 0.18 s before cancelling the workers — {'done': 2000, 'shed': 0, 'failed': 0}, with nothing lost.

Diagnostic Hook — is the worker healthy or merely alive?

Four numbers, all per worker process. Loop lag, sampled continuously: it is the single best indicator of an ASGI service in trouble, and it distinguishes "our dependency is slow" (lag flat, latency high) from "our handlers are blocking" (lag rises with load). Requests in flight divided by workers: if it hovers near one, something is serialising requests and more workers will only multiply the inefficiency. Queue depth and shed count for any in-process background work: depth that never returns to zero means the workers are undersized. Executor queue wait: to_thread calls share one pool with sync dependencies and sync background tasks, and a rising wait there means blocking work is crowding out everything else. Alert on loop lag above 100 ms for a minute, and on background queue depth that stays above zero for five.

The service is too slow: what actually helps? A decision on What does the measurement say with 3 outcomes. The service is too slow: what actually helps? What does the measurement say? loop lag rises with load more workers or replicas handlers are CPU-bound lag flat, latency high fix the dependency pool size, query, upstream one request per worker find the blocking call workers would only hide it Adding workers to an I/O-bound service measured 808 rps at one worker and 670 at four.

Failure modes

Failure mode Root cause Detection Fix
Every request slow under light load A blocking call in a handler, dependency or middleware Loop lag tracks the call's duration asyncio.to_thread, or an async client
Pool errors on startup Pool created at import, before a loop exists Fails at import or binds to a dead loop Create it in the lifespan
Database refuses connections after scaling Pool size is per worker Connections ≈ max_size × workers Divide the total by the worker count
Memory spikes on large responses Buffered bodies times concurrency RSS tracks response size StreamingResponse with chunked output
Background work disappears on deploy create_task with no owner Work missing after restarts BackgroundTasks, tracked tasks, or a durable queue
Silent background failures Unobserved task exceptions "Task exception was never retrieved" in stderr Log in a done callback
502s during rolling deploys Server stops accepting before the load balancer notices Errors correlate with deploys Readiness false first, then a drain delay
Throughput flat as workers increase The bottleneck is a dependency, not CPU Loop lag stays flat Fix the query, pool or upstream instead

Frequently Asked Questions

Where should I create a database pool in a FastAPI or Starlette app?

In the lifespan startup block, which runs with a loop available and pairs with a teardown. Yield it as state so handlers reach it through request.state or a dependency. Creating it at module import has no loop and produces errors that look unrelated.

How many uvicorn workers does an async service need?

It depends on the handlers. CPU-bound work scaled from 58 rps at one worker to 364 at eight in testing; an I/O-bound handler served 808 rps at one worker and 670 at four. Measure loop lag under load: if it rises, more workers help; if it stays flat, the bottleneck is elsewhere.

Why is my whole ASGI service slow when only one endpoint is?

Because every request in a worker shares one event loop. A synchronous call — a blocking driver, a large JSON parse, a template render — stops every other request for its duration. Measure loop lag, then move the offending work into a thread or process.

What happens to background tasks when an ASGI server shuts down?

BackgroundTasks are waited for; bare asyncio tasks are not. Measured on a one-second job with shutdown triggered at 100 ms, the server waited until 1.05 s in the first case and exited at 0.29 s in the second, abandoning the work. Track tasks and cancel them in the lifespan, or use a durable queue.

How do I stream a large response without buffering it in memory?

Return StreamingResponse with an async generator that yields bytes in chunks of roughly 64 KB, and keep the data source streamed too — a server-side database cursor rather than a full fetch. Measured on a 13 MB body, peak memory dropped from 14.6 MiB to 0.9 MiB.