Skip to content

Diagnosing Connection Pool Exhaustion in Async Clients

Latency has a floor that moves. Under light load the endpoint responds in 40 ms; add concurrency and it becomes 200 ms, then 900 ms, in neat multiples of the downstream's own response time. The downstream reports no change. Nothing has errored, nothing is retrying, CPU is flat. This is pool exhaustion, and the multiples are the giveaway: requests are queueing for a connection, so total latency becomes (queue position ÷ pool size) × downstream latency.

It is one of the few async performance problems that is fully explained by arithmetic, which makes it satisfying to diagnose — provided you can see the queue. This guide covers the symptoms that distinguish pool exhaustion from a slow dependency, exposing the pool-wait metric that proves it, finding leaked connections (the other cause of an empty pool), and the sizing calculation that fixes it without simply moving the queue somewhere else.

Prerequisites

  • Python 3.11+ with httpx or aiohttp, and optionally asyncpg for the database case:
pip install httpx aiohttp
Client p99 as concurrency rises, pool = 10 Four bars showing latency scaling linearly with concurrency under pool exhaustion. Client p99 as concurrency rises, pool = 10 10 concurrent 52 ms — 1.1x 20 concurrent 98 ms — 2.0x 40 concurrent 205 ms — 4.1x 80 concurrent 410 ms — 8.0x Downstream p99 held at ~50 ms throughout the sweep. Integer multiples of an unchanged downstream latency mean queueing, not slowness.

1. Recognise the signature

Pool exhaustion has a fingerprint no other failure shares.

concurrency   client p99   downstream p99   ratio
    10          52 ms         48 ms          1.1x     <- healthy
    20          98 ms         49 ms          2.0x     <- queueing starts
    40         205 ms         50 ms          4.1x     <- pool size is ~10
    80         410 ms         51 ms          8.0x     <- latency scales linearly

Client latency rising in integer multiples of an unchanged downstream latency means requests are waiting for a resource, not for work. The multiplier tells you the pool size directly: at 40 concurrent requests and a 4× multiplier, roughly 10 connections are available. Compare this with a genuinely slow dependency, where the downstream's own p99 rises too, and with a CPU-bound stall, where loop lag rises. Verify: run a concurrency sweep against a stub with fixed latency; the ratio should track concurrency ÷ pool size.

2. Expose the pool wait

Neither httpx nor aiohttp exports a pool-wait metric, so wrap the call.

import time
import httpx

POOL_WAIT = []          # replace with a histogram in production


class TimedTransport(httpx.AsyncHTTPTransport):
    async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
        start = time.monotonic()
        response = await super().handle_async_request(request)
        # `elapsed` on the response covers the exchange; the difference is queueing
        POOL_WAIT.append(time.monotonic() - start - response.elapsed.total_seconds())
        return response


client = httpx.AsyncClient(
    transport=TimedTransport(limits=httpx.Limits(max_connections=100,
                                                 max_keepalive_connections=20)),
    timeout=httpx.Timeout(connect=2.0, read=5.0, pool=0.5),
)

Setting a short pool timeout is half the diagnosis on its own: with it, exhaustion surfaces as a countable PoolTimeout instead of an unbounded wait that looks like slowness. For aiohttp, the equivalent signal is the gap between connect and sock_connect timeouts, plus connector._available_connections() sampled on an interval. Verify: under the sweep from step 1, pool wait should be near zero at low concurrency and grow linearly once the pool is saturated.

3. Rule out leaked connections

An exhausted pool with idle traffic means connections are being held, not used.

import asyncio


async def pool_report(client: httpx.AsyncClient, interval: float = 10.0) -> None:
    pool = client._transport._pool                # private, but stable in practice
    while True:
        await asyncio.sleep(interval)
        connections = list(pool.connections)
        active = sum(1 for c in connections if c.is_active())
        idle = sum(1 for c in connections if c.is_idle())
        print(f"pool: {len(connections)} total, {active} active, {idle} idle, "
              f"{len(pool._requests)} waiting")

Three shapes, three causes. Many active connections with real traffic means the pool is simply too small. Many active connections with no traffic means a leak — usually a streaming response whose body was never read or closed, so the connection can never be returned. Few connections plus many waiters means the limit is below the demand. Streaming leaks are the most common: client.stream() without consuming or closing holds its connection indefinitely. Verify: stop traffic entirely; after a few seconds every connection should be idle. Any that stay active are leaked.

The pool is empty A decision diamond on pool counters with three outcome cards. The pool is empty What do the pool counters show? many active, real traffic pool too small size it from rate x latency many active, no traffic connections leaked unread streaming responses few connections, many waiters limit below demand check limit_per_host Three shapes, three causes — the counters distinguish them in seconds.

4. Check the per-host limit, not just the total

Both clients enforce a per-host cap that is usually much lower than the total.

import httpx
import aiohttp

# httpx: total, keepalive, and (via the connector) per-host behaviour
limits = httpx.Limits(max_connections=100, max_keepalive_connections=20)

# aiohttp: `limit` is global, `limit_per_host` is the one that usually binds
connector = aiohttp.TCPConnector(limit=100, limit_per_host=30, ttl_dns_cache=300)

A service that talks to one upstream is bounded by the per-host limit regardless of how generous the total is — aiohttp's default limit_per_host=0 means unlimited, but many templates set it to 10 or 30 and then wonder why raising limit changed nothing. Check which of the two is actually binding before tuning either. Verify: raise only the per-host limit and re-run the sweep; if latency improves, that was the binding constraint.

5. Size it, then bound what feeds it

The pool size follows from Little's Law, and the concurrency above it must be bounded too.

TARGET_RPS = 400          # requests per second to this host
P99_LATENCY = 0.06        # seconds, measured at the downstream
HEADROOM = 1.5

pool_size = int(TARGET_RPS * P99_LATENCY * HEADROOM)     # 400 * 0.06 * 1.5 = 36

Raising the pool without bounding request concurrency simply relocates the queue: 5,000 concurrent tasks against a 36-connection pool still queue, they just queue somewhere with a different name. Pair the sized pool with a semaphore at or just below the pool size, so callers are rejected quickly rather than parked invisibly. And check the other end: the downstream's own connection limit, and the file-descriptor ceiling (ulimit -n) that caps everything. Verify: after sizing, the concurrency sweep shows a flat ratio near 1.0 up to the target rate, and PoolTimeout count stays at zero.

Sizing, and then bounding what feeds it A left-to-right calculation of pool size and the concurrency cap that must accompany it. Sizing, and then bounding what feeds it target rate 400 req/s x p99 latency 0.06 s x headroom 1.5 pool size 36 connections cap concurrency at or below 36 Raising the pool without bounding callers just renames the queue.

Verification

  • Latency stops scaling with concurrency. The client-to-downstream p99 ratio stays near 1.0 across the sweep up to target load.
  • Pool wait is near zero. The measured queueing time stays in single-digit milliseconds at peak.
  • Connections return. With traffic stopped, all pooled connections go idle within seconds — no leaks.

Pitfalls and edge cases

  • Creating a client per request. Every request then pays a fresh TCP and TLS handshake and the pool never helps; reuse one client, as in reusing aiohttp ClientSession.
  • No pool timeout. Exhaustion becomes an invisible wait rather than a countable error.
  • Tuning the total while the per-host limit binds. Nothing changes; check both.
  • Unread streaming responses. The connection is held until the response is closed; use async with client.stream(...).
  • Raising the pool past the downstream's capacity. You move the queue to their side, where it is someone else's incident.
  • Ignoring file descriptors. Pool size times hosts plus listeners must stay under ulimit -n.
  • A shared pool across dependencies. One slow host exhausts it for everyone — the case for bulkheads.

Frequently Asked Questions

How do I know if latency is caused by connection pool exhaustion?

Run a concurrency sweep against a downstream with stable latency. If client p99 rises in near-integer multiples of the downstream's unchanged p99, requests are queueing for a connection, and the multiplier at a given concurrency reveals the effective pool size. A genuinely slow dependency raises the downstream's own latency; a blocking call raises event loop lag instead.

What causes an async connection pool to be exhausted with no traffic?

Leaked connections. The usual culprit is a streaming response whose body was never fully read or closed, so the connection cannot be returned to the pool. Sample the pool's active and idle counts after traffic stops: any connection still active seconds later is leaked. Always consume streams inside an async context manager.

How large should an async HTTP connection pool be?

Target requests per second times the downstream's p99 latency, times about 1.5 for headroom — 400 rps at 60 ms gives roughly 36 connections. Then bound the concurrency feeding it at or just below that number, otherwise the queue simply moves from the pool to your task set, and check the total against the process file-descriptor limit.

Why does raising max_connections not help in aiohttp?

Because limit_per_host is usually the binding constraint. A service that talks to a single upstream is capped by the per-host value no matter how large the global limit is. Raise limit_per_host — or set it to 0 for unlimited per host — and re-measure before touching the global limit.