Skip to content

Limiting Concurrent Requests with asyncio.Semaphore

The code that gets written first is await asyncio.gather(*[fetch(u) for u in urls]), and it works beautifully until the list has 50,000 entries. Then the process opens as many sockets as the kernel allows, the upstream starts refusing connections, memory climbs with every buffered response, and the failure looks like a network problem rather than what it is: unbounded concurrency. asyncio.Semaphore is the standard fix — a counter of permits that suspends callers when they are all taken — but wrapping the request in async with sem: only solves half of it. The tasks still all exist, they still all hold their arguments and buffers, and the memory curve barely moves.

This guide covers the whole fix: choosing a permit count from the actual constraint, bounding task creation as well as task execution, adding acquisition timeouts so a saturated limiter fails fast instead of hanging, and measuring how long callers wait so you can tell a protective limit from a self-inflicted bottleneck.

Prerequisites

  • Python 3.11+ for asyncio.TaskGroup and asyncio.timeout(). The HTTP examples use httpx:
pip install httpx
A permit through its lifecycle Four states of a semaphore permit with a return edge closing the cycle. A permit through its lifecycle available acquired in flight released async with sem request sent response or error returned to the pool The return edge is why `async with` matters: exceptions and cancellation must still release.

1. Cap in-flight work

The basic pattern: acquire before the operation, release automatically on exit, including on exception or cancellation.

import asyncio
import httpx

LIMIT = asyncio.Semaphore(20)


async def fetch(client: httpx.AsyncClient, url: str) -> bytes:
    async with LIMIT:                       # suspends when 20 are already running
        response = await client.get(url, timeout=10.0)
        response.raise_for_status()
        return response.content

async with is the only form worth using: a manual acquire()/release() pair leaks a permit whenever an exception or a CancelledError skips the release, and a leaked permit is permanent — the limit silently drops to 19, then 18, until nothing runs at all. Verify: run with 200 URLs and log LIMIT._value periodically; it must return to 20 when the batch finishes.

2. Bound task creation, not just execution

A semaphore controls how many coroutines run past the acquire(), not how many tasks exist. With 50,000 URLs you still create 50,000 tasks, each holding its arguments, its context, and a slot in the loop's bookkeeping — often a gigabyte before a single response arrives.

import asyncio


async def worker(queue: asyncio.Queue, client, results: list) -> None:
    while True:
        url = await queue.get()
        try:
            results.append(await fetch(client, url))
        finally:
            queue.task_done()


async def crawl(client, urls: list[str]) -> list[bytes]:
    queue: asyncio.Queue = asyncio.Queue(maxsize=100)     # bounded: backpressure
    results: list[bytes] = []
    async with asyncio.TaskGroup() as tg:
        workers = [tg.create_task(worker(queue, client, results), name=f"fetch.{i}")
                   for i in range(20)]                    # 20 tasks, not 50,000
        for url in urls:
            await queue.put(url)                          # suspends when full
        await queue.join()
        for w in workers:
            w.cancel()
    return results

Twenty long-lived workers pulling from a bounded queue give the same concurrency with constant memory, and the await queue.put() propagates backpressure all the way to whatever is producing URLs. Use the semaphore form for a few hundred items; use the worker-pool form beyond that — the sizing rules are in worker pool implementations. Verify: compare RSS between the two shapes on a 50,000-item input; the pool version should be flat where the gather version grows linearly.

Two ways to run 50,000 URLs at 20 concurrency Two columns contrasting task-per-item with a bounded queue and fixed worker pool. Two ways to run 50,000 URLs at 20 concurrency gather + semaphore simple, memory grows 50,000 tasks created at once Each holds args and context RSS grows with input size Fine up to a few hundred items bounded queue + workers constant memory 20 worker tasks, one queue maxsize items resident Backpressure reaches the producer The shape for large inputs Both cap execution at 20; only one caps what the process is holding.

3. Choose the permit count from the real constraint

The number is not a preference; it is derived from whichever resource runs out first.

# Whichever of these is smallest is your permit count.
by_pool     = 20               # connection pool size for this host
by_memory   = 512_000_000 // 8_000_000     # container budget / p99 response size = 64
by_upstream = 100 * 0.2        # published 100 req/s x 200 ms latency = 20 in flight
by_cpu      = 8                # if each response needs meaningful parsing
permits = min(by_pool, by_memory, by_upstream, by_cpu)     # -> 8

The upstream row is Little's Law: sustaining a rate of R with latency L requires R × L in flight, so a quota of 100/s at 200 ms is saturated by exactly 20 concurrent calls. Setting the semaphore above the connection pool size is the most common mistake — it simply relocates the queue from your semaphore into the HTTP client's pool, where you cannot see it. Verify: raise the permit count by 50% and re-run; if throughput does not improve, the limiter is not the bottleneck and the extra permits only add queueing elsewhere.

4. Fail fast when the limiter is saturated

A caller blocked forever on acquire() is invisible: the request is neither running nor failing. An acquisition timeout turns saturation into a fast, countable error.

import asyncio


async def fetch_or_shed(client, url: str, budget: float = 2.0) -> bytes | None:
    try:
        async with asyncio.timeout(budget):
            await LIMIT.acquire()
    except TimeoutError:
        SHED.inc()                                # load-shedding is a metric, not a crash
        return None
    try:
        response = await client.get(url, timeout=10.0)
        return response.content
    finally:
        LIMIT.release()

Note the explicit acquire()/release() here: the timeout must cover only the acquisition, not the request, so async with LIMIT cannot be used inside the same timeout() block. Keep the release in a finally so the permit returns on every path, and size the budget from what the caller can still use — a request whose client has already given up is work you should shed, not run. Verify: saturate the limiter deliberately and confirm the shed counter rises while in-flight work stays exactly at the permit count.

5. Measure the wait, not just the limit

The one number that tells you whether the limiter is protecting you or throttling you is time spent inside acquire().

import asyncio
import time
from contextlib import asynccontextmanager


class MeteredSemaphore:
    def __init__(self, permits: int) -> None:
        self._sem = asyncio.Semaphore(permits)
        self.permits = permits
        self.wait_total = 0.0
        self.waiters = 0

    @asynccontextmanager
    async def __call__(self):
        start = time.monotonic()
        self.waiters += 1
        try:
            async with self._sem:
                self.wait_total += time.monotonic() - start
                self.waiters -= 1
                yield
        except BaseException:
            self.waiters -= 1
            raise

    @property
    def in_use(self) -> int:
        return self.permits - self._sem._value       # private, but stable in practice

Export wait_total as a histogram, waiters as a gauge, and in_use / permits as saturation. A saturation pinned at 1.0 with a rising wait means demand exceeds the limit — the correct response is to decide deliberately whether to raise the limit, shed load, or add capacity, rather than to discover it as unexplained latency. Verify: under a normal load test, p99 acquisition wait should be a small fraction of total request time; if it dominates, the limit is your bottleneck.

What the wait time is telling you A decision diamond on acquisition wait with three outcome cards. What the wait time is telling you Is acquisition wait a large share of latency? no, near zero limit is not binding raise it only if throughput needs it yes, and saturation is 1.0 demand exceeds the limit shed load or add capacity yes, but saturation is low permits leaked check for a missing release Saturation without wait is healthy; wait without saturation is a bug.

Verification

  • Concurrency is what you configured. Instrument the upstream (or a local stub) and confirm the maximum simultaneous in-flight requests equals the permit count, never more.
  • Memory is flat. Processing 10× the input does not multiply RSS — the signature that task creation, not just execution, is bounded.
  • Permits are conserved. After a run that includes failures and cancellations, the semaphore's available permits return to the configured value.

Pitfalls and edge cases

  • Manual acquire without finally. Any exception between acquire and release leaks a permit permanently. Prefer async with; when you cannot, use try/finally.
  • A semaphore created at import time. In older code a semaphore captured the loop at construction; keep construction inside async def main() so it always binds to the running loop.
  • Sharing one semaphore across unrelated work. A single global cap makes a slow endpoint throttle a fast one. Use one limiter per constraint — per host, per pool, per tenant.
  • Cap above the connection pool size. The queue simply moves into the HTTP client where it is invisible. Match the permit count to the pool, or size the pool up deliberately.
  • Assuming a cap enforces a rate. Concurrency times 1/latency is your rate, and latency moves. If the upstream publishes requests per second, add a token bucket.
  • Cancellation while waiting. A task cancelled inside acquire() must not leave a permit consumed; async with handles this correctly, hand-rolled wrappers frequently do not.
  • Nested limiters acquired in different orders. Two code paths taking a global and a per-host semaphore in opposite orders deadlock under load. Fix the order globally.

Frequently Asked Questions

How many permits should an asyncio.Semaphore have?

Take the minimum of the real constraints: the connection pool size for that host, the container memory budget divided by the p99 response size, the upstream quota multiplied by median latency (Little's Law), and any CPU cost per response. Whichever is smallest is the limit; anything larger just relocates the queue somewhere you cannot observe it.

Does asyncio.Semaphore limit requests per second?

No. It limits how many operations run concurrently. The resulting rate is concurrency divided by latency, so the same semaphore produces very different rates as the upstream speeds up or slows down. Enforce a per-second quota with a token bucket or sliding window, and use the semaphore for resource occupancy.

Why does gather with a semaphore still use so much memory?

Because the semaphore bounds execution, not creation. Every coroutine in the gather becomes a task immediately, holding its arguments and context even while parked on acquire(). For large inputs, feed a bounded asyncio.Queue with a fixed pool of worker tasks so only the queue's maxsize items are resident.

How do I add a timeout to acquiring a semaphore?

Wrap only the acquisition: async with asyncio.timeout(budget): await sem.acquire(), then run the work and release in a finally block. Do not wrap the whole async with sem: block, because that timeout would also cancel the work itself rather than shedding the request while it is still queued.