Skip to content

Rate Limiting & Throttling in Asyncio

An async client is a machine for exceeding other people's limits. Ten lines of TaskGroup and a list comprehension will happily issue 5,000 concurrent requests, and the first thing you learn in production is that almost nothing downstream wants them: the API returns 429, the database's connection pool queues, the third-party SDK silently degrades, and your own service turns a burst of work into a burst of retries that makes everything worse. Throttling is the discipline of shaping that outflow deliberately — deciding how many at once and how many per second, which are different questions with different tools.

This section covers both. Concurrency caps (asyncio.Semaphore) bound how many operations are in flight; rate limiters (token bucket, sliding window) bound how many start per unit time. Real systems need both, plus a third layer: cooperative behaviour when the upstream tells you it has had enough. The parent overview, Concurrent Execution & Worker Patterns, covers how work gets distributed; this section covers how to stop it arriving faster than the world can absorb.

Scope of this section:

  • Concurrency limits versus rate limits: which failure each one prevents.
  • Token bucket and sliding-window limiters that work with the event loop rather than against it.
  • Per-host, per-tenant, and global limiter composition without deadlock.
  • Cooperative handling of 429 / Retry-After and adaptive throttling under sustained pressure.
  • The metrics that show whether a limiter is protecting you or silently becoming the bottleneck.

Architectural principles

  • A concurrency cap is not a rate limit. A semaphore of 10 permits allows 10 in flight — which is 10 requests per second if each takes a second, or 10,000 per second if each takes a millisecond. If the downstream publishes a per-second quota, you need a limiter that counts starts over time, not one that counts occupancy.
  • Limit at the narrowest waist. Put the limiter where the constraint actually is: per host for HTTP, per pool for a database, per tenant for fairness. A single global limiter over heterogeneous work throttles your fast paths to protect your slow ones.
  • Throttling must apply backpressure, not queue infinitely. A limiter with an unbounded waiting line converts a rate problem into a memory problem. Pair every limiter with a bounded queue or an acquisition timeout so the caller learns that the system is full.
  • The upstream's opinion wins. Your configured limit is a guess; a 429 with Retry-After is data. Treat it as the authoritative signal and back off, rather than retrying into a wall — the discipline is covered in retry and backoff strategies.
  • Fairness is a design decision. asyncio.Semaphore releases waiters in FIFO order, which is fair; a naive sleep-until-allowed loop is not, and starves whoever is unlucky. Choose deliberately when tenants share a limiter.
Two different limits, two different failures Two columns contrasting concurrency caps and rate limits. Two different limits, two different failures concurrency cap how many at once Semaphore, pool size, worker count Protects memory and sockets Rate = permits ÷ latency Fails as connection errors rate limit how many per second Token bucket, sliding window Protects a published quota Independent of latency Fails as 429 responses Little's Law links them: in-flight = rate × latency. Most systems need both.

Execution model: what "waiting" costs on the loop

A limiter's job is to make a coroutine wait, and on an event loop, waiting is nearly free — a suspended coroutine holds a few kilobytes and no thread. That is what makes fine-grained throttling practical in async code and impractical with threads: 10,000 coroutines parked on a semaphore cost megabytes, while 10,000 blocked threads cost gigabytes and a scheduler meltdown.

But "free" applies only to suspension, not to the objects doing the waiting. Each waiter is a Future in the semaphore's internal queue plus the task holding it, and each queued item in a bounded queue is retained memory. The cost of a limiter is therefore the cost of its waiting line, which is why the line must be bounded somewhere. When a limiter is saturated and the caller keeps creating tasks, the memory grows with the arrival rate, not with the service rate — the classic unbounded-queue failure described in async queue management.

The scheduling detail that matters: asyncio.Semaphore.acquire() parks the caller in a FIFO deque of futures, and release() wakes exactly one waiter through loop.call_soon(). There is no thundering herd and no priority inversion, but there is also no priority at all — a limiter cannot prefer interactive work over batch work unless you build separate limiters for each class. For time-based limiters, the analogous detail is that asyncio.sleep() schedules a timer on the loop's heap; a limiter that sleeps per request creates one timer per request, which is fine at hundreds per second and wasteful at hundreds of thousands.

What a saturated limiter actually holds Four stages from arriving work to the downstream constraint, highlighting the waiting line. What a saturated limiter actually holds arriving work tasks created waiting line futures in a deque permits N in flight downstream the real constraint Suspension is nearly free; the waiting line is not — bound it or it becomes the leak.

Pattern catalogue

Concurrency cap with a semaphore

The simplest and most common control: bound how many operations run at once.

import asyncio

LIMIT = asyncio.Semaphore(20)


async def fetch(client, url: str) -> bytes:
    async with LIMIT:                       # 20 in flight, FIFO for the rest
        response = await client.get(url)
        return response.content


async def fetch_all(client, urls: list[str]) -> list[bytes]:
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(fetch(client, u)) for u in urls]
    return [t.result() for t in tasks]

Use it when the constraint is a resource that is occupied for the duration of the call: connections, file descriptors, memory per in-flight response. Note the trade-off — creating all the tasks up front still allocates one task per URL even though only 20 run, so for very large inputs, limit task creation too, with a bounded queue and a fixed worker pool.

Token bucket for a per-second quota

A token bucket refills at a fixed rate and allows short bursts up to its capacity — the right model when a provider publishes "N requests per second, bursts allowed".

import asyncio


class TokenBucket:
    def __init__(self, rate: float, capacity: float | None = None) -> None:
        self.rate = rate                              # tokens per second
        self.capacity = capacity if capacity is not None else rate
        self._tokens = self.capacity
        self._updated = asyncio.get_event_loop().time()
        self._lock = asyncio.Lock()

    async def acquire(self, tokens: float = 1.0) -> None:
        while True:
            async with self._lock:
                loop = asyncio.get_running_loop()
                now = loop.time()
                self._tokens = min(self.capacity,
                                   self._tokens + (now - self._updated) * self.rate)
                self._updated = now
                if self._tokens >= tokens:
                    self._tokens -= tokens
                    return
                deficit = tokens - self._tokens
            await asyncio.sleep(deficit / self.rate)  # sleep OUTSIDE the lock

The critical detail is sleeping outside the lock: holding an asyncio.Lock across an await asyncio.sleep() serialises every waiter behind the first one and turns a rate limiter into a queue with a single server. Refilling lazily on each acquisition (rather than with a background timer) keeps the limiter free when idle.

Sliding window for a strict quota

When the provider enforces "no more than N in any 60-second window", a token bucket's burst allowance can still trip it. A deque of timestamps enforces the window exactly.

import asyncio
from collections import deque


class SlidingWindow:
    def __init__(self, limit: int, window: float) -> None:
        self.limit, self.window = limit, window
        self._events: deque[float] = deque()
        self._lock = asyncio.Lock()

    async def acquire(self) -> None:
        while True:
            async with self._lock:
                loop = asyncio.get_running_loop()
                now = loop.time()
                while self._events and now - self._events[0] >= self.window:
                    self._events.popleft()
                if len(self._events) < self.limit:
                    self._events.append(now)
                    return
                wait = self.window - (now - self._events[0])
            await asyncio.sleep(wait)

It is exact but O(limit) in memory, so it suits limits in the thousands, not the millions. Prefer it when overshooting costs you an account suspension; prefer the token bucket when smoothness matters more than exactness.

Composed limits: global, per host, per tenant

Real systems have several limits at once. Compose them by acquiring in a fixed global order — the same rule that prevents lock-ordering deadlocks.

import asyncio
from contextlib import asynccontextmanager

GLOBAL = asyncio.Semaphore(200)
PER_HOST: dict[str, asyncio.Semaphore] = {}
PER_TENANT: dict[str, TokenBucket] = {}


@asynccontextmanager
async def throttle(host: str, tenant: str):
    host_sem = PER_HOST.setdefault(host, asyncio.Semaphore(20))
    await PER_TENANT[tenant].acquire()          # 1. rate  (never held, just delays)
    async with GLOBAL:                          # 2. broad concurrency
        async with host_sem:                    # 3. narrow concurrency
            yield

Acquire rate limits before concurrency limits: a rate limiter's wait does not occupy a slot, so doing it first avoids holding a scarce permit while merely waiting for a clock. Then always take the broader semaphore before the narrower one.

Cooperative handling of upstream 429s

The most important limiter is the one the server tells you about.

import asyncio
import random


async def call_with_quota(client, url: str, attempts: int = 5):
    for attempt in range(attempts):
        response = await client.get(url)
        if response.status_code != 429:
            return response
        retry_after = float(response.headers.get("Retry-After", 0)) or \
            min(2 ** attempt, 30) * random.uniform(0.5, 1.0)
        await asyncio.sleep(retry_after)
    raise RuntimeError("rate limited beyond retry budget")

Honour Retry-After when present; fall back to exponential backoff with jitter when it is not. The pattern is expanded, including how to make every concurrent caller back off together rather than one at a time, in handling 429 and Retry-After responses in async clients.

Choosing the instrument Five-row matrix comparing semaphores, token buckets, sliding windows, bounded queues and quota gates. Choosing the instrument bounds bursts use when Semaphore in flight n/a sockets, memory, pool Token bucket starts/second up to capacity quota with burst allowance Sliding window starts/window none hard quota, no overshoot Bounded queue items resident maxsize backpressure at ingress Quota gate everything, on 429 n/a the upstream said stop Pick per constraint, then acquire rate limits before concurrency limits.

Resource boundaries

Sizing a limiter is an arithmetic exercise, not a guess. Little's Law connects the two axes: in-flight = rate × latency. If the upstream allows 100 requests per second and the median call takes 200 ms, then 20 concurrent calls saturate the quota; a semaphore of 200 would simply queue 180 of them at the far end while you burn 429s.

Constraint Instrument How to size it
Provider quota (req/s) Token bucket rate = published quota × 0.9 safety margin; capacity = one second of burst
Provider quota (req/window) Sliding window limit = published limit − in-flight allowance
Connections / file descriptors Semaphore Pool size, matched to connection pool sizing
Memory per in-flight response Semaphore container memory budget ÷ p99 response size
Fairness between tenants Per-tenant limiter quota ÷ tenants, with a shared burst pool
Total system load Bounded queue at ingress Enough depth to absorb a burst, not a backlog

The waiting line needs its own bound. An acquisition timeout (async with asyncio.timeout(2): await limiter.acquire()) converts an over-full limiter into a fast, visible failure — far better than a request that waits 40 seconds for a permit and then finds its client gone.

Integrated production example

This crawler combines every layer: a bounded ingest queue, a fixed worker pool, a per-host concurrency cap, a global token bucket, cooperative 429 handling, and the metrics to see all of it.

import asyncio
import logging
import random
import time
from collections import defaultdict

log = logging.getLogger("throttle")


class TokenBucket:
    def __init__(self, rate: float, capacity: float | None = None) -> None:
        self.rate, self.capacity = rate, capacity if capacity is not None else rate
        self._tokens, self._updated = self.capacity, time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self, tokens: float = 1.0) -> float:
        waited = 0.0
        while True:
            async with self._lock:
                now = time.monotonic()
                self._tokens = min(self.capacity,
                                   self._tokens + (now - self._updated) * self.rate)
                self._updated = now
                if self._tokens >= tokens:
                    self._tokens -= tokens
                    return waited
                delay = (tokens - self._tokens) / self.rate
            waited += delay
            await asyncio.sleep(delay)


class Throttled:
    """Per-host concurrency + global rate, with cooperative 429 backoff."""

    def __init__(self, rate: float, per_host: int = 8) -> None:
        self.bucket = TokenBucket(rate)
        self.host_sems: dict[str, asyncio.Semaphore] = defaultdict(
            lambda: asyncio.Semaphore(per_host))
        self.paused_until: dict[str, float] = {}
        self.stats = {"waited": 0.0, "throttled": 0, "calls": 0}

    async def get(self, client, host: str, path: str):
        pause = self.paused_until.get(host, 0.0) - time.monotonic()
        if pause > 0:                                   # a 429 is still in effect
            await asyncio.sleep(pause)
        self.stats["waited"] += await self.bucket.acquire()
        async with self.host_sems[host]:
            self.stats["calls"] += 1
            response = await client.get(f"https://{host}{path}")
            if response.status_code == 429:
                delay = float(response.headers.get("Retry-After", 0)) or \
                    random.uniform(1.0, 5.0)
                # Pause the whole host, not just this caller: every concurrent
                # task sees the same quota, so they must all back off together.
                self.paused_until[host] = time.monotonic() + delay
                self.stats["throttled"] += 1
                raise TimeoutError(f"429 from {host}, paused {delay:.1f}s")
            return response


async def worker(name: str, queue: asyncio.Queue, api: Throttled, client) -> None:
    while True:
        host, path = await queue.get()
        try:
            async with asyncio.timeout(10):
                await api.get(client, host, path)
        except TimeoutError as exc:                     # includes the 429 pause
            log.warning("%s: %s", name, exc)
        except Exception:
            log.exception("%s: unexpected failure on %s%s", name, host, path)
        finally:
            queue.task_done()


async def monitor(queue: asyncio.Queue, api: Throttled, interval: float = 15.0) -> None:
    while True:
        await asyncio.sleep(interval)
        log.info("queue=%d calls=%d throttled=%d wait_total=%.1fs",
                 queue.qsize(), api.stats["calls"], api.stats["throttled"],
                 api.stats["waited"])


async def main(client, targets: list[tuple[str, str]]) -> None:
    queue: asyncio.Queue = asyncio.Queue(maxsize=500)   # bounded: backpressure
    api = Throttled(rate=90.0, per_host=8)              # 90 req/s of a 100 quota
    async with asyncio.TaskGroup() as tg:
        for i in range(16):
            tg.create_task(worker(f"w{i}", queue, api, client), name=f"crawl.w{i}")
        tg.create_task(monitor(queue, api), name="crawl.monitor")
        for target in targets:
            await queue.put(target)                     # suspends when full
        await queue.join()
        raise SystemExit(0)                             # stops the TaskGroup cleanly

Three decisions carry the design. The queue is bounded, so a producer faster than the quota is throttled at the source instead of filling memory. The 429 response pauses the host, not the caller, because every concurrent task shares the same upstream quota and would otherwise take turns discovering the limit. And the accumulated wait time is a first-class statistic: it is the difference between "we are fast" and "we are fast because we are allowed to be".

Diagnostic Hook — is the limiter protecting you or throttling you?

Export three numbers per limiter. Wait time (seconds spent inside acquire(), as a histogram): near zero means the limiter is not binding; growing means it is now your bottleneck. Saturation (permits in use ÷ total, or tokens remaining ÷ capacity): pinned at the limit means demand exceeds the quota. Upstream rejections (429s per minute): any sustained non-zero rate means your local limit is set above the real one, so lower it — a 429 costs a full round trip and often counts against the quota anyway. Alert when wait time exceeds a tenth of your request budget, and treat a rising 429 rate as a configuration bug rather than something to retry harder.

Reading a limiter from three numbers Four limiter readings with escalating urgency. Reading a limiter from three numbers wait time p99 near zero not binding wait time growing now the bottleneck saturation pinned at 1.0 demand exceeds quota 429 rate above 1% local limit set too high Bar length encodes how urgently each reading needs a decision, not a measured quantity. Wait time and saturation say whether the limit is protecting you or costing you.

Failure modes

Failure mode Root cause Detection Fix
Sustained 429s despite a limiter Local rate set above the real quota, or several instances sharing one quota Upstream rejection rate stays non-zero Divide the quota by instance count; add a safety margin
Limiter serialises everything await asyncio.sleep() while holding the limiter's lock Throughput equals one request per delay, regardless of the limit Sleep outside the lock; only mutate state inside it
Memory growth under load Unbounded waiting line at the limiter Task count and RSS climb while throughput is flat Bound the ingest queue; add an acquisition timeout
Deadlock between limiters Two limiters acquired in different orders by different paths Tasks parked on acquire() in a cycle Acquire in one global order: rate, then broad, then narrow
Tenant starvation One shared limiter, one heavy tenant p99 wait time skewed by tenant Per-tenant limiters with a shared burst pool
Burst at process start Full bucket plus a full queue on deploy A spike of upstream errors right after rollout Start the bucket partially filled; stagger instance start
Thundering herd after a pause Every waiter wakes at the same instant after a Retry-After A synchronised spike then another 429 Add jitter to the resume time per caller

Frequently Asked Questions

What is the difference between a semaphore and a rate limiter in asyncio?

A semaphore bounds concurrency — how many operations are in flight at once — while a rate limiter bounds starts per unit time. They are related by Little's Law: in-flight equals rate times latency. If a downstream publishes a per-second quota, a semaphore alone cannot enforce it, because the same concurrency produces wildly different rates as latency changes.

How do I implement a token bucket rate limiter with asyncio?

Track a token count and the timestamp of the last refill, add rate times elapsed seconds on each acquisition, and sleep for the deficit divided by the rate when there are not enough tokens. Guard the state mutation with an asyncio.Lock, but always await the sleep outside that lock, otherwise every waiter queues behind the first and the limiter degrades into a single-server queue.

Should the rate limit be shared across service instances?

If the quota belongs to the upstream, yes — divide it. Ten instances each configured for the full quota will collectively exceed it tenfold. Either divide the published quota by the instance count with a safety margin, or coordinate through a shared store such as Redis when the traffic split is uneven enough that static division wastes quota.

How should an async client react to a 429 response?

Honour Retry-After when the header is present, and fall back to exponential backoff with jitter when it is not. Crucially, apply the pause at the level that shares the quota — usually the host or the account — so all concurrent callers back off together rather than each discovering the limit in turn, and add per-caller jitter so they do not resume in a synchronised burst.

Where should the limiter live: at the client, the queue, or the worker?

At the narrowest waist that matches the constraint. Bound task creation with a queue at ingress, bound in-flight work with a semaphore in the worker, and bound starts per second with a rate limiter in the client wrapper for that host. One limiter at only one of those layers usually leaves either memory or quota unprotected.