Skip to content

Token Bucket Rate Limiter for Asyncio Clients

You have an upstream that allows 100 requests per second and a client that can issue thousands. A semaphore does not solve it — capping concurrency at 20 gives you 100 requests per second only while latency happens to be 200 ms, and produces 400 per second the moment the upstream gets faster. What you need is a limiter that counts starts over time: a token bucket that hands out permits at a fixed rate, allows a bounded burst when you have been idle, and makes callers wait exactly as long as the arithmetic requires.

The implementation is thirty lines, and roughly half of the ones you find in the wild have the same two bugs: they hold the lock across the sleep (which serialises every caller behind the first, capping you at one request per delay), or they refill with a background task (which keeps a timer alive forever and drifts under load). This guide builds one that avoids both, sizes the burst honestly, and — most importantly — verifies that the rate you get is the rate you asked for.

Prerequisites

  • Python 3.11+; the limiter itself is standard library only. The example client uses httpx:
pip install httpx
One acquisition, step by step Four sequential steps of a token bucket acquisition with side annotations. One acquisition, step by step refill: tokens += elapsed × rate clamped at capacity enough tokens? compare against the request cost subtract and return the fast path — no waiting otherwise sleep deficit ÷ rate outside the lock, then re-check Lazy refill means an idle limiter costs nothing and can never drift.

1. Model the bucket

A token bucket has three pieces of state: how many tokens are available, when they were last computed, and the ceiling. Tokens accrue at rate per second and stop at capacity.

import asyncio
import time


class TokenBucket:
    def __init__(self, rate: float, capacity: float | None = None) -> None:
        if rate <= 0:
            raise ValueError("rate must be positive")
        self.rate = rate
        self.capacity = rate if capacity is None else capacity
        self._tokens = self.capacity          # start full: an idle client may burst
        self._updated = time.monotonic()

Use time.monotonic() (or loop.time()), never time.time(): an NTP correction mid-request would otherwise mint or destroy tokens. Starting full means the first burst after startup is allowed — reasonable for a long-lived client, dangerous when fifty replicas deploy at once, which step 5 addresses. Verify: instantiate with rate=10 and confirm _tokens == 10.0 before any acquisition.

2. Refill lazily instead of with a background task

The naive design runs a timer that adds tokens every tick. It is worse in every way: it keeps a task alive for an idle limiter, its resolution is bounded by the tick, and it drifts when the loop is busy. Computing the refill on demand is exact and free when unused.

    def _refill(self) -> None:
        now = time.monotonic()
        elapsed = now - self._updated
        if elapsed > 0:
            self._tokens = min(self.capacity, self._tokens + elapsed * self.rate)
            self._updated = now

Elapsed time times rate is the number of tokens that would have accrued, clamped by capacity. Because it is derived from a monotonic clock rather than accumulated ticks, it cannot drift: a limiter that was starved of CPU for two seconds credits exactly two seconds of tokens the moment anyone asks. Verify: call _refill() after time.sleep(0.5) on a rate=10 bucket and confirm the token count rose by ~5, capped at capacity.

3. Acquire without holding the lock across the sleep

This is the step that separates a working limiter from a broken one. The lock protects the token arithmetic; it must be released before the caller waits.

    async def acquire(self, tokens: float = 1.0) -> float:
        """Wait until `tokens` are available. Returns seconds spent waiting."""
        if tokens > self.capacity:
            raise ValueError("request larger than bucket capacity would never succeed")
        waited = 0.0
        while True:
            async with self._lock:                   # short, non-awaiting section
                self._refill()
                if self._tokens >= tokens:
                    self._tokens -= tokens
                    return waited
                deficit = tokens - self._tokens
                delay = deficit / self.rate
            await asyncio.sleep(delay)               # <- outside the lock
            waited += delay

Two subtleties. The loop re-checks after sleeping because another caller may have consumed the tokens meanwhile — the sleep is an estimate, not a reservation. And the guard against tokens > capacity prevents the silent infinite loop where a caller waits forever for a bucket that can never hold enough. Verify: with rate=5, capacity=5, launch 20 concurrent acquire() calls and confirm the last one returns at ~3 s (5 immediate, then 5/s), not at 20× the delay, which is the signature of a lock held across the sleep.

Where the lock is released decides the rate Two columns contrasting a lock held across a sleep with the correct release-before-sleep pattern. Where the lock is released decides the rate sleep inside the lock the common bug Every waiter queues on the lock One grant per sleep interval Configured rate is ignored Throughput collapses sleep outside the lock correct Lock held only for arithmetic Waiters wake independently Observed rate matches config Re-check after waking The same thirty lines behave completely differently depending on this one placement.

4. Wire it into the client

The limiter belongs in one place — the wrapper around the calls that share the quota — so no code path can bypass it.

import httpx


class RateLimitedClient:
    def __init__(self, client: httpx.AsyncClient, rate: float, burst: float) -> None:
        self._client = client
        self._bucket = TokenBucket(rate, burst)
        self.wait_seconds = 0.0                       # exported as a metric

    async def get(self, url: str, *, cost: float = 1.0, **kwargs) -> httpx.Response:
        self.wait_seconds += await self._bucket.acquire(cost)
        return await self._client.get(url, **kwargs)


async def main() -> None:
    async with httpx.AsyncClient(timeout=10.0) as raw:
        api = RateLimitedClient(raw, rate=90.0, burst=90.0)   # 90 of a 100/s quota
        async with asyncio.TaskGroup() as tg:
            for url in urls:
                tg.create_task(api.get(url))

The cost parameter is worth keeping even if you start with 1.0 everywhere: quota systems frequently price a bulk endpoint at ten units, and a limiter that cannot express that will exceed the quota while looking correct. Verify: issue 200 requests through the wrapper against a local stub and confirm total elapsed time is approximately (200 − burst) / rate seconds.

5. Size the burst and the safety margin

Two numbers turn a correct limiter into a well-behaved one.

QUOTA = 100.0              # documented upstream limit, requests/second
INSTANCES = 4              # replicas sharing that quota
SAFETY = 0.9               # leave headroom for clock skew and retries

rate = QUOTA * SAFETY / INSTANCES        # 22.5 req/s per instance
burst = rate                             # one second of burst; raise only if the
                                         # upstream documents a burst allowance
bucket = TokenBucket(rate, burst)

Divide by replica count — this is the single most common production failure, where each of ten instances is configured with the full quota and the fleet exceeds it tenfold. Keep the burst modest: capacity is the size of the spike you will send after any idle period, and a large capacity plus a synchronised deploy is exactly how a fleet trips a limit it was meant to respect. If the deploy spike matters, start the bucket empty (self._tokens = 0.0) so a fresh process ramps in. Verify: count requests reaching a stub server in the first second after starting all replicas simultaneously; it must stay under the quota.

6. Confirm the observed rate

A limiter is only correct if measurement says so. Run it against a counter and compare with the arithmetic.

import asyncio
import time


async def measure(rate: float, burst: float, n: int) -> None:
    bucket = TokenBucket(rate, burst)
    start = time.monotonic()
    stamps: list[float] = []

    async def one() -> None:
        await bucket.acquire()
        stamps.append(time.monotonic() - start)

    async with asyncio.TaskGroup() as tg:
        for _ in range(n):
            tg.create_task(one())

    elapsed = time.monotonic() - start
    print(f"{n} acquisitions in {elapsed:.2f}s -> {n / elapsed:.1f}/s (target {rate})")
    per_second = [sum(1 for s in stamps if i <= s < i + 1) for i in range(int(elapsed) + 1)]
    print("per-second:", per_second)          # first bucket may equal `burst`


asyncio.run(measure(rate=20.0, burst=20.0, n=200))
# 200 acquisitions in 9.02s -> 22.2/s (target 20.0)
# per-second: [20, 20, 20, 20, 20, 20, 20, 20, 20, 20]

The aggregate rate exceeds the target slightly because the initial burst is free; the per-second histogram is the honest view and should show no bucket above rate after the first. Verify: no second in the histogram exceeds rate (other than the initial burst), and the run time matches (n − burst) / rate within a few percent.

Acquisitions per second, rate=20 burst=20 Six bars showing a steady twenty per second and a single collapsed bar for the buggy variant. Acquisitions per second, rate=20 burst=20 second 1 (burst) 20 second 2 20 second 3 20 second 4 20 second 5 20 with the lock bug 1 — one per delay 200 acquisitions measured with the step 6 harness on an idle loop. The per-second histogram is the honest check; the aggregate average hides the burst.

Verification

  • Shape is correct. The per-second histogram shows at most burst in the first second and at most rate thereafter.
  • Concurrency is not serialised. With 20 concurrent callers on a rate=5 bucket, tokens are granted five per second in FIFO order; total time matches the arithmetic rather than 20 sequential delays.
  • Upstream agrees. After deploying, the upstream's rejection (429) rate is zero over a full traffic peak — the definitive test that your local rate is below the real limit.

Pitfalls and edge cases

  • Sleeping while holding the lock. The classic bug: throughput collapses to one acquisition per delay regardless of the configured rate. Mutate state inside the lock; wait outside it.
  • Using wall-clock time. time.time() can jump backwards or forwards; a backwards jump freezes the limiter until the clock catches up. Use time.monotonic() or loop.time().
  • A bucket per call site. Each instance has its own tokens, so N independent buckets grant N times the rate. Construct one and inject it.
  • Requests larger than capacity. A cost above capacity can never be satisfied and loops forever without the explicit guard in step 3.
  • Forgetting the replica divisor. Per-process limits multiply by replica count. Divide the quota, or coordinate through a shared store when the split is uneven.
  • Thundering herd on resume. Many callers computing the same delay wake in the same tick. Add a few milliseconds of jitter per caller when the limiter is heavily contended.
  • The bucket is not a queue. It grants permission, not ordering or fairness beyond wake order. If interactive work must beat batch work, use two buckets, not one.

Frequently Asked Questions

Why does my asyncio token bucket only allow one request per second regardless of the rate?

The lock is being held across the sleep. If acquire() awaits asyncio.sleep() inside async with self._lock, every other caller is blocked on the lock for the whole delay, so the limiter grants one permit per sleep interval no matter how many tokens accrue. Compute the delay inside the lock, release it, then sleep, then re-check.

What burst capacity should a token bucket use?

Start with one second of rate — capacity equals rate — which allows a short spike after an idle period without exceeding the quota over any rolling second. Increase it only if the upstream documents a burst allowance, and reduce it (or start the bucket empty) if many replicas deploy simultaneously, since a full bucket on every instance produces a synchronised burst.

Should the token bucket refill on a timer or on demand?

On demand. Lazy refill — tokens plus elapsed time times rate, clamped to capacity — is exact, costs nothing while the limiter is idle, and cannot drift when the loop is busy. A background timer keeps a task alive forever, quantises the rate to its tick, and falls behind exactly when the process is under load.

How is a token bucket different from a sliding window limiter?

A token bucket smooths traffic to an average rate and permits bursts up to its capacity; a sliding window enforces a hard count within a rolling interval and permits no overshoot. Use the bucket when the provider allows bursts and you value smooth throughput, and the sliding window when exceeding the documented count has real consequences such as account suspension.