Skip to content

Prioritizing Requests Under a Shared Rate Limit

When interactive requests and a batch job share one upstream rate limit, a first-in-first-out limiter makes the user wait behind the batch. A priority-aware limiter lets urgent calls overtake — and then raises the opposite question of how much the batch is allowed to starve. Measured on Python 3.14 with a limiter granting 100 permits per second, a batch of 1,000 low-priority calls queued at once and interactive calls arriving at 20 per second: with a FIFO limiter the interactive calls waited a median 8.1 s, and only 96 of 200 got a permit inside the 10-second test. With strict priority, all 200 were served, with a median wait of 1.2 ms and a p99 of 10.3 ms — one permit interval — while the batch still received 797 permits. When interactive traffic rose to 90 per second, strict priority left the batch only the spare capacity, 101 permits in 10 s. Reserving every fifth permit for the batch doubled that to 200, at the cost of interactive p99 rising to 1.2 s; aging low-priority requests after 2 s finished the whole batch but pushed interactive p99 to 2.8 s. This guide builds the limiter and the policies, and shows how to choose.

Prerequisites

1. Measure the FIFO problem

A limiter that hands out permits in arrival order treats a batch job's queued calls and a user's request identically:

class FifoLimiter:
    def __init__(self, rate: float):
        self.interval = 1 / rate
        self.waiters: asyncio.Queue[asyncio.Future] = asyncio.Queue()

    async def run(self):                       # one background task hands out permits
        next_at = 0.0
        while True:
            fut = await self.waiters.get()
            if fut.done():                     # caller was cancelled while waiting
                continue
            fut.set_result(None)
            next_at = max(next_at + self.interval, time.monotonic())
            await asyncio.sleep(next_at - time.monotonic())

    async def acquire(self):
        fut = asyncio.get_running_loop().create_future()
        self.waiters.put_nowait(fut)
        await fut

Measured with 100 permits per second: a batch enqueued 1,000 calls at once, then interactive calls arrived at 20 per second for 10 seconds. Every interactive call queued behind the remaining batch. The median interactive wait was 8.1 s, the p99 9.99 s, and only 96 of the 200 interactive calls were granted before the test ended — the rest were still waiting behind batch work. The limiter was doing its job perfectly; the order was the problem.

Verify: record wait time per request class; under FIFO, the interactive wait tracks the batch backlog, not the interactive load.

100 permits/s shared by interactive calls and a batch A grid of 5 rows by 4 columns. 100 permits/s shared by interactive calls and a batch policy interactive load interactive p50 / p99 batch permits in 10 s FIFO 20/s 8.1 s / 9.99 s (96 of 200 served) 996 strict priority 20/s 1.2 ms / 10.3 ms 797 strict priority 90/s 6.2 ms / 10.7 ms 101 every 5th permit to batch 90/s 617 ms / 1.2 s 200 aging after 2 s 90/s 2.4 s / 2.8 s 300 (all done) Python 3.14, 10-second runs, batch queued at time zero.

2. Hand out permits by priority

Replace the queue with a heap ordered by priority, then by arrival, so equal-priority callers stay in order:

class PriorityLimiter:
    def __init__(self, rate: float):
        self.interval = 1 / rate
        self.heap: list[tuple[int, int, asyncio.Future]] = []
        self.seq = itertools.count()
        self.wake = asyncio.Event()

    async def run(self):
        next_at = 0.0
        while True:
            while not self.heap:
                self.wake.clear()
                await self.wake.wait()
            _, _, fut = heapq.heappop(self.heap)
            if fut.done():
                continue
            fut.set_result(None)
            next_at = max(next_at + self.interval, time.monotonic())
            await asyncio.sleep(next_at - time.monotonic())

    async def acquire(self, priority: int = 1):    # 0 = interactive, 1 = batch
        fut = asyncio.get_running_loop().create_future()
        heapq.heappush(self.heap, (priority, next(self.seq), fut))
        self.wake.set()
        await fut

The sequence number matters twice: it keeps same-priority callers first-in-first-out, and it stops heapq from ever comparing two futures, which would raise TypeError. Measured under the same load: all 200 interactive calls were served, with a median wait of 1.2 ms and a p99 of 10.3 ms — at most one permit interval, the time until the next permit. The batch still received 797 permits in the 10 seconds, the capacity left after interactive use. Total throughput was unchanged; only the order moved.

Verify: interactive p99 wait is no more than about one permit interval while a batch backlog exists.

3. Clean up cancelled waiters

A caller with a timeout gives up while waiting. Its future stays in the heap until the limiter reaches it; the if fut.done(): continue check skips it there instead of granting a permit to nobody:

async def call_api(limiter, priority, timeout):
    async with asyncio.timeout(timeout):
        await limiter.acquire(priority)        # cancellation cancels the waiter future
        return await client.get("/resource")

Cancelling the task that awaits fut cancels fut itself, so done() is true for every abandoned waiter. Without the check, each abandoned waiter would consume a permit, and a burst of timeouts would silently cut the limiter's useful rate. If callers abandon waits in large numbers, the heap also grows with dead entries until they are popped; a periodic rebuild that drops done futures bounds that.

Verify: cancel a set of waiters and confirm the limiter's granted-permit count only includes callers that actually proceeded.

How a permit reaches a caller A flow of 5 stages. How a permit reaches a caller acquire() push (priority, seq, future) Limiter task pop lowest priority first Skip futures cancelled by timeouts Grant set_result on a live future Pace sleep until next permit time One background task owns the order and the pace.

4. Decide how much starvation is acceptable

Strict priority gives low-priority work only what high-priority work leaves. With interactive traffic at 90 per second against a limit of 100, the batch received 101 permits in 10 seconds — the 10% spare — and would take 30 seconds to clear 300 calls. If interactive traffic reached 100 per second, the batch would get nothing at all. Whether that is acceptable depends on the batch: a nightly export can wait, a billing run with a deadline cannot. Two measured ways to give the batch a floor:

class SharePriorityLimiter:
    """Every `every`-th permit goes to low priority when it is waiting."""
    def __init__(self, rate: float, every: int = 5):
        self.interval, self.every, self.n = 1 / rate, every, 0
        self.hi: deque[asyncio.Future] = deque()
        self.lo: deque[asyncio.Future] = deque()
        self.wake = asyncio.Event()

    def _next_queue(self):
        self.n += 1
        if self.lo and (not self.hi or self.n % self.every == 0):
            return self.lo
        return self.hi

With every fifth permit reserved, the batch got 200 permits in 10 seconds — exactly 20% — and interactive calls, now limited to 80 per second against 90 arriving, built a backlog: p50 617 ms, p99 1.2 s. The alternative, aging, lowers a waiting request's effective priority number as it waits, so that after 2 s a batch call outranks a new interactive one; it finished all 300 batch calls within 4.8 s, and interactive p99 rose to 2.8 s. Both are honest about the same fact: when demand exceeds the limit, someone waits, and the policy only chooses who.

Verify: for each request class, write down the minimum share it must receive, and test that share at full load from the other class.

5. Choose priorities at the edge and expose them

Priority is a property of the caller, not of the HTTP call, so assign it where the work enters — a request handler is interactive, a job worker is batch — and pass it through:

INTERACTIVE, BATCH = 0, 1

async def get_profile(user_id):                     # request handler
    await limiter.acquire(INTERACTIVE)
    return await upstream.get(f"/users/{user_id}")

async def export_job(user_ids):                     # background job
    for uid in user_ids:
        await limiter.acquire(BATCH)
        await upstream.get(f"/users/{uid}")

Keep the number of priority levels small — two or three. Each level needs its own wait-time metric, and more levels make starvation analysis harder without making anything faster. Export queue depth and p99 wait per level; a growing batch depth with flat interactive waits is the policy working, while rising interactive waits mean total demand exceeds the limit and no ordering will help. For limits enforced across several processes, the same priorities can be applied per process in front of a shared limiter such as GCRA rate limiting with Redis.

Verify: dashboards show queue depth and p99 wait for each priority level separately.

Interactive p99 wait at 90/s vs batch permits per 10 s 3 horizontal bars comparing strict priority (batch 101) with the others. Interactive p99 wait at 90/s vs batch permits per 10 s strict priority (batch 101) 10.7 ms every 5th to batch (batch 200) 1,215 ms aging 2 s (batch 300, done) 2,790 ms Limit 100 permits/s; interactive demand 90/s; batch of 300 queued at time zero. A floor for the batch is paid for in interactive wait.

Verification

Prioritization under a shared limit is working when:

  • Interactive p99 wait stays near one permit interval while a batch backlog exists — 10.3 ms here at 100 permits per second.
  • Total permits granted match the limit, so prioritizing changed order, not throughput.
  • Cancelled waiters are skipped, not granted.
  • Each class's minimum share is written down and tested at full load from the other class.

Diagnostic Hook: if interactive waits rise while the limiter has a batch backlog, check the order before the limit: an interactive p50 close to the batch backlog divided by the rate — 8.1 s here — means requests are being served first-in-first-out.

Pitfalls & edge cases

  • FIFO with a batch backlog. Measured: interactive median 8.1 s.
  • Heap entries without a sequence number. Equal priorities then compare futures and raise TypeError.
  • Granting permits to cancelled waiters. Every timeout wastes a permit.
  • Strict priority with no floor. Measured: the batch got only the spare 10%.

Frequently Asked Questions

How do I prioritize requests under one rate limit in asyncio?

Keep waiters in a heap of (priority, sequence, future) and have one background task pop the lowest priority and resolve its future at the permitted rate.

Does a priority limiter reduce throughput?

No. It grants the same number of permits; it only changes the order. Interactive median wait fell from 8.1 s to 1.2 ms while the batch kept the remaining capacity.

How do I stop low-priority requests from starving?

Reserve a share, such as every fifth permit when low-priority work is waiting, or age waiting requests. Both cost interactive latency once demand exceeds the limit.

How many priority levels should a rate limiter have?

Two or three. Each level needs its own wait-time metric and minimum share, and more levels add analysis without adding capacity.