Skip to content

Load Shedding When the Event Loop Is Overloaded

An asyncio service under overload does not fall over; it accepts everything and serves almost nothing. Because create_task always succeeds and queues are often unbounded, work piles up, every request's latency grows by the depth of the queue, and the client times out long before its response is ready — at which point the server finishes the work anyway and sends it to a connection nobody is reading. In a simulation of 4,000 requests per second offered to a service with capacity for about 1,600, that pattern produced 308 useful requests per second and 7,016 responses nobody was waiting for. Adding one rule — reject any request that has already waited 50 ms for a slot — raised useful throughput to 1,556 per second and cut p99 from 245 ms to 55 ms. This guide builds that rule and the signals behind it.

Prerequisites

  • Python 3.11+. The mechanism is a timestamp and a comparison; no libraries required.
  • Bounded concurrency from Worker Pool Implementations — shedding needs a queue to measure, which means a limit.
  • Loop lag measurement from health and readiness probes, the complementary signal.
Goodput under 2.5x overload 2 bars comparing no shedding with the others. Goodput under 2.5x overload no shedding 308 rps shed after 50 ms queued 1,556 rps 4,000 requests per second offered to a service whose capacity is about 1,600. Five times the useful throughput, from rejecting work nobody was still waiting for.

1. Understand what overload actually costs

The failure is not that requests are slow. It is that the service spends its capacity on requests whose callers have gone. In the unshed run:

no shedding shed after 50 ms queued
offered 7,640 7,640
served within the client timeout 616 3,112
shed 0 4,528
answered too late 7,024 0
work done for a departed caller 7,016 0
goodput 308 rps 1,556 rps

Without shedding, 92% of the service's capacity went into responses that arrived after the client had given up. The capacity was there; it was spent on work with no value. Shedding does not create capacity, it stops wasting it — which is why the shed run's goodput is close to the service's actual capacity of about 1,600 rps.

The clients see a difference too: a fast 503 at 50 ms instead of a timeout at 250 ms, which lets them fail over, retry elsewhere, or degrade immediately.

Verify: instrument "responses completed after the client timeout" — a non-zero value is wasted capacity.

2. Shed on queue wait time

The signal that matters is how long a request has already waited. Stamp the arrival, and check after acquiring a slot:

async def handle(request):
    arrived = time.perf_counter()
    async with capacity:                               # a Semaphore or pool slot
        waited = time.perf_counter() - arrived
        if waited > SHED_THRESHOLD:                    # 50 ms in the measurements
            SHED.inc()
            return Response(status_code=503, headers={"retry-after": "1"})
        return await do_work(request)

Checking after the wait rather than before is the important detail. Before the wait you are guessing at the queue depth; after it you know exactly how long this request has been queued, which is the only number that predicts whether the caller is still there.

The threshold comes from the client's timeout: set it to a fraction — a fifth to a half — of the deadline, leaving room for the work itself plus the response. If callers send a deadline, subtract instead of guessing:

        if deadline is not None and deadline - time.time() < ESTIMATED_SERVICE_TIME:
            return Response(status_code=503)           # cannot finish in time; do not start

That is the ideal version, and propagating deadlines with contextvars covers carrying the value. Without a deadline, a fixed threshold gets most of the benefit.

Verify: under synthetic overload, p99 latency settles near the threshold rather than climbing with the queue.

Where the shed decision belongs 5 stages from stamp arrival to serve if fresh. Where the shed decision belongs stamp arrival one perf_counter call wait for a slot bounded concurrency check the wait over the threshold? reject if stale 503, before any work serve if fresh within the budget Rejecting after the wait, not before it, is what makes the decision accurate.

3. Do not rely on loop lag alone

Event-loop lag is a genuine overload signal — a handler that blocked for 0.6 s produced 587 ms of measured lag in the readiness probe — but it detects a blocked loop, not a full queue. In the overload run above, the work was I/O-bound, the loop was never blocked, and a lag-based shedding policy shed nothing at all while the service degraded to 308 rps.

Both signals are worth having, because they catch different failures:

  • Queue wait time catches saturation of any bounded resource — worker slots, connection pools, semaphores — which is the common case.
  • Loop lag catches CPU work or a blocking call on the loop thread, which queue wait attributes to the wrong place.
  • In-flight count is a cheap proxy for both, but says nothing about how long anything has waited.

Shed on queue wait; alert on lag.

Verify: run an I/O-bound overload and confirm your lag metric stays flat — then check what your shedding policy did.

Which signal detects which kind of overload A grid of 4 rows by 2 columns. Which signal detects which kind of overload signal detects blind to queue wait time saturation of any bounded pool nothing: it is the direct measure event loop lag CPU work blocking the loop queueing behind a semaphore in-flight count concurrency above a limit how long each has waited upstream errors a failing dependency local overload Measured: with the loop never blocked, a lag-based policy shed nothing while the service collapsed.

4. Choose what to shed, not just when

Uniform shedding is a large improvement over none, and priority-aware shedding is a large improvement over uniform. The rule of thumb is to shed work whose caller has the most alternatives:

  1. Retries and hedges first. The caller has an explicit plan for failure, and shedding a retry directly counters the amplification described in retry budgets.
  2. Background, batch and prefetch next. It can run in ten minutes without anyone noticing.
  3. Interactive first attempts last. This is the traffic the service exists for.
THRESHOLDS = {"background": 0.005, "retry": 0.010, "interactive": 0.050}
if waited > THRESHOLDS.get(request_class(request), 0.050):
    return Response(status_code=503)

Different thresholds per class give you graceful degradation for free: background work stops almost immediately, retries stop soon after, and interactive traffic keeps the whole remaining capacity. Classification usually comes from a header the callers already send, or from the route.

Verify: under overload, background traffic is shed before interactive traffic, per your metrics.

What should be shed first? A decision on What class is this request with 3 outcomes. What should be shed first? What class is this request? a retry or a hedge shed first the caller already has a plan background or batch shed next it can run later interactive, first attempt shed last this is the work that matters Shedding uniformly is better than not shedding, but priority classes are much better.

5. Make shedding visible and honest

A shed request is a failed request from the caller's perspective, so it must be reported as such rather than hidden:

  • Return 503 with Retry-After, not 500. The distinction tells clients and proxies that this is back-pressure rather than a bug, and Retry-After lets you spread the retries — see classifying retryable errors.
  • Count sheds separately from errors, labelled by request class and by the reason.
  • Do not log one line per shed. At 4,528 sheds in two seconds, logging is its own overload; count them and log a summary periodically.
  • Feed readiness, if the instance is shedding heavily: reporting not-ready removes it from the load balancer and lets the rest of the fleet absorb the traffic, which is the fleet-level version of the same idea.

Finally, test it. Load shedding is code that only runs under conditions you never see in staging by accident, and an untested shed path is as likely to throw as to help. A load test that pushes past capacity, in CI, is the only way to know the numbers above apply to your service.

Verify: a load test above capacity produces 503s with Retry-After, flat p99, and no log flood.

Latency of the requests that were served 4 bars comparing p50, no shedding with the others. Latency of the requests that were served p50, no shedding 131.7 ms p99, no shedding 245.0 ms p50, shedding 50.0 ms p99, shedding 55.3 ms Same run as the goodput figure; the client timeout was 250 ms. Shedding bounds latency at the threshold you chose rather than at the client timeout.

Verification

Load shedding works when:

  • Goodput stays near capacity as offered load rises past it.
  • p99 settles near the shed threshold, not at the client timeout.
  • No work completes after its caller's deadline — the wasted-work counter is zero.
  • Shed responses are 503 with Retry-After, counted and not logged per request.
  • Priorities are honoured: background and retry traffic sheds before interactive.
  • The path is load-tested, not only reasoned about.

Pitfalls & edge cases

  • Unbounded queues. Without a limit there is no wait to measure and nothing to shed; bounded concurrency is a prerequisite.
  • Shedding before the wait. Deciding from queue depth rather than measured wait misjudges bursts in both directions.
  • A threshold near the client timeout. By then the work is already wasted; shed at a fraction of the deadline.
  • Shedding uniformly at the front door. A cheap health check and an expensive report are not interchangeable; classify.
  • Logging every shed. The logging itself becomes the bottleneck at exactly the wrong moment.
  • Clients that retry immediately. Shedding without Retry-After and a client-side budget turns rejections into more load.

Frequently Asked Questions

What is load shedding in an async service?

Rejecting requests the service cannot serve in time, so that capacity goes to requests it can. In the simulation here, shedding anything that had waited more than 50 ms for a slot raised useful throughput from 308 to 1,556 requests per second and cut p99 latency from 245 ms to 55 ms.

How do I detect that an asyncio service is overloaded?

Measure how long requests wait for a bounded resource — a semaphore slot, a pool connection — by stamping arrival and checking after acquisition. Event-loop lag is a complementary signal that catches blocking work on the loop, but it stays flat during I/O-bound queueing, where it shed nothing in the measured run.

Should I shed load based on event loop lag?

Only as one signal among several. Lag detects CPU work or blocking calls on the loop thread; it does not detect requests queued behind a full connection pool. Shed on measured queue wait time, and use lag for alerting and for readiness.

What status code should a shed request return?

503 Service Unavailable with a Retry-After header. It signals back-pressure rather than a bug, so clients and proxies treat it as retryable, and Retry-After lets you spread the retries rather than receiving them all at once.

What should be shed first under overload?

Retries and hedged requests, because the caller already has a failure plan; then background, batch and prefetch work; and interactive first attempts last. Different wait thresholds per class give graceful degradation without any extra machinery.