Skip to content

Implementing Retry Budgets to Prevent Retry Storms

Retries are self-limiting when a dependency is healthy and self-amplifying when it is not. In a simulation of 2,000 requests with up to three attempts each, a 5% failure rate produced 1.05 upstream calls per request; a 90% failure rate produced 2.68. The load on a struggling service therefore rises by the exact factor of the retry policy at the exact moment it can least absorb it — and because every client retries at once, the effect is synchronised. Backoff and jitter spread the retries out; they do not reduce how many there are. A retry budget does: it caps retries as a fraction of traffic, so the retry rate can never exceed what the upstream was sized for.

Prerequisites

Upstream calls per client request 3 bars comparing healthy, no budget with the others. Upstream calls per client request healthy, no budget 1.05x brownout, no budget 2.68x brownout, 10% budget 1.10x Simulation of 2,000 requests with up to three attempts each. Retries are cheap while things work and ruinous exactly when they do not.

1. Measure your own amplification first

Amplification is upstream_calls / client_requests, and it is worth instrumenting before adding any mechanism. Two counters are enough:

ATTEMPTS = Counter("upstream_attempts_total", labelnames=["upstream"])
REQUESTS = Counter("client_requests_total", labelnames=["upstream"])

The ratio is 1.0 in a healthy system and rises with the failure rate. Watch it during an incident: the moment it jumps from 1.05 to 2.7 is the moment your clients turned a degradation into an outage. Most services discover this ratio for the first time during a postmortem.

Verify: the ratio is visible on a dashboard, per upstream, and alerts above a threshold such as 1.5.

2. Implement the bucket

Every request deposits a fraction of a token; every retry withdraws a whole one. Tokens decay so the budget reflects recent traffic rather than history:

class RetryBudget:
    """Retries are allowed while the bucket has tokens; deposits come from requests."""

    def __init__(self, ratio: float = 0.1, ttl: float = 10.0):
        self.ratio, self.ttl = ratio, ttl              # 10% of traffic, 10-second memory
        self.tokens = 0.0
        self.last = time.monotonic()
        self.denied = 0

    def _decay(self) -> None:
        now = time.monotonic()
        elapsed, self.last = now - self.last, now
        self.tokens = max(0.0, self.tokens - elapsed * self.tokens / self.ttl)

    def on_request(self) -> None:
        self._decay()
        self.tokens = min(self.tokens + self.ratio, 10_000)

    def allow_retry(self) -> bool:
        self._decay()
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        self.denied += 1                               # a metric worth alerting on
        return False

The ratio is the policy: 0.1 permits one retry per ten requests, averaged over the window. The exponential decay means a service that was busy an hour ago has no credit now, which is what keeps a burst of retries after a quiet period from being funded by ancient traffic.

Verify: with no traffic, allow_retry() returns False; after 100 requests at ratio=0.1, it returns True ten times.

3. Check the budget only on retries

The first attempt is never denied — a budget is not a rate limiter for requests, and denying real traffic would turn a partial outage into a total one:

async def call_with_budget(op, budget, *, attempts=3, per_attempt=0.5):
    budget.on_request()
    last = None
    for attempt in range(attempts):
        try:
            async with asyncio.timeout(per_attempt):
                return await op()
        except RETRYABLE as exc:
            last = exc
        if attempt + 1 < attempts and not budget.allow_retry():
            raise RetriesExhausted("retry budget exhausted") from last
        await asyncio.sleep(backoff(attempt))
    raise RetriesExhausted(f"{attempts} attempts failed") from last

Only errors that are worth retrying should reach the withdrawal — see classifying retryable errors, because spending budget on a 400 Bad Request helps nobody.

Simulated against the same workload, the 10% budget left the healthy case untouched (1.05x, one retry denied out of 2,000 requests) and cut the brownout from 2.68x to 1.10x. The cost is visible too: successful requests during the brownout fell from 549 to 230. That is the trade a budget makes explicitly — fewer successes now, a dependency that can recover instead of being held down.

Verify: under healthy conditions the denial counter stays near zero; if it does not, the ratio is too low for your failure rate.

Where the budget check sits 5 stages from request arrives to attempt 2. Where the budget check sits request arrives deposit 0.1 tokens attempt 1 never denied failure is it retryable? withdraw a token or give up now attempt 2 only if funded Denied retries fail fast with the original error, which is the point.

4. See what it buys during a real degradation

The success-rate cost above assumes an upstream whose failure rate is independent of load, which is the pessimistic case. When the upstream is capacity-limited — the usual reality — retries make the failures they are responding to. Simulating 900 requests per tick against a service whose capacity drops to a fifth for three ticks:

peak upstream load requests served
no budget 2,300 per tick 15,900
10% retry budget 1,169 per tick 15,900

The budget halved the peak load and served exactly the same number of requests. The retries it denied were ones that were going to fail anyway, because they were sent into a queue that was already beyond capacity. This is the general shape: when the upstream is saturated, retry amplification produces nothing but load.

Verify: in a load test that degrades a dependency, peak upstream RPS with the budget is materially lower with no loss of successful requests.

Peak load during a three-tick upstream degradation 3 bars comparing incoming requests with the others. Peak load during a three-tick upstream degradation incoming requests 900/tick peak, no budget 2,300/tick peak, 10% budget 1,169/tick Both runs served the same 15,900 requests in total; only the peak differs. The budget halved the peak without costing a single successful request here.

5. Scope it per dependency, and pair it with a breaker

One budget per upstream, per client process. A shared budget lets a failing dependency spend the credit earned by a healthy one, which is how a single bad service starves retries for everything else.

budgets = defaultdict(lambda: RetryBudget(ratio=0.1))
budget = budgets[upstream_name]

Per-process is normally accurate enough: with many instances, each sees a representative sample of traffic and the aggregate behaves like the intended global policy. Sharing state through Redis adds a network call to every retry decision and a new dependency to the path you are trying to protect.

A budget and a circuit breaker solve adjacent problems and compose well: the budget caps amplification while the dependency is partly working, and the breaker stops calls entirely when it is not. Add a Retry-After header or a server-side load-shedding signal and the upstream can also push back directly, which is better than any client-side guess.

Verify: each upstream has its own budget and its own denial metric, and a failure in one does not raise denials for another.

Retry budget or circuit breaker? 2 columns contrasting retry budget, circuit breaker. Retry budget or circuit breaker? retry budget caps retry amplification first attempts always pass degrades smoothly no half-open probing per client process circuit breaker stops calling entirely all attempts blocked when open binary state probes to recover needs sharing to be global They compose: the budget limits amplification, the breaker stops a hopeless dependency.
The three numbers a retry budget needs A grid of 4 rows by 2 columns. The three numbers a retry budget needs parameter typical controls deposit per request 0.1 tokens the retry rate allowed: 10% token TTL 10 seconds how fast the budget forgets minimum floor 1 retry/second low-traffic clients can still retry scope per upstream one failure cannot spend another budget A budget is a rate limit on retries, not on requests: the first attempt is never denied.

Verification

A retry budget is doing its job when:

  • Amplification is instrumented, per upstream, and visible during incidents.
  • Only retries are denied: first attempts always proceed.
  • Healthy traffic sees no denials, or very few.
  • Denials spike during degradation, and peak upstream load stays near the incoming rate.
  • Budgets are per dependency, not shared across all upstreams.
  • Denial is a fast failure that chains the original error as its cause.

Pitfalls & edge cases

  • Applying the budget to first attempts. That is request rate limiting; it denies traffic that would have succeeded.
  • A ratio below the normal failure rate. If 5% of calls genuinely need a retry, a 2% budget denies real work continuously; set the ratio above the healthy failure rate with margin.
  • No floor for low-traffic clients. A client sending one request per minute never accumulates a token; add a small minimum retry rate so it is not permanently denied.
  • Budget without backoff. Ten allowed retries fired simultaneously still spike the upstream; the budget caps volume, jitter spreads it.
  • Forgetting the decay. A bucket that only accumulates eventually funds an unlimited retry storm after a quiet period.
  • Counting hedged requests as retries. Hedging is amplification too, and should draw on the same budget.

Frequently Asked Questions

What is a retry budget?

A cap on retries expressed as a fraction of traffic rather than a per-request attempt count. Each request deposits a fraction of a token into a decaying bucket and each retry withdraws one, so retries can never exceed — for a 10% budget — about one in ten requests, no matter how many are failing.

Why isn't exponential backoff enough to prevent retry storms?

Backoff changes when retries arrive, not how many. With three attempts and a 90% failure rate, the simulated upstream still received 2.68 calls per client request, just spread out. A budget reduces the count itself, cutting the same brownout to 1.10 calls per request.

How do I pick the retry budget ratio?

Start above your healthy failure rate with margin — 10% is a common default when normal failures are about 1%. Then watch the denial metric: near-zero denials in healthy operation mean the ratio is workable, and constant denials mean it is too tight for your real error rate.

Should the retry budget be shared across processes?

Usually not. Per-process budgets are simpler and, with several instances each seeing representative traffic, approximate the intended global policy well. Sharing state through Redis adds a network call to every retry decision and a dependency on the path you are protecting.

Do retry budgets replace circuit breakers?

No, they complement each other. A budget caps amplification while a dependency is partly working and lets successful traffic through; a breaker stops calling a dependency that is failing outright and probes for recovery. Most resilient clients run both, plus per-attempt timeouts.