Skip to content

Setting Per-Attempt and Total Timeouts for Retries

Retry loops are written twice. The first version wraps the whole thing in one timeout, which produces a client that makes exactly one attempt against a hung server and then gives up — measured below: 1 attempt in 1.00 s where the retry policy promised five. The second version puts a timeout on each attempt, which retries properly but can run for the sum of every attempt and every backoff, blowing far past whatever the caller was willing to wait. Both timeouts are necessary, and they interact: the per-attempt cap must shrink as the budget runs out, or the last attempt outlives the deadline it belongs to. This guide builds a retry helper with both, verified against a failing server, a hung server and a server that never recovers.

Prerequisites

  • Python 3.11+ for asyncio.timeout(), the context-manager form used throughout.
  • Retry mechanics from Retry & Backoff Strategies — this page is about bounding a retry loop, not about when to retry.
  • Timeout semantics from Timeouts & Deadlines, particularly that asyncio.timeout cancels the block and raises TimeoutError at its own boundary.
Four timeouts, four different jobs A grid of 4 rows by 2 columns. Four timeouts, four different jobs timeout bounds without it connect reaching the server a black-holed SYN hangs for minutes per attempt one call end to end one hung attempt eats the budget backoff cap the wait between tries exponential growth outlives the caller total budget everything, including waits the caller times out first Per-attempt and total are the pair most code is missing; the others usually have defaults.

1. See what a single total timeout does

Start with the version most codebases have — a retry loop wrapped in one timeout:

async with asyncio.timeout(1.0):
    for _ in range(5):
        try:
            return await call_upstream()
        except ConnectionError:
            continue

Against a server that accepts the connection and never replies, this made one attempt and raised after 1.00 s. The retry loop is decorative: the first attempt consumes the whole budget, because nothing bounds an individual call. Every "we retry three times" runbook that is not backed by a per-attempt timeout is describing behaviour the code cannot produce.

The failure mode matters because hanging is common. A dropped packet on an established connection, an overloaded server that accepted your request and stopped scheduling it, a load balancer holding a connection to a dead backend — all present as "no response, no error".

Verify: count the attempts your retry loop actually makes against a server that never responds.

2. Add a per-attempt timeout

Bound each attempt individually, and retries begin to function:

for attempt in range(attempts):
    try:
        async with asyncio.timeout(per_attempt):
            return await call_upstream()
    except (TimeoutError, ConnectionError) as exc:
        last = exc
    await asyncio.sleep(backoff(attempt))

With per_attempt=0.2 inside the same one-second window the client made 4 attempts instead of one. Pick the value from the upstream's latency distribution, not from a round number: something above p99 and well below the caller's patience. Too tight and you abandon requests that would have succeeded, adding load while lowering the success rate; too loose and you are back to one attempt.

A per-attempt timeout is also what makes hedging possible later: both techniques depend on knowing when an attempt has taken too long to be worth waiting for.

Verify: the attempt count rises to roughly budget / (per_attempt + backoff) against a hung server.

Attempts made in one second against a hung server 2 bars comparing per-attempt 200 ms + total 1 s with the others. Attempts made in one second against a hung server per-attempt 200 ms + total 1 s 4 attempts total 1 s only 1 attempt Measured against a server that accepts the request and never answers. Without a per-attempt timeout the retry loop never gets to retry.

3. Bound the whole call with a deadline

The per-attempt timeout alone permits attempts × (per_attempt + backoff) of total runtime — with five attempts, a 2-second cap and growing backoff, over fifteen seconds for a call the caller abandoned after three. Track a deadline on the loop's clock and check it before each attempt:

async def retry(op, *, attempts=5, per_attempt=0.2, total=1.0, base=0.05):
    loop = asyncio.get_running_loop()
    deadline = loop.time() + total                     # monotonic, not wall clock
    last: Exception | None = None
    for attempt in range(attempts):
        remaining = deadline - loop.time()
        if remaining <= 0:
            raise TimeoutError(f"budget exhausted after {attempt} attempts") from last
        try:
            async with asyncio.timeout(min(per_attempt, remaining)):
                return await op()                      # the attempt cannot outlive the budget
        except (TimeoutError, ConnectionError) as exc:
            last = exc
        delay = min(base * 2 ** attempt, max(0.0, deadline - loop.time()))
        await asyncio.sleep(delay * (0.5 + random.random() / 2))
    raise TimeoutError(f"{attempts} attempts failed") from last

min(per_attempt, remaining) is the line that keeps the promise: the final attempt is clipped to whatever the budget has left. The backoff is clipped the same way, so the loop never sleeps past its own deadline. Use loop.time() — a monotonic clock — rather than time.time(), which can jump backwards and turn a one-second budget into a much longer one.

Measured behaviour across three scenarios:

scenario attempts elapsed outcome
transient failure, recovers 3 0.26 s success
server hangs forever 4 1.00 s budget exhausted after 4 attempts
server always fails, 20 attempts allowed 4 0.80 s budget stopped it, not the attempt count

The third row is the useful one: the attempt count is a safety net, and the budget is what actually governs.

Verify: elapsed time never exceeds the total, whatever the attempt count and backoff schedule.

Four attempts inside one deadline 3 lanes over time. Four attempts inside one deadline attempt try 1 try 2 try 3 try 4 (clipped) backoff total budget one deadline for the whole call time → The last attempt is shortened to what the budget has left, never beyond it.

4. Raise a failure the caller can distinguish

TimeoutError from a per-attempt timeout and TimeoutError from the budget mean different things — "this attempt was slow" versus "we ran out of time" — and callers above you act on that difference. Chaining with raise ... from last preserves the cause, so the traceback shows the last real failure underneath the budget exhaustion:

TimeoutError: budget exhausted after 4 attempts

The above exception was the direct cause of:
ConnectionError: upstream down

For a client library, a dedicated exception type carrying the attempt count and elapsed time is better still: it lets a caller retry at a higher level for a budget exhaustion but not for a deterministic failure. Whatever you raise, do not swallow the cause — "the request timed out" without the underlying ConnectionError sends the next person debugging it to the wrong subsystem.

Verify: the raised exception's __cause__ is the last attempt's failure, not None.

5. Derive the budget from the caller

A budget invented in isolation is wrong twice: too long and you hold resources for a caller that has already given up, too short and you fail requests that had time left. Where a deadline arrives with the request — a gRPC deadline, an X-Request-Deadline header, a parent asyncio.timeout — derive from it:

remaining = deadline_var.get() - loop.time() if deadline_var.get() else DEFAULT_BUDGET
budget = max(0.0, remaining - SAFETY_MARGIN)           # leave time to send the response

Propagating deadlines with contextvars covers carrying that value down a call stack without threading it through every signature. The safety margin exists so the service can answer the caller with a proper error rather than being cancelled mid-response.

When no caller deadline exists, choose from the workload: an interactive request gets the latency target, typically 1–3 seconds; a background job gets seconds and leans on the queue to retry later, because a worker blocking for minutes is capacity removed from the pool.

Verify: a request arriving with 200 ms left makes at most one short attempt rather than a full retry schedule.

What should the total budget be? A decision on Who is waiting for this call with 3 outcomes. What should the total budget be? Who is waiting for this call? a caller with a deadline what remains of it minus a safety margin an interactive user your latency target typically 1-3 seconds a background worker seconds, not minutes let the queue retry instead A budget longer than the caller will wait is a budget that only wastes capacity.

Verification

A bounded retry loop is correct when:

  • Attempts are individually capped, so a hung upstream is abandoned and retried.
  • The total is enforced on a monotonic clock, and elapsed time never exceeds it.
  • The last attempt is clipped to the remaining budget rather than the nominal per-attempt value.
  • Backoff counts against the budget, including its jitter.
  • The raised error carries the cause, and distinguishes budget exhaustion from a deterministic failure.
  • The budget comes from the caller when the caller supplies one.

Pitfalls & edge cases

  • Retrying non-idempotent work after a timeout. A timeout does not mean the server did nothing; it may have committed. Use idempotency keys, as in classifying retryable errors.
  • Nested budgets that multiply. Three layers each retrying three times is 27 upstream calls. Retry at one layer, and propagate the deadline down the others.
  • Catching TimeoutError outside the async with. asyncio.timeout converts the cancellation into TimeoutError at its own boundary; catching it further out mixes it with other timeouts. Catch at the block.
  • Sleeping past the deadline. Exponential backoff outgrows short budgets quickly; clip the sleep against the remaining time.
  • time.time() for deadlines. NTP steps and suspends make wall-clock arithmetic unreliable; loop.time() is monotonic.
  • Timeouts without a circuit breaker. A permanently failing dependency will absorb the full budget of every request; pair this with circuit breakers and bulkheads.

Frequently Asked Questions

Why does my retry loop only make one attempt?

Because the only timeout is around the whole loop. A hung upstream consumes the entire budget in the first attempt, leaving nothing for the rest. Put asyncio.timeout(per_attempt) around each individual call, and keep the total as a separate deadline.

How do I set a total timeout for a retry loop in asyncio?

Record deadline = loop.time() + total before the loop, and before each attempt check the remaining time, cap the attempt with asyncio.timeout(min(per_attempt, remaining)), and clip the backoff sleep to what is left. Use loop.time(), which is monotonic, rather than time.time().

What is a good per-attempt timeout?

Somewhere above the upstream's p99 latency and well below the caller's patience — commonly two to three times the median. Too tight and you abandon requests that would have succeeded while adding load; too loose and the retry loop cannot fit a second attempt into the budget.

Should the backoff sleep count against the total timeout?

Yes. The caller is waiting during the sleep just as during the call. Clip each sleep to the remaining budget, and skip the final sleep entirely when no time is left for another attempt.

How do I tell budget exhaustion apart from a single slow call?

Raise a distinct error for the budget — or at minimum use raise TimeoutError(...) from last so the last real failure is chained as the cause. Callers can then retry at a higher level for a budget exhaustion while treating a deterministic failure as final.