Skip to content

Propagating Deadlines Across Async Service Calls

Three services, each with a sensible five-second timeout, produce a fifteen-second request. Worse, the client gave up after two — so ten of those seconds were spent computing a response nobody would read, holding connections and database locks the whole time. This is what per-call timeouts do when they are configured independently: they bound each hop and nothing else, and the total grows with the depth of the call graph.

A deadline fixes it by inverting the model. Instead of "each call gets five seconds", the rule becomes "this request expires at 14:32:07.412, and every hop inherits what remains". Work that cannot finish in the remaining time is never started; a downstream service that receives a 200 ms budget can refuse immediately rather than doing four seconds of doomed work. This guide implements that with contextvars, propagates it across process boundaries, and covers the clock-skew and retry subtleties that trip up the naive version.

Prerequisites

Independent timeouts versus one deadline Two columns contrasting additive timeouts with a propagated deadline. Independent timeouts versus one deadline timeout per hop adds up Gateway 5 s, orders 5 s, stock 5 s Worst case 15 s Client gave up at 2 s Work continues unread one deadline bounded Client sets 2 s at the edge Each hop passes what remains Deepest hop may refuse early Total never exceeds 2 s The deadline is set once, at the boundary that knows what the caller wants.

1. Understand why per-call timeouts do not compose

Each hop's timeout bounds only that hop, so the worst case is their sum.

gateway   timeout=5s  ─┐
  orders  timeout=5s  ─┼─ worst case 15s, client patience 2s
    stock timeout=5s  ─┘

deadline model:
  client sets 2s  →  gateway has 2.0s  →  orders gets 1.85s  →  stock gets 1.6s
                     (each hop subtracts what it has already spent)

The deadline model is strictly better in three ways: total latency is bounded by what the client asked for, downstream services can reject work they cannot finish, and any hop can report why it stopped rather than a generic timeout. Verify: add a deliberate 3 s delay at the deepest service; the client should fail at its own budget, and the deepest service should never start the work.

2. Carry the deadline in a ContextVar

contextvars propagate automatically into tasks created from the current context, which is exactly the scope a request needs.

import asyncio
import contextvars
import time

_deadline: contextvars.ContextVar[float | None] = contextvars.ContextVar(
    "deadline", default=None)


def set_budget(seconds: float) -> None:
    """Called once at the request boundary."""
    _deadline.set(time.monotonic() + seconds)


def remaining(reserve: float = 0.02) -> float:
    """Seconds left for the next call, minus a small reserve for responding."""
    deadline = _deadline.get()
    if deadline is None:
        return float("inf")
    return max(deadline - time.monotonic() - reserve, 0.0)

Storing an absolute instant rather than a duration is what makes it composable: every reader computes its own remaining time, so nothing has to be threaded through call signatures. Use time.monotonic() so an NTP correction mid-request cannot move the deadline. Verify: call remaining() at three depths of the call chain; the values should decrease monotonically.

3. Apply it at every call site

asyncio.timeout_at() takes an absolute deadline on the loop's clock, which is exactly what a shared budget is.

import asyncio


class DeadlineExceeded(TimeoutError):
    pass


async def call_with_budget(coro, *, want: float, label: str):
    left = remaining()
    if left <= 0:
        raise DeadlineExceeded(f"{label}: no budget left before starting")
    async with asyncio.timeout(min(want, left)):      # whichever is tighter
        return await coro

Two rules make this work. Take the minimum of the call's own sensible timeout and what remains, so a fast dependency is still bounded tightly. And check for an already-exhausted budget before starting: beginning a call with zero time left burns a connection and a downstream request for a result that will be discarded. Verify: with a budget of 500 ms and three sequential 300 ms calls, the third should raise DeadlineExceeded without issuing a request.

What each hop actually gets Three lanes showing a deadline budget shrinking across service hops. What each hop actually gets gateway 2000 ms budget set here orders 1850 ms — minus what gateway spent stock 1600 ms — and refuses below a floor t = 0 … 2000 ms Every hop subtracts its own elapsed time; nobody hands out more than it has.

4. Propagate across the network

Inside a process the ContextVar is enough; between services the remaining time has to travel on the wire.

import time
import httpx


async def call_downstream(client: httpx.AsyncClient, url: str) -> httpx.Response:
    left = remaining()
    headers = {"X-Request-Timeout-Ms": str(int(left * 1000))}    # relative, not absolute
    async with asyncio.timeout(left):
        return await client.get(url, headers=headers)


# On the receiving side, at the request boundary:
async def inbound_middleware(request, call_next):
    header = request.headers.get("X-Request-Timeout-Ms")
    budget = min(float(header) / 1000, MAX_BUDGET) if header else DEFAULT_BUDGET
    set_budget(budget)
    return await call_next(request)

Send a relative remaining time, not an absolute timestamp: clocks between hosts differ by milliseconds to seconds, and an absolute deadline interpreted against a skewed clock either expires instantly or never. Always clamp the incoming value to your own maximum, since a caller can otherwise ask you to hold resources for an hour. gRPC standardises this as the grpc-timeout header and its Deadline API; over plain HTTP the header name is yours to choose but the semantics should match. Verify: log the received budget at each hop; it should shrink by roughly the network latency plus each hop's own work.

5. Handle retries and background work correctly

Two cases need explicit decisions or the model breaks.

import asyncio


async def with_retries(factory, attempts: int = 3):
    """Retries share the budget — they do not each get a fresh one."""
    last: Exception | None = None
    for _ in range(attempts):
        left = remaining()
        if left <= 0.05:                       # not enough time for another try
            break
        try:
            async with asyncio.timeout(left):
                return await factory()
        except TimeoutError as exc:
            last = exc
    raise DeadlineExceeded("budget exhausted across retries") from last


async def detached(coro) -> None:
    """Work that must outlive the request needs its OWN budget."""
    ctx = contextvars.copy_context()
    ctx.run(_deadline.set, None)               # clear the inherited deadline
    spawn(coro)                                # see the background-task pattern

Retries inside a deadline share the budget, which is the correct behaviour — three attempts of a call that has already used most of the time should not extend the request. Background work is the opposite: it inherits the ContextVar automatically, so a fire-and-forget audit write would be cancelled the moment the request's deadline passes. Clear or replace the deadline for anything that must outlive the request. Verify: a background task spawned near the end of a request still completes after the request returns.

Sending the deadline downstream A decision diamond on deadline encoding with three outcome cards. Sending the deadline downstream Timestamp or remaining duration? absolute timestamp breaks on clock skew expires instantly or never remaining milliseconds correct what gRPC standardised nothing at all the chain is unbounded again each hop invents its own Clocks between hosts disagree; elapsed time does not.

Verification

  • Total latency is bounded by the client's budget. No request outlives the deadline it was given, regardless of call-graph depth.
  • Doomed work is never started. Under an injected slow dependency, the deepest service logs "no budget left" rather than doing the work.
  • Budgets shrink monotonically. Logged remaining time decreases at every hop, and no hop receives more than its caller had.

Pitfalls and edge cases

  • Sending an absolute deadline over the wire. Clock skew makes it expire immediately or never; send remaining milliseconds.
  • Not clamping the received budget. A caller can otherwise pin your resources for an arbitrary duration.
  • Giving each retry a fresh budget. Three attempts then take three times the deadline; retries share what remains.
  • Inheriting the deadline in background tasks. ContextVars propagate into created tasks, so detached work dies with the request unless you clear it.
  • No reserve for the response. Consuming the entire budget in the last call leaves nothing to serialise and send the reply.
  • Using time.time(). A clock adjustment mid-request moves the deadline; use time.monotonic().
  • A deadline without a floor. Passing 5 ms downstream guarantees failure; refuse below a minimum and fall back instead.

Frequently Asked Questions

Why do per-call timeouts not add up to a request timeout?

Because each one bounds only its own call. A chain of three hops with five-second timeouts has a fifteen-second worst case, and none of the hops knows how much of the client's patience is already spent. A deadline inverts this: one absolute expiry is set at the boundary and every hop uses whatever remains, so total latency is bounded by what the client asked for.

How do I propagate a deadline through async calls in Python?

Store the absolute expiry in a ContextVar at the request boundary, and compute the remaining time at each call site. ContextVars propagate automatically into tasks created from that context, so no signatures change. Apply the remaining time with asyncio.timeout, taking the minimum of it and the call's own sensible bound.

Should a deadline be sent as a timestamp or a duration between services?

As a remaining duration, in milliseconds. Absolute timestamps depend on synchronised clocks, and even modest skew makes a deadline expire instantly or never. This is why gRPC's grpc-timeout header carries a relative value; over HTTP, use your own header with the same semantics and clamp what you receive.

Do retries get their own deadline?

No — retries share the request's remaining budget. Check the remaining time before each attempt and stop when there is not enough left for a meaningful try. Giving each attempt a fresh timeout multiplies the request's worst case by the retry count, which is exactly what the deadline model exists to prevent.