Skip to content

Adding Timeouts and Fallbacks to Degraded Dependencies

A circuit breaker turns a slow failure into a fast one. That is progress, but on its own it just means users get an error sooner. The resilience payoff comes from what you serve instead: yesterday's price from cache, a product page without the recommendations rail, a write accepted into a queue for later. That is degradation — deliberately reduced functionality in exchange for staying up — and it has to be designed per dependency, because the right answer for a pricing service ("serve stale, marked") is the wrong answer for a payment authorisation ("fail loudly").

This guide covers the mechanics: allocating a deadline budget across the dependencies of one request, choosing a fallback per dependency class, serving stale data with honest bounds, composing partial responses, hedging against tail latency, and measuring degradation so "we are running degraded" is a number rather than a hunch.

Prerequisites

One 2 s budget, sliced across three calls Three lanes showing a request budget sliced across dependency calls. One 2 s budget, sliced across three calls budget 2000 ms of client patience calls profile: 800 ms slice pricing: 500 ms recs: what remains reserve render the response t = 0 … 2000 ms Each slice comes from what is left, so four dependencies cannot mean four timeouts.

1. Give the whole request one deadline budget

A request that calls four dependencies with a 2-second timeout each can take eight seconds. Budget from the outside in.

import asyncio
import time


class Deadline:
    """One budget for the whole request, shared by every dependency call."""

    def __init__(self, budget: float) -> None:
        self.expires_at = time.monotonic() + budget

    @property
    def remaining(self) -> float:
        return max(self.expires_at - time.monotonic(), 0.0)

    def slice(self, want: float, reserve: float = 0.05) -> float:
        """A per-call timeout that cannot outlive the request's own deadline."""
        return max(min(want, self.remaining - reserve), 0.0)


async def handle(request) -> dict:
    deadline = Deadline(budget=2.0)             # the client's patience
    profile = await profiles.call(deadline.slice(0.8))
    prices = await pricing.call(deadline.slice(0.5))
    return render(profile, prices)

The reserve keeps a few milliseconds for serialising the response, so the last call cannot consume the entire remainder and leave nothing for actually replying. Anything that would get a zero-length slice should skip straight to its fallback — attempting a call you have no time to receive is pure waste. Verify: with an artificially slow first dependency, the total handler time stays under the budget and the later calls degrade rather than overrun.

2. Choose the fallback by dependency class

There are four honest answers, and picking the wrong one is worse than having none.

# (a) OPTIONAL — omit the section entirely
recommendations = await safe(recs.fetch(user), fallback=lambda: [])

# (b) CACHEABLE — serve stale, marked as such
price = await safe(pricing.fetch(sku), fallback=lambda: cache.get_stale(sku))

# (c) DEFERRABLE — accept now, apply later
await safe(analytics.record(event), fallback=lambda: outbox.enqueue(event))

# (d) CRITICAL — no fallback exists; fail cleanly and fast
auth = await payments.authorise(order)          # let the error propagate

Classify every dependency in the request path before writing any code, because the classification is a product decision, not an engineering one. The dangerous case is (d) dressed as (b): serving a stale authorisation or a cached balance is not degradation, it is incorrectness. Verify: for each dependency, write down which class it is and confirm the code matches; anything unclassified is an outage waiting to be explained.

3. Serve stale data with an explicit bound

"Stale" without a limit becomes "wrong". Attach an age and refuse to serve beyond it.

import time
from dataclasses import dataclass


@dataclass
class Cached:
    value: dict
    stored_at: float

    def age(self) -> float:
        return time.monotonic() - self.stored_at


class StaleCache:
    def __init__(self, fresh_for: float = 60.0, stale_for: float = 900.0) -> None:
        self.fresh_for, self.stale_for = fresh_for, stale_for
        self._items: dict[str, Cached] = {}

    def put(self, key: str, value: dict) -> None:
        self._items[key] = Cached(value, time.monotonic())

    def get(self, key: str, *, allow_stale: bool) -> tuple[dict | None, bool]:
        item = self._items.get(key)
        if item is None:
            return None, False
        if item.age() <= self.fresh_for:
            return item.value, False                       # fresh
        if allow_stale and item.age() <= self.stale_for:
            return item.value, True                        # stale but usable
        return None, False                                 # too old to serve

Two bounds, two meanings: fresh_for is when you would rather re-fetch, and stale_for is the point past which stale data is misleading enough to be worse than an error. Return the staleness flag so the caller can mark the response — a degraded: true field, or a Warning header — because a user who knows the price may be a few minutes old is served, while one who does not is misled. Verify: with the dependency down for longer than stale_for, the endpoint starts failing rather than serving hours-old data.

The four honest fallbacks Four-row matrix of dependency classes and their fallbacks. The four honest fallbacks fallback example optional omit the section recommendations rail cacheable serve stale, marked catalogue text, prices deferrable queue and apply later analytics, audit events critical fail fast, no fallback payment authorisation Classify before coding: a critical dependency dressed as cacheable is incorrectness.

4. Compose partial responses concurrently

When several optional sections can degrade independently, gather them with individual timeouts and assemble whatever arrived.

import asyncio


async def optional(name: str, coro, timeout: float, default):
    try:
        async with asyncio.timeout(timeout):
            return name, await coro, False
    except (TimeoutError, ConnectionError, CircuitOpen):
        DEGRADED[name] += 1
        return name, default, True


async def build_page(user_id: str, deadline: Deadline) -> dict:
    budget = deadline.slice(0.6)
    results = await asyncio.gather(
        optional("profile", profiles.fetch(user_id), budget, {}),
        optional("orders", orders.recent(user_id), budget, []),
        optional("recs", recs.fetch(user_id), budget, []),
    )
    page = {name: value for name, value, _ in results}
    page["degraded_sections"] = [name for name, _v, degraded in results if degraded]
    return page

gather rather than TaskGroup is deliberate here: a TaskGroup cancels its siblings when one child fails, which is the opposite of what a partial response needs. Each section carries its own timeout and its own default, so one slow section costs the page its own content and nothing else. Verify: stub one section to hang; the page returns within the budget with that section listed in degraded_sections and the others populated.

5. Hedge against tail latency, carefully

When a dependency is usually fast but occasionally very slow, a second request after a short delay converts a p99 problem into a p99.9 one.

import asyncio


async def hedged(factory, hedge_after: float = 0.05, timeout: float = 0.5):
    """Send a second attempt if the first has not answered by `hedge_after`."""
    first = asyncio.create_task(factory())
    done, _pending = await asyncio.wait({first}, timeout=hedge_after)
    if done:
        return first.result()
    second = asyncio.create_task(factory())
    try:
        done, pending = await asyncio.wait({first, second}, timeout=timeout,
                                           return_when=asyncio.FIRST_COMPLETED)
        if not done:
            raise TimeoutError("both hedged attempts timed out")
        for task in pending:
            task.cancel()                        # stop the loser immediately
        return next(iter(done)).result()
    finally:
        for task in (first, second):
            if not task.done():
                task.cancel()

Hedging is only safe for idempotent reads, and only worth it when the latency distribution has a heavy tail — set hedge_after near the p95 so hedged requests stay a small percentage of traffic. Hedging a dependency that is uniformly slow simply doubles its load at the worst possible moment, which is why the breaker must sit outside this. Verify: with 5% of calls artificially slow, hedging should cut p99 substantially while raising total request volume by roughly 5%.

6. Make degradation visible

A fallback that nobody counts is indistinguishable from a bug report.

# Per dependency, per outcome:
#   dependency_calls_total{name,outcome="ok|timeout|open|shed"}
#   dependency_fallback_total{name,kind="stale|empty|queued"}
#   response_degraded_ratio = degraded_responses / total_responses

The ratio of degraded responses is the number to put on a dashboard and in an SLO, because it is what users experience — a service can be "up" by every liveness measure while serving 40% of its pages without prices. Include the degraded flag in the response body or headers so client-side telemetry and support tickets can corroborate it. Verify: during an injected dependency outage, the degraded ratio matches the share of traffic that touches that dependency, and returns to zero afterwards.

What a degraded response still contains A row of four page sections with their degradation state. What a degraded response still contains profile fresh orders fresh prices stale, marked recs omitted A page missing one rail beats a page that never rendered — provided you say so.

Verification

  • The budget holds. Under injected slowness in any single dependency, total request latency stays within the request-level deadline.
  • Degradation is bounded. Stale data is never served beyond stale_for, and the endpoint fails once it is exceeded.
  • It is visible. Degraded responses are counted and labelled, and the ratio returns to zero when the dependency recovers.

Pitfalls and edge cases

  • Per-call timeouts with no request budget. Four two-second timeouts is an eight-second request; slice one budget instead.
  • Unbounded staleness. A cache with no maximum age serves increasingly wrong data for as long as the outage lasts.
  • Silent degradation. If the response looks identical, users report "wrong data" rather than "degraded service", and you debug the wrong thing.
  • Fallbacks that call the same dependency. A cache warmed synchronously from the failing service is not a fallback.
  • TaskGroup for partial responses. One failing child cancels the siblings; use gather when sections must degrade independently.
  • Hedging non-idempotent calls. Two attempts means two effects; hedge reads only.
  • Fallback paths that are never exercised. Test them in CI and in game days — an untested fallback fails at exactly the wrong moment.
  • Degrading a critical dependency. Serving a stale authorisation or balance is incorrectness wearing resilience clothing.

Frequently Asked Questions

How do I stop per-dependency timeouts adding up?

Allocate one deadline for the whole request and derive each call's timeout from what remains, keeping a small reserve for rendering the response. Any dependency whose slice would be zero should go straight to its fallback rather than starting a call that cannot finish in time. This keeps total latency bounded by the client's patience regardless of how many dependencies are involved.

How long is it acceptable to serve stale cached data?

Set two bounds: a freshness window after which you prefer to re-fetch, and a maximum staleness after which serving the value is worse than failing. Minutes are usually acceptable for descriptive data such as catalogue text or recommendations; seconds at most for prices or inventory; and never for authorisation, balances or anything a user acts on financially.

Should a partial response use gather or TaskGroup?

Use gather with per-section timeouts and defaults. A TaskGroup cancels its remaining children as soon as one fails, which is correct when the whole operation is all-or-nothing and wrong when each section should degrade independently. Collect the outcomes, assemble whatever succeeded, and list the degraded sections in the response.

When is hedging a request worth the extra load?

Only for idempotent reads on a dependency with a long latency tail — a fast p50 and a slow p99. Fire the hedge at around the p95 so only a few percent of requests are duplicated, and cancel the loser as soon as either answers. If the dependency is uniformly slow, hedging just doubles its load; that case needs a breaker and a fallback instead.