Skip to content

Circuit Breakers & Bulkheads in Asyncio

A dependency does not usually fail cleanly. It gets slow — 50 ms becomes 8 seconds — and in an async service that is far more dangerous than an outright error, because slowness accumulates. Every request that touches the failing dependency parks a task, holds a connection, and consumes a slot in whatever bounds your concurrency. Ten seconds later the pool is exhausted, and requests that never needed that dependency at all are queued behind the ones that do. One degraded service has taken the whole process with it.

Two patterns contain that. A bulkhead partitions the resource so a failing dependency can only consume its own share — the ship's compartment that keeps a breach local. A circuit breaker watches the failure rate and, once it crosses a threshold, stops attempting the call entirely for a cooling-off period, converting a slow failure into a fast one. Neither makes the dependency work; both stop it from taking down everything else. The parent overview, Resilience, Cancellation & Error Handling, covers timeouts and retries; this section covers what to do when retrying is the wrong answer.

Scope of this section:

  • Circuit breaker mechanics: the three states, what counts as a failure, and how a breaker recovers.
  • Choosing thresholds and cooling periods that fit real traffic rather than a textbook.
  • Bulkheads: per-dependency semaphores and pools that stop one failure consuming everything.
  • Composing breaker, bulkhead, timeout and retry in the right order.
  • Fallbacks and degraded modes — and the metrics that show whether a breaker is helping or hiding.

Architectural principles

  • Timeouts bound one call; breakers bound the whole relationship. A per-call timeout still lets every request pay the full timeout. A breaker notices the pattern and stops paying it.
  • Retries make a failing dependency worse. Retrying into an overloaded service multiplies its load exactly when it can least absorb it. The breaker's job is to be the circuit-level "stop", above the retry logic.
  • Isolate by dependency, not by endpoint. Bulkheads should mirror your failure domains: one per downstream service or database, so exhaustion in one leaves the others untouched.
  • Failure needs a definition. Timeouts and 5xx are failures; a 404 or a validation error is not. Counting business errors as failures opens breakers during perfectly healthy traffic.
  • A tripped breaker must degrade, not just fail. The value is in what you do instead: a cached answer, a partial response, a queued write. A breaker with no fallback converts a slow error into a fast one — better, but not the whole win.
The breaker as a state machine Four breaker states with transitions for tripping, cooling down and probing. The breaker as a state machine closed open half-open closed again failure ratio exceeded cooldown elapsed probe succeeded probe failed Half-open admits exactly one probe, so recovery cannot re-flood the dependency.

Execution model: what "contained" means on one event loop

An async service has no thread pool insulating one dependency from another; everything shares a single loop and, usually, a single set of bounded resources. When a downstream slows from 50 ms to 8 s, the number of tasks parked on it grows as arrival rate × latency — Little's Law again — and each parked task holds whatever it acquired before it started waiting: a connection from a pool, a permit from a limiter, memory for its request state. Nothing here is CPU contention; the loop is idle. The exhaustion is entirely in the bounded resources the waiting tasks hold.

That is why a bulkhead is a semaphore rather than a thread pool. Give each dependency its own permit count and the arithmetic changes: a dependency that slows to 8 s can hold at most its own permits, so its share of tasks stalls and everything else proceeds. The cost of the bulkhead is that requests to the sick dependency now fail fast at acquisition instead of waiting — which is the intended behaviour, expressed as a decision you made rather than as pool exhaustion you discovered.

The breaker sits one level above and is pure bookkeeping: a counter, a state, and a clock. Because it makes its decision before the call, a tripped breaker costs microseconds and holds nothing — no connection, no permit, no task parked for the duration of a timeout. That difference is the entire point: under a downstream outage with a breaker open, your latency stays flat and your resource usage stays at baseline, where without one both track the failing dependency.

A dependency slows from 50 ms to 8 s Two columns contrasting uncontained and contained responses to a slow dependency. A dependency slows from 50 ms to 8 s no containment shared bounds Parked tasks grow with arrival rate Each holds a pooled connection Pool empties in seconds Unrelated endpoints fail too breaker + bulkhead bounded blast radius Bulkhead caps parked tasks Breaker stops calling entirely Open circuit holds nothing Other endpoints unaffected The loop is idle in both cases — what runs out is the bounded resources.

Pattern catalogue

The three-state breaker

Closed (calls pass), open (calls fail immediately), half-open (one probe decides).

import asyncio
import time


class CircuitOpen(RuntimeError):
    """Raised instead of calling a dependency that is currently failing."""


class CircuitBreaker:
    def __init__(self, threshold: int = 5, cooldown: float = 30.0) -> None:
        self.threshold, self.cooldown = threshold, cooldown
        self.failures = 0
        self.opened_at = 0.0
        self.state = "closed"

    def _allow(self) -> bool:
        if self.state == "closed":
            return True
        if time.monotonic() - self.opened_at >= self.cooldown:
            self.state = "half-open"            # let exactly one probe through
            return True
        return False

    def record(self, ok: bool) -> None:
        if ok:
            self.failures, self.state = 0, "closed"
        else:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.threshold:
                self.state, self.opened_at = "open", time.monotonic()

Half-open is the state that makes recovery safe: after the cooldown, exactly one request is allowed through, and its outcome decides whether the breaker closes or re-opens for another cooldown. Letting all waiting traffic through at once instead is how a recovering service gets knocked over again immediately.

Wrapping the call

The breaker is only useful if nothing can bypass it, so wrap the client rather than the call sites.

import asyncio


async def call_with_breaker(breaker: CircuitBreaker, coro_factory, *,
                            timeout: float = 2.0):
    if not breaker._allow():
        raise CircuitOpen("dependency is unavailable (breaker open)")
    try:
        async with asyncio.timeout(timeout):
            result = await coro_factory()
    except (TimeoutError, ConnectionError, OSError):
        breaker.record(ok=False)                # transport-level failure
        raise
    except HTTPStatusError as exc:
        breaker.record(ok=exc.response.status_code < 500)   # 4xx is not our problem
        raise
    breaker.record(ok=True)
    return result

Note the discrimination between error classes. A timeout or a 5xx means the dependency is unhealthy; a 404 or a 422 means your request was wrong, and counting those trips the breaker during entirely normal traffic — one of the most common ways a breaker causes the outage it was meant to prevent.

Bulkhead per dependency

One semaphore per downstream, sized so that total permits stay within your capacity.

import asyncio
from contextlib import asynccontextmanager


class Bulkhead:
    def __init__(self, name: str, permits: int, queue_timeout: float = 0.25) -> None:
        self.name, self.permits = name, permits
        self._sem = asyncio.Semaphore(permits)
        self.queue_timeout = queue_timeout
        self.rejected = 0

    @asynccontextmanager
    async def __call__(self):
        try:
            async with asyncio.timeout(self.queue_timeout):
                await self._sem.acquire()
        except TimeoutError:
            self.rejected += 1
            raise CircuitOpen(f"{self.name} bulkhead full")     # fail fast, hold nothing
        try:
            yield
        finally:
            self._sem.release()


PAYMENTS = Bulkhead("payments", permits=20)
SEARCH = Bulkhead("search", permits=40)

The short acquisition timeout is what makes it a bulkhead rather than just a queue: when the compartment is full, callers are rejected in a quarter of a second instead of joining a line that will outlive their client's own deadline.

Composition order

Breaker outermost, bulkhead next, timeout innermost — with retries inside the breaker so a retry storm still counts toward tripping it.

async def fetch_profile(user_id: str) -> dict:
    if not BREAKER._allow():                       # 1. cheapest check first
        return cached_profile(user_id)             #    fallback, no resources held
    async with PROFILE_BULKHEAD():                 # 2. bounded share of capacity
        try:
            async with asyncio.timeout(1.5):       # 3. bound this attempt
                result = await client.get_profile(user_id)
        except (TimeoutError, ConnectionError):
            BREAKER.record(ok=False)
            raise
        BREAKER.record(ok=True)
        return result

The order matters because each layer is cheaper than the one inside it. Checking the breaker first means an open circuit costs nothing; acquiring the bulkhead second means a saturated dependency rejects before a connection is taken; the timeout last bounds the attempt that actually happens.

Fallbacks and degraded modes

What you do instead is where the resilience actually lives.

async def profile_or_degraded(user_id: str) -> dict:
    try:
        return await fetch_profile(user_id)
    except CircuitOpen:
        return await cache.get(user_id) or {"id": user_id, "degraded": True}
    except TimeoutError:
        return {"id": user_id, "degraded": True}          # partial response

Serve stale data with an explicit marker, omit an optional section, or queue a write for later — but make the degradation visible in the response and in your metrics, so "we are running degraded" is a fact rather than a suspicion. Fallback patterns per dependency type are covered in adding timeouts and fallbacks to degraded dependencies.

Composition order, cheapest check first A vertical composition of breaker, bulkhead, timeout and retry. Composition order, cheapest check first circuit breaker open? cost is microseconds, holds nothing bulkhead permit for this dependency only, short queue timeout per-call timeout bounds the attempt that actually happens retry, inside the breaker so a retry storm still trips it Each layer is cheaper than the one inside it, so failures cost less the earlier they are caught.

Resource boundaries

Sizing is where breakers and bulkheads succeed or fail. The numbers come from your capacity and traffic, not from defaults.

Parameter Starting point How to refine it
Failure threshold 5 consecutive, or 50% over 20 calls Raise if a single blip trips it; a rate window suits high-traffic paths better than a count
Cooldown 30 s Long enough for a restart or failover; too short re-probes a service still recovering
Half-open probes 1 Increase only when a single probe is too noisy a signal
Bulkhead permits Little's Law: peak rate × p99 latency Sum across dependencies must stay under the pool/fd budget
Bulkhead queue timeout 100–500 ms Below the client's own patience; above your normal acquisition time
Per-call timeout p99 latency × 2 Under the caller's remaining deadline, so a deadline propagates

One constraint ties them together: the sum of all bulkhead permits must not exceed the resource they draw on. Four dependencies at 40 permits each need 160 connections; if the pool holds 100, the bulkheads are decorative and exhaustion will still happen in the pool.

Integrated production example

A dependency wrapper combining a rate-window breaker, a bulkhead, per-call timeouts, fallbacks and metrics.

import asyncio
import collections
import logging
import time

log = logging.getLogger("resilience")


class CircuitOpen(RuntimeError):
    pass


class Breaker:
    """Rate-window breaker: opens when the failure ratio over the last N calls
    exceeds `ratio`, with a minimum sample size so one bad call cannot trip it."""

    def __init__(self, name: str, window: int = 20, ratio: float = 0.5,
                 minimum: int = 5, cooldown: float = 30.0) -> None:
        self.name, self.ratio, self.minimum, self.cooldown = name, ratio, minimum, cooldown
        self.results: collections.deque[bool] = collections.deque(maxlen=window)
        self.state, self.opened_at = "closed", 0.0
        self.stats = {"rejected": 0, "opened": 0, "probes": 0}

    def allow(self) -> bool:
        if self.state == "closed":
            return True
        if time.monotonic() - self.opened_at >= self.cooldown:
            self.state = "half-open"
            self.stats["probes"] += 1
            return True
        self.stats["rejected"] += 1
        return False

    def record(self, ok: bool) -> None:
        if self.state == "half-open":
            if ok:
                self.results.clear()
                self.state = "closed"
                log.info("breaker %s closed after successful probe", self.name)
            else:
                self._open()
            return
        self.results.append(ok)
        if len(self.results) >= self.minimum:
            failures = self.results.count(False) / len(self.results)
            if failures >= self.ratio:
                self._open()

    def _open(self) -> None:
        self.state, self.opened_at = "open", time.monotonic()
        self.stats["opened"] += 1
        log.warning("breaker %s OPEN for %.0fs", self.name, self.cooldown)


class Dependency:
    """One breaker + one bulkhead + one timeout budget per downstream service."""

    def __init__(self, name: str, permits: int = 20, timeout: float = 1.5,
                 queue_timeout: float = 0.25) -> None:
        self.name = name
        self.breaker = Breaker(name)
        self.sem = asyncio.Semaphore(permits)
        self.permits = permits
        self.timeout, self.queue_timeout = timeout, queue_timeout
        self.stats = {"calls": 0, "failed": 0, "shed": 0, "fallback": 0}

    async def call(self, factory, fallback=None):
        if not self.breaker.allow():
            self.stats["fallback"] += 1
            if fallback is None:
                raise CircuitOpen(f"{self.name} unavailable")
            return await fallback()

        try:                                          # bulkhead: bounded share
            async with asyncio.timeout(self.queue_timeout):
                await self.sem.acquire()
        except TimeoutError:
            self.stats["shed"] += 1                   # compartment full: shed early
            if fallback is None:
                raise CircuitOpen(f"{self.name} saturated")
            return await fallback()

        try:
            self.stats["calls"] += 1
            async with asyncio.timeout(self.timeout):
                result = await factory()
        except (TimeoutError, ConnectionError, OSError) as exc:
            self.stats["failed"] += 1
            self.breaker.record(ok=False)
            if fallback is None:
                raise
            log.warning("%s failed (%s) — serving fallback", self.name, type(exc).__name__)
            self.stats["fallback"] += 1
            return await fallback()
        else:
            self.breaker.record(ok=True)
            return result
        finally:
            self.sem.release()

    @property
    def in_use(self) -> int:
        return self.permits - self.sem._value


PROFILES = Dependency("profiles", permits=20, timeout=1.5)
SEARCH = Dependency("search", permits=40, timeout=0.8)


async def get_profile(client, cache, user_id: str) -> dict:
    async def live():
        return await client.get_profile(user_id)

    async def cached():
        return (await cache.get(user_id)) or {"id": user_id, "degraded": True}

    return await PROFILES.call(live, fallback=cached)


async def report(interval: float = 30.0) -> None:
    while True:
        await asyncio.sleep(interval)
        for dep in (PROFILES, SEARCH):
            log.info("%s state=%s calls=%d failed=%d shed=%d fallback=%d in_use=%d/%d",
                     dep.name, dep.breaker.state, dep.stats["calls"], dep.stats["failed"],
                     dep.stats["shed"], dep.stats["fallback"], dep.in_use, dep.permits)

The design choices worth stealing: the breaker uses a ratio over a window with a minimum sample size, so a single timeout on a quiet endpoint cannot open it, while a genuinely failing dependency trips within a couple of seconds of traffic. Every rejection path — open circuit, full bulkhead, failed call — routes through the same fallback, so degraded behaviour is one code path rather than three. And the permit release lives in finally alongside a return in else, which is the shape that survives cancellation.

Diagnostic Hook — is the breaker protecting you or hiding a bug?

Export per dependency: state (as a gauge: 0 closed, 1 half-open, 2 open), calls, failures, shed and fallback counts, bulkhead saturation (in-use ÷ permits), and time spent open per hour. Two readings matter most. A breaker that opens and closes repeatedly — flapping — means the threshold is too tight or the cooldown too short, and it is adding latency variance rather than removing it. A breaker that never opens while the dependency's error rate is high means your failure predicate is wrong, usually because 4xx responses are being counted as successes and the real failures are hidden inside them. Alert on total time-open per hour, not on individual transitions, and always alert on fallback rate: that is the number your users actually feel.

Reading the breaker metrics Four-row matrix of breaker readings, meanings and actions. Reading the breaker metrics reading means do opens and closes repeatedly flapping threshold too tight widen window, lengthen cooldown never opens, errors high blind failure predicate wrong count 5xx and timeouts only open 15% of the hour dependency is genuinely sick escalate to the owning team verify the fallback fallback rate rising users feel it degraded responses are frequent alert on this, not transitions Time-open per hour and fallback rate are the two numbers worth alerting on.

Failure modes

Failure mode Root cause Detection Fix
One slow dependency stalls everything No bulkhead; all requests share one pool Pool exhaustion while CPU is idle One semaphore per dependency, sized by Little's Law
Breaker opens during healthy traffic 4xx counted as failures Breaker opens while the dependency's own metrics are green Count only timeouts, 5xx and transport errors
Breaker flaps Threshold too tight, cooldown too short State gauge oscillates every few seconds Use a ratio over a window with a minimum sample; lengthen cooldown
Recovering service knocked over again All traffic released at once when the cooldown ends Downstream re-fails immediately after every recovery Half-open with a single probe, then ramp
Breaker never opens Failures swallowed by an inner retry High latency, no state change Record failures at the breaker layer, outside the retry
Bulkheads exceed real capacity Sum of permits above pool/fd limits Exhaustion persists despite bulkheads Budget permits against the shared resource
Fallback is worse than the failure Stale cache served indefinitely Users report inconsistent data long after recovery Bound fallback staleness; mark degraded responses

Frequently Asked Questions

When should I use a circuit breaker instead of retries?

Retries suit transient, independent failures — a dropped packet, one bad node behind a load balancer. A breaker suits correlated failure: the dependency itself is down or overloaded, where retrying multiplies load on a service that is already struggling. Use both, layered: retry a couple of times inside, and let the breaker observe the outcome so a sustained failure trips it and stops the attempts entirely.

What failure threshold and cooldown should a circuit breaker use?

Start with a failure ratio of about 50% over a window of 20 calls with a minimum of 5 samples, and a 30 second cooldown. A raw consecutive-failure count works for low-traffic paths, but on busy endpoints it trips on noise. The cooldown should be long enough for a restart or failover to complete — too short and the half-open probe hits a service still recovering.

How is a bulkhead different from a rate limiter?

A rate limiter controls how fast you send; a bulkhead controls how much of your capacity any one dependency may occupy. The bulkhead's purpose is isolation: with a per-dependency semaphore, a downstream that slows to eight seconds can stall only its own permits, leaving connections and tasks available for everything else. They compose well — the limiter protects the downstream, the bulkhead protects you.

Where should the circuit breaker sit relative to timeouts and retries?

Outermost. Check the breaker before acquiring anything, so an open circuit costs no connection, no permit and no waiting task. Inside it comes the bulkhead, then the per-call timeout. Keep retries inside the breaker's observation so a retry storm counts toward tripping it rather than hiding the failure from it.

Should a circuit breaker be shared across service instances?

Usually not. A per-instance breaker reacts to what that instance actually observes and needs no coordination, which keeps it fast and dependency-free. A shared breaker trips more consistently but adds a network round trip to a hot path and a new shared failure mode. Prefer per-instance breakers, and rely on the downstream's own load shedding for global protection.