Skip to content

Implementing an Async Circuit Breaker

Adding a circuit breaker is easy; adding one that improves your availability rather than damaging it is not. The three failure modes are predictable. A breaker that counts every exception opens during ordinary 404s and takes down a healthy path. A breaker with a raw consecutive-failure count flaps on a busy endpoint, adding latency variance instead of removing it. And a breaker that lets all queued traffic through the instant its cooldown expires knocks over the service it was waiting for.

This guide builds one that avoids all three: failure classification that distinguishes "the dependency is sick" from "your request was wrong", a rate window with a minimum sample size, a half-open state that admits exactly one probe, safe behaviour under concurrent callers on one event loop, and a test for every transition so the behaviour is verified rather than assumed.

Prerequisites

What each state does with a request Four-row matrix of breaker states and their request handling. What each state does with a request request is records next state closed passed through outcome in the window open if ratio exceeded open rejected immediately nothing half-open after cooldown half-open (probe) allowed, exactly one probe result closed on success, open on failure half-open (others) rejected nothing unchanged Only two paths admit traffic; both record, and only one can close the circuit.

1. Define what counts as a failure

Get this wrong and nothing else matters. The breaker's question is "is the dependency unhealthy?", not "did this call succeed?".

import httpx


def is_dependency_failure(exc: BaseException) -> bool:
    """True when the exception indicates the dependency is unhealthy."""
    if isinstance(exc, (TimeoutError, ConnectionError, OSError)):
        return True                       # transport-level: it is not answering
    if isinstance(exc, httpx.HTTPStatusError):
        return exc.response.status_code >= 500      # 5xx: it is broken
    return False                          # 4xx, validation, business errors: not ours

A 404 means your request asked for something that does not exist; a 422 means it was malformed. Both are successful interactions with a healthy service, and counting them opens the breaker during entirely normal traffic — the single most common way a breaker causes an outage. Equally important: asyncio.CancelledError must never count, because a cancelled call says nothing about the dependency. Verify: unit-test the predicate against a 404, a 500, a timeout and a CancelledError; only the middle two should return True.

2. Track failures over a window, not consecutively

Consecutive counting is fine at low traffic and terrible at high traffic, where an unrelated blip resets the count or three unlucky calls trip it.

import collections


class Window:
    """Ratio of failures over the last `size` outcomes."""

    def __init__(self, size: int = 20, minimum: int = 5) -> None:
        self.size, self.minimum = size, minimum
        self._results: collections.deque[bool] = collections.deque(maxlen=size)

    def record(self, ok: bool) -> None:
        self._results.append(ok)

    @property
    def ratio(self) -> float:
        if len(self._results) < self.minimum:
            return 0.0                       # too few samples to judge
        return self._results.count(False) / len(self._results)

    def clear(self) -> None:
        self._results.clear()

The minimum sample size is what stops a quiet endpoint from tripping on its first two failures — with minimum=5, a single timeout on an endpoint that sees one request a minute cannot open anything. A deque with maxlen gives O(1) recording and needs no pruning pass. Verify: four failures with minimum=5 yield a ratio of 0.0; the fifth produces 1.0.

3. Implement the state machine

Three states, and exactly one transition each way.

import time


class CircuitOpen(RuntimeError):
    pass


class CircuitBreaker:
    def __init__(self, name: str, *, ratio: float = 0.5, window: int = 20,
                 minimum: int = 5, cooldown: float = 30.0) -> None:
        self.name, self.ratio, self.cooldown = name, ratio, cooldown
        self._window = Window(window, minimum)
        self._state = "closed"
        self._opened_at = 0.0
        self._probe_in_flight = False

    @property
    def state(self) -> str:
        return self._state

    def allow(self) -> bool:
        """Called before every attempt. Never blocks."""
        if self._state == "closed":
            return True
        if self._state == "open":
            if time.monotonic() - self._opened_at < self.cooldown:
                return False
            self._state = "half-open"                # cooldown elapsed
        # half-open: admit exactly one probe at a time
        if self._probe_in_flight:
            return False
        self._probe_in_flight = True
        return True

    def record(self, ok: bool) -> None:
        if self._state == "half-open":
            self._probe_in_flight = False
            if ok:
                self._window.clear()
                self._state = "closed"
            else:
                self._trip()
            return
        self._window.record(ok)
        if self._window.ratio >= self.ratio:
            self._trip()

    def _trip(self) -> None:
        self._state, self._opened_at = "open", time.monotonic()
        self._probe_in_flight = False

The _probe_in_flight flag is what makes half-open safe: without it, every request that arrives after the cooldown is admitted simultaneously and the recovering dependency receives the full load it just failed under. Because the loop is single-threaded and neither allow() nor record() awaits, the flag needs no lock — a property worth preserving deliberately. Verify: with the breaker open and the cooldown elapsed, call allow() three times; exactly one should return True.

Ratio window versus consecutive counting Three lanes contrasting consecutive counting with a ratio window over the same call outcomes. Ratio window versus consecutive counting outcomes ok fail ok fail fail fail consecutive count ratio over window 4 of 6 failed: opens here most recent calls, left to right A ratio with a minimum sample size trips on real degradation and ignores noise.

4. Wrap calls so nothing bypasses the breaker

Put the breaker in the client wrapper, not at each call site.

import asyncio


class Guarded:
    def __init__(self, breaker: CircuitBreaker, timeout: float = 1.5) -> None:
        self.breaker, self.timeout = breaker, timeout

    async def call(self, factory, *, fallback=None):
        if not self.breaker.allow():
            if fallback is not None:
                return await fallback()
            raise CircuitOpen(f"{self.breaker.name} is open")
        try:
            async with asyncio.timeout(self.timeout):
                result = await factory()
        except asyncio.CancelledError:
            self.breaker.record(ok=True)       # our cancellation, not their fault
            raise
        except BaseException as exc:
            self.breaker.record(ok=not is_dependency_failure(exc))
            if fallback is not None and is_dependency_failure(exc):
                return await fallback()
            raise
        self.breaker.record(ok=True)
        return result

Handling CancelledError separately matters more than it looks: a shutdown that cancels a hundred in-flight requests would otherwise record a hundred failures and open every breaker in the process, so the service comes back up with all its circuits tripped. Verify: cancel a task mid-call and confirm the breaker's state and window are unchanged.

5. Add the observability that makes it debuggable

A breaker without metrics is a source of unexplained latency changes.

STATE_VALUE = {"closed": 0, "half-open": 1, "open": 2}


class ObservableBreaker(CircuitBreaker):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.transitions = 0
        self.time_open = 0.0
        self.rejected = 0

    def allow(self) -> bool:
        allowed = super().allow()
        if not allowed:
            self.rejected += 1
        return allowed

    def _trip(self) -> None:
        was = self._state
        super()._trip()
        self.transitions += 1
        log.warning("breaker %s %s -> open (ratio %.2f)", self.name, was, self._window.ratio)

    def snapshot(self) -> dict:
        open_now = time.monotonic() - self._opened_at if self._state == "open" else 0.0
        return {"state": STATE_VALUE[self._state], "rejected": self.rejected,
                "transitions": self.transitions, "open_seconds": open_now}

Export the state as a numeric gauge (0/1/2) so dashboards can show it alongside latency, plus rejection count and transitions. Alert on time spent open per hour rather than on individual transitions — a breaker that opens for two seconds twice a day is working; one that is open 15% of the hour is telling you about a real dependency problem. Verify: the gauge tracks state changes within one scrape interval during an injected outage.

6. Test every transition

Each edge of the state machine deserves a test, and they are all fast because nothing needs a real network.

import asyncio


class FakeClock:
    def __init__(self) -> None:
        self.now = 1000.0

    def monotonic(self) -> float:
        return self.now


async def test_breaker(monkeypatch) -> None:
    clock = FakeClock()
    monkeypatch.setattr(time, "monotonic", clock.monotonic)
    cb = CircuitBreaker("test", ratio=0.5, window=10, minimum=4, cooldown=30.0)

    # closed -> stays closed below the minimum sample size
    for _ in range(3):
        cb.record(ok=False)
    assert cb.state == "closed"

    # closed -> open once the ratio is met with enough samples
    cb.record(ok=False)
    assert cb.state == "open"
    assert cb.allow() is False                    # rejects during cooldown

    # open -> half-open after the cooldown, one probe only
    clock.now += 31
    assert cb.allow() is True
    assert cb.allow() is False                    # second caller is rejected

    # half-open -> open on a failed probe
    cb.record(ok=False)
    assert cb.state == "open"

    # half-open -> closed on a successful probe
    clock.now += 31
    assert cb.allow() is True
    cb.record(ok=True)
    assert cb.state == "closed"
    assert cb.allow() is True

Injecting the clock is what keeps the suite fast and deterministic — a test that sleeps for a real 30 second cooldown will be deleted by the first person who runs the suite. Verify: the whole state-machine test runs in milliseconds and fails if you remove the _probe_in_flight guard.

Latency during a downstream outage Three bars comparing latency with and without an open circuit breaker. Latency during a downstream outage no breaker: every call waits the timeout 1.5 s per request breaker open: fallback served 3 ms half-open probe (1 per cooldown) 1.5 s, once per 30 s Measured against a stub returning 500s, with a 1.5 s per-call timeout and a 30 s cooldown. An open circuit costs microseconds and holds no connection, permit or parked task.

Verification

  • Healthy traffic never trips it. A load test containing 20% 404s leaves the breaker closed throughout.
  • A real outage trips it fast. With the dependency returning 500s, the breaker opens within roughly minimum calls and latency for that path drops to the fallback's cost.
  • Recovery is gentle. When the dependency returns, exactly one probe is admitted per cooldown, and the breaker closes on the first success.

Pitfalls and edge cases

  • Counting 4xx as failures. Opens the circuit during normal traffic; classify by transport errors and 5xx only.
  • Counting CancelledError. A shutdown or a client disconnect then trips every breaker in the process.
  • Letting all traffic through at cooldown end. The recovering dependency gets the full load immediately; admit one probe.
  • Consecutive-failure counting on busy endpoints. It flaps. Use a ratio over a window with a minimum sample size.
  • Wall-clock time. time.time() jumps with NTP; use time.monotonic() for the cooldown.
  • A breaker per call site. Each instance sees a fraction of the traffic and none of them trips. One breaker per dependency, injected.
  • Retries inside, invisible outside. If an inner retry swallows failures, the breaker never learns; record the outcome at the breaker layer.
  • Sharing a breaker across event loops. State is not thread-safe across loops; give each loop its own instance or guard it explicitly.

Frequently Asked Questions

What should count as a failure in a circuit breaker?

Only signals that the dependency itself is unhealthy: timeouts, connection errors, other transport failures, and 5xx responses. Client errors such as 404 or 422 mean a healthy service rejected a specific request, and counting them opens the circuit during normal traffic. Cancellation must also be excluded, otherwise a shutdown that cancels in-flight calls trips every breaker in the process.

How does the half-open state work in a circuit breaker?

When the cooldown expires the breaker moves to half-open and admits exactly one request as a probe, rejecting all others. If the probe succeeds the breaker closes and normal traffic resumes; if it fails the breaker re-opens for another cooldown. Admitting only one probe is what stops a recovering dependency from being hit with full load the instant the cooldown ends.

Does an asyncio circuit breaker need a lock?

Not if its state transitions contain no await. On a single event loop, a function that only reads and writes attributes runs to completion without interleaving, so allow() and record() are atomic by construction. Introduce a lock only if you add an await inside those methods — for example to publish state to an external store — and prefer keeping them synchronous instead.

How do I test a circuit breaker without waiting for the cooldown?

Inject the clock. Patch time.monotonic with a fake that you advance manually, so a 30 second cooldown is a single assignment in the test. This keeps the whole state-machine suite in the millisecond range and makes every transition — including the half-open probe limit — deterministic.