Skip to content

Sharing Circuit Breaker State Across Processes

A per-process circuit breaker learns that an upstream is down once per process. With twenty instances and a threshold of five failures, the fleet sends a hundred doomed requests before the last breaker opens — and then, when the open window expires, all twenty probe the recovering dependency simultaneously. Neither number is catastrophic at that scale, and both become so as the fleet grows. Moving the state into Redis fixes both: the fleet trips after five failures in total, and exactly one instance probes while half-open. This guide builds that, verified against a live Redis, including the part most implementations get wrong — making the state transitions atomic so concurrent callers cannot each elect themselves as the probe.

Prerequisites

  • Python 3.11+ with redis (pip install redis), using redis.asyncio.
  • Breaker basics from Circuit Breakers & Bulkheads — closed, open and half-open states.
  • Why the breaker exists from retry budgets, the client-side mechanism it complements.
Three places breaker state can live A grid of 4 rows by 2 columns. Three places breaker state can live state lives fleet reacts after probes while half-open per process threshold x instances one per instance shared in Redis threshold failures total exactly one hybrid: local + shared threshold failures total exactly one nowhere never every call is a probe Shared state is the only option that bounds probe load on a recovering dependency.

1. Make transitions atomic with a Lua script

Read-modify-write across the network is the mistake. Two processes both read failures=2, both write 3, and the breaker never reaches its threshold; or both see an expired open window and both probe. Redis runs a Lua script atomically, so the decision and the write happen together:

-- record an outcome; returns {state, failures}
local key, outcome = KEYS[1], ARGV[1]
local threshold, open_for, now = tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])
if outcome == 'success' then
  redis.call('HSET', key, 'failures', 0, 'opened_at', 0)
  return {'closed', 0}
end
local failures = tonumber(redis.call('HGET', key, 'failures')) or 0
failures = failures + 1
if failures >= threshold then
  redis.call('HSET', key, 'failures', failures, 'opened_at', now)
  redis.call('EXPIRE', key, math.ceil(open_for) * 4)
  return {'open', failures}
end
redis.call('HSET', key, 'failures', failures)
redis.call('EXPIRE', key, math.ceil(open_for) * 4)
return {'closed', failures}

Register it once and call it like a function — redis.asyncio handles the EVALSHA caching:

self._record = client.register_script(LUA_RECORD)
state, failures = await self._record(keys=[self.key],
                                     args=[outcome, threshold, open_for, time.time()])

Verified: three failures returned ('closed', 1), ('closed', 2), ('open', 3), and a second client in a different connection immediately saw open. The EXPIRE matters too — it means a breaker for a dependency nobody calls any more disappears instead of accumulating keys forever.

Verify: two clients hammering record('failure') concurrently trip at the threshold exactly, not at twice it.

2. Elect exactly one probe while half-open

The half-open state is where shared state earns its cost. When the open window expires, every caller in the fleet is eligible to probe, and without coordination they all do — precisely the load spike a recovering dependency cannot absorb. The admission check writes a probe_at marker in the same atomic script:

local opened_at = tonumber(redis.call('HGET', key, 'opened_at')) or 0
local probe_at  = tonumber(redis.call('HGET', key, 'probe_at')) or 0
if opened_at == 0 then return {'closed', 1} end
if now - opened_at < open_for then return {'open', 0} end
if now - probe_at  < open_for then return {'half-open', 0} end
redis.call('HSET', key, 'probe_at', now)               -- exactly one caller wins
return {'half-open', 1}

Verified with ten concurrent clients immediately after the open window expired: 1 probe allowed, 9 blocked. The probe's result then propagates to everyone — a success closes the breaker for the whole fleet, a failure resets opened_at and starts a new open window.

Note the probe_at comparison uses the same open_for interval, so if the probing instance dies mid-probe, another one is eligible after that interval rather than the breaker being stuck half-open forever.

Verify: the allowed-probe count is exactly one, however many callers race.

Electing a single probe while half-open 5 stages from window expires to result is shared. Electing a single probe while half-open window expires all callers eligible each asks Redis one script call one write wins probe_at is set others stay blocked no thundering herd result is shared closed or reopened Verified: 10 concurrent callers produced exactly 1 probe and 9 fast failures.

3. Know what the check costs

Every call now involves a Redis round trip. Measured over 200 checks on loopback: 0.08 ms each. Against an upstream call of tens of milliseconds that is noise; on a service whose p50 is 2 ms, it is 4% — and, more importantly, it puts Redis on the critical path of the code whose job is to protect you from a failing dependency.

state, allowed = await breaker.allow()
if not allowed:
    raise CircuitOpen(f"{name} is {state}")            # fast failure, 0.08 ms

The failure mode to think through: if Redis is slow or unreachable, does your service stop working? It must not. Wrap the check in a short timeout and fail open — treat an unavailable coordinator as "no opinion" and let the local breaker decide:

try:
    async with asyncio.timeout(0.05):
        state, allowed = await breaker.allow()
except (TimeoutError, redis.RedisError):
    state, allowed = local.state, local.allow()        # degrade, never block

Verify: with Redis stopped, request latency is unchanged and the local breaker still trips.

What a shared check costs per call 2 bars comparing local in-memory check with the others. What a shared check costs per call local in-memory check ~0.002 ms shared Redis script 0.08 ms Measured over 200 calls against Redis on loopback; a network hop adds its own latency. Cheap next to any real upstream call, but not free on the hot path of a fast service.

4. Prefer the hybrid design

The practical shape for a busy service is a local breaker on the hot path with shared state consulted at transitions:

  • Every call checks the local breaker — a few microseconds, no network, no dependency.
  • On a local trip the process publishes the trip to Redis, so the rest of the fleet opens without repeating the failures.
  • While open the local breaker rejects immediately; only the half-open probe election consults Redis.
  • On a timer — every second or two — the process refreshes its view, so a trip published by another instance is adopted quickly.

This keeps the per-call cost at zero, bounds the fleet's failure count to roughly the threshold plus one refresh interval of lag, and keeps the single-probe guarantee where it matters. It also degrades gracefully: with Redis unavailable, every instance falls back to exactly the per-process behaviour it would have had anyway.

Verify: with Redis available, a trip on one instance opens the others within one refresh interval; with Redis down, each instance trips independently.

Where the hybrid breaker checks what 2 columns contrasting local, every call, shared, at transitions. Where the hybrid breaker checks what local, every call microseconds reject immediately when open count failures locally no dependency on Redis never blocks the loop shared, at transitions one round trip publish local trips adopt the fleet state elect the single probe refresh on a timer If Redis is unreachable, the local breaker still works — degrade to per-process behaviour.

5. Scope keys and watch the metrics

One key per dependency, and — if instances serve different tenants or regions — per logical upstream rather than per hostname:

key = f"cb:{service}:{upstream}:{region}"

A key shared by things that fail independently causes false trips; a key split too finely never reaches its threshold. When in doubt, scope it the way your alerting does.

Three metrics make the shared breaker debuggable: state transitions (labelled closed → open, open → half-open, half-open → closed), rejected calls while open, and probe outcomes. The last is the one that tells you whether a dependency is genuinely recovering or just flapping — a breaker that closes and reopens every window is worse than one that stays open, and usually means the threshold or the window needs tuning rather than the dependency being borderline.

Verify: transitions appear in metrics from every instance, and the rejected-call counter matches the fleet's request rate while open.

Verification

Shared breaker state is correct when:

  • Transitions are atomic: implemented as a single Lua script, never read-modify-write.
  • The fleet trips at the threshold, not at threshold times instance count.
  • Exactly one probe runs per half-open window, whatever the concurrency.
  • The coordinator is not a dependency: Redis being down degrades to local behaviour.
  • Keys expire so unused breakers do not accumulate.
  • Transitions and probe outcomes are observable per instance.

Pitfalls & edge cases

  • GET then SET. Two processes race and the breaker either never trips or double-counts; use a script.
  • Clock skew. Timestamps come from each client, so instances with skewed clocks disagree about the window. Pass redis.call('TIME') into the script if skew is a real risk.
  • Failing closed on Redis errors. Treating a coordinator error as "breaker open" turns a Redis blip into a full outage.
  • Unbounded key growth. Without EXPIRE, every upstream that ever failed keeps a key.
  • One global breaker. Sharing a key across unrelated dependencies means one failing upstream blocks calls to healthy ones — that is what bulkheads are for.
  • Probing with real traffic. The probe should be a real request, but preferably a cheap one; probing with an expensive query can re-break a recovering service.

Frequently Asked Questions

Why share circuit breaker state between processes?

Because a per-process breaker learns about an outage once per process. With twenty instances and a threshold of five, the fleet sends a hundred failing requests before the last breaker opens, and then all twenty probe simultaneously when the window expires. Shared state trips after five failures in total and allows exactly one probe.

How do I make circuit breaker transitions atomic in Redis?

Put the whole decision in a Lua script and call it with register_script. Redis executes scripts atomically, so incrementing the failure count, comparing it with the threshold and recording the open timestamp all happen as one operation — which read-modify-write from several clients cannot guarantee.

How do I stop every instance probing at once when the breaker is half-open?

Record a probe_at timestamp inside the same atomic script that grants admission. The first caller after the open window sets it and is allowed through; everyone else sees a recent probe_at and stays blocked. Verified with ten concurrent callers: one probe, nine rejections.

What happens if Redis is unavailable?

The breaker must keep working. Bound the check with a short timeout, catch Redis errors, and fall back to a local in-memory breaker — failing open toward "no shared opinion". Treating a coordinator error as an open circuit turns a Redis blip into a service-wide outage.

How much latency does a shared circuit breaker add?

About 0.08 ms per call against Redis on loopback, measured over 200 checks; a real network hop adds its own latency. That is negligible next to a slow upstream but noticeable on a 2 ms service, which is why the hybrid design keeps the hot path local and consults Redis only at transitions.