Skip to content

Bulkhead Isolation with Per-Dependency Semaphores

The incident report says the recommendations service was slow. What it does not say — and what the graphs show — is that checkout, login and search all failed at the same time, even though none of them calls recommendations. The mechanism is always the same: a shared bound. Every request that touched the slow dependency held a connection from the shared pool for eight seconds instead of fifty milliseconds, the pool emptied, and every other request queued behind it. One compartment flooded and the whole ship went down.

A bulkhead is the compartment wall: a per-dependency semaphore sized so that even a totally stalled downstream can occupy only its own share of your capacity. It is perhaps fifteen lines of code, and the value is entirely in the sizing — permits derived from Little's Law, budgeted against the resource they actually consume, with a short acquisition timeout so a full compartment sheds instead of queueing. This guide covers all three, plus the metrics that tell you which compartment is filling.

Prerequisites

One shared bound versus four compartments Two columns contrasting a shared pool with per-dependency compartments. One shared bound versus four compartments shared pool no isolation Recommendations slows to 8 s Its requests hold pooled connections Pool empties Checkout and login fail too per-dependency bulkheads isolation Recommendations pins its own 10 permits Other compartments untouched Excess callers shed in 250 ms Checkout latency unchanged The wall is a semaphore; the value is entirely in where you put it.

1. Give each dependency its own compartment

One semaphore per downstream service, created once, injected everywhere that talks to it.

import asyncio
from contextlib import asynccontextmanager


class Bulkhead:
    def __init__(self, name: str, permits: int, queue_timeout: float = 0.25) -> None:
        self.name, self.permits, self.queue_timeout = name, permits, queue_timeout
        self._sem = asyncio.Semaphore(permits)
        self.rejected = 0

    @asynccontextmanager
    async def __call__(self):
        try:
            async with asyncio.timeout(self.queue_timeout):
                await self._sem.acquire()
        except TimeoutError:
            self.rejected += 1
            raise BulkheadFull(f"{self.name} at capacity ({self.permits} in flight)")
        try:
            yield
        finally:
            self._sem.release()

    @property
    def in_use(self) -> int:
        return self.permits - self._sem._value


class BulkheadFull(RuntimeError):
    pass


PAYMENTS = Bulkhead("payments", permits=15)
CATALOGUE = Bulkhead("catalogue", permits=40)
RECOMMENDATIONS = Bulkhead("recommendations", permits=10)   # optional: small share

Partition by failure domain, not by endpoint: everything that fails together should share a compartment, and everything that can fail independently should not. The optional dependency gets the smallest allocation because a stalled recommendations service should be able to affect at most 10 requests at a time. Verify: with the recommendations stub sleeping 30 s, RECOMMENDATIONS.in_use should pin at 10 while checkout latency stays flat.

2. Size permits with Little's Law

Permits are not a preference; they follow from throughput and latency.

# in-flight = arrival rate x latency
PEAK_RPS   = 300          # requests per second that touch this dependency
P99_LATENCY = 0.05        # seconds, healthy
HEADROOM   = 2.0          # allow for latency drift before shedding

permits = int(PEAK_RPS * P99_LATENCY * HEADROOM)    # 300 * 0.05 * 2 = 30

At 300 requests per second and 50 ms latency, 15 calls are in flight on average, so 30 permits carries peak traffic with room for the p99 to double before anything sheds. The headroom factor is the judgement call: too little and you shed during ordinary latency variance, too much and a sick dependency can occupy more of your capacity than you intended. Verify: under a normal load test, in-use permits should sit near PEAK_RPS × P99_LATENCY and rejections should be zero.

3. Budget the permits against the shared resource

Bulkheads are decorative if their total exceeds the pool they all draw on.

POOL_SIZE = 100                       # database connections / HTTP pool ceiling
RESERVED  = 10                        # health checks, migrations, admin paths

allocations = {
    "payments":        15,            # critical, must always have room
    "catalogue":       40,            # highest volume
    "recommendations": 10,            # optional, smallest share
    "search":          25,
}
assert sum(allocations.values()) <= POOL_SIZE - RESERVED     # 90 <= 90

The assertion belongs in your startup code, not in a comment: it converts "we thought we had isolated them" into a failure at boot. Reserve a slice for operational paths so a saturated service is still diagnosable. If the sum does not fit, the honest fix is to shrink the optional dependency's share, not to raise the pool and hope. Verify: the assertion fires if you raise any allocation, and pool exhaustion never appears in logs while individual bulkheads are rejecting.

Permits budgeted against a 100-connection pool Five bars summing to the pool size. Permits budgeted against a 100-connection pool catalogue 40 permits search 25 permits payments 15 permits recommendations 10 permits reserved for ops 10 — health, admin Assert the total at startup: 40 + 25 + 15 + 10 + 10 = 100. The sum must fit the pool, or the compartments are decorative.

4. Shed fast instead of queueing

The acquisition timeout is what distinguishes a bulkhead from a queue with extra steps.

import asyncio


async def get_recommendations(user_id: str) -> list[dict]:
    try:
        async with RECOMMENDATIONS():                 # 250 ms to get a permit
            async with asyncio.timeout(0.8):          # 800 ms for the call itself
                return await client.recommend(user_id)
    except (BulkheadFull, TimeoutError):
        return []                                     # degrade: no recommendations

Two bounds, two meanings: the queue timeout says "the compartment is full, do not wait", and the call timeout says "this attempt has taken too long". Keep the queue timeout well under the caller's own patience — a request that waits 3 s for a permit and then runs an 800 ms call has already blown a 2 s client deadline, so the permit was wasted on work nobody will read. Verify: saturate the bulkhead and confirm excess callers return in about 250 ms with the degraded response, not after the full call timeout.

5. Reserve capacity for the paths that matter

Not every caller of a dependency deserves the same share. Split the compartment when priority differs.

class TieredBulkhead:
    """Interactive traffic keeps a reserved slice that batch work cannot touch."""

    def __init__(self, name: str, total: int, batch_share: float = 0.4) -> None:
        self.name = name
        batch = int(total * batch_share)
        self._interactive = asyncio.Semaphore(total)          # may use everything
        self._batch = asyncio.Semaphore(batch)                # capped below total

    @asynccontextmanager
    async def interactive(self):
        async with self._interactive:
            yield

    @asynccontextmanager
    async def batch(self):
        async with self._batch:                               # batch takes both
            async with self._interactive:
                yield

Batch work must acquire both semaphores, so it can never occupy more than its share, while interactive traffic can use the whole compartment when batch is idle. This is a cheaper and more predictable arrangement than two separate pools, because idle capacity is still usable. Verify: saturate with batch work and confirm interactive requests still acquire permits within their queue timeout.

6. Watch saturation per compartment

The point of partitioning is that the graph now tells you which dependency is sick.

import asyncio
import logging

log = logging.getLogger("bulkhead")


async def report(bulkheads: list[Bulkhead], interval: float = 15.0) -> None:
    while True:
        await asyncio.sleep(interval)
        for bh in bulkheads:
            saturation = bh.in_use / bh.permits
            log.info("%s in_use=%d/%d saturation=%.0f%% rejected=%d",
                     bh.name, bh.in_use, bh.permits, saturation * 100, bh.rejected)
            if saturation > 0.9:
                log.warning("%s near capacity — check its latency", bh.name)

Saturation is the leading indicator: it rises as soon as a dependency's latency does, well before rejections start. Rejections are the lagging one, and they are exactly the count of requests you deliberately shed to protect everything else — which makes them a success metric, not an error metric, provided the fallback is acceptable. Verify: during an injected slowdown, exactly one compartment's saturation rises while the others stay flat.

A compartment is filling A decision diamond on saturation and rejections with three outcome cards. A compartment is filling What do saturation and rejections say? saturation high, few rejections latency has risen check that dependency's p99 saturation pinned, many rejections working as designed confirm the fallback is acceptable all compartments filling the problem is you look at loop lag and capacity Saturation leads, rejections lag — and one compartment rising is the whole diagnosis.

Verification

  • Isolation holds. With one dependency stubbed to 30 s latency, requests that do not use it show unchanged p99, and its compartment pins at its permit count.
  • Nothing exceeds the pool. The sum of permits stays within the shared resource; pool-exhaustion errors disappear from the logs entirely.
  • Shedding is fast and visible. Excess callers return within the queue timeout, and rejections are counted per compartment.

Pitfalls and edge cases

  • Bulkheads that sum above the shared pool. The queue simply moves into the pool where it is invisible; assert the budget at startup.
  • One semaphore for several dependencies. Any of them can starve the others — the exact failure the pattern exists to prevent.
  • No acquisition timeout. Callers queue indefinitely, so a full compartment holds tasks and memory instead of shedding.
  • Sizing by intuition. Permits should follow from peak rate times latency, with explicit headroom, not from a round number.
  • Forgetting the release path. Use async with (or try/finally) so cancellation cannot leak a permit permanently.
  • Ignoring the retry multiplier. If each request may retry three times, the effective in-flight count is higher than the arrival rate suggests; size for the retried rate.
  • A bulkhead without a fallback. Shedding is only an improvement if the caller degrades gracefully; otherwise you have converted a slow failure into a fast one.

Frequently Asked Questions

How many permits should a bulkhead semaphore have?

Apply Little's Law: peak requests per second that touch the dependency, multiplied by its healthy p99 latency, multiplied by a headroom factor of about two. For 300 rps at 50 ms, that is 30 permits. Then check the sum across all bulkheads against the shared connection pool, and shrink the least critical shares until it fits.

How is a bulkhead different from a connection pool limit?

A pool limit is shared, so whichever dependency is slowest consumes it all. A bulkhead is per dependency, so a stalled downstream can occupy only its own allocation. In practice they compose: the bulkheads partition the pool, and the sum of their permits must stay within the pool's real size or the partitioning is only notional.

Should the bulkhead reject or queue when full?

Reject, quickly. Give the acquisition a short timeout — 100 to 500 ms — so a full compartment sheds load to a fallback instead of accumulating waiters that hold tasks and memory. Queueing past the caller's own deadline wastes capacity on work whose client has already given up.

Can one bulkhead protect interactive and batch traffic together?

Yes, with a tiered arrangement: batch callers acquire both a batch-specific semaphore and the shared one, while interactive callers acquire only the shared one. Batch work is then capped at its share while interactive traffic can still use the whole compartment when batch is idle — better utilisation than two separate pools with fixed sizes.