Skip to content

Implementing Health and Readiness Probes for asyncio

A health endpoint that returns {"status": "ok"} unconditionally answers the question "is this HTTP server accepting connections?" — which was never in doubt, because the orchestrator just connected to it. The questions worth answering are different and specific: is the event loop still responsive, are the dependencies this instance needs actually reachable, and should this instance be sent traffic right now. Async services have one signal for the first question that synchronous services do not: event-loop lag, which is cheap to measure and correlates directly with whether requests can be served. This guide builds all three probes, with the lag measurement verified against a deliberately blocked loop.

Prerequisites

Three probes, three questions A grid of 4 rows by 2 columns. Three probes, three questions probe answers on failure liveness is the process wedged? the container is restarted readiness should it get traffic now? removed from the load balancer startup has it finished booting? the other probes wait dependencies belongs in readiness only never in liveness A dependency check in a liveness probe restarts every instance when that dependency fails.

1. Separate liveness from readiness

The two probes have different consequences, and conflating them causes outages.

  • Liveness asks whether the process is broken beyond recovery. Failure means restart. It must therefore depend on nothing external — a liveness probe that checks the database restarts every instance in the fleet the moment the database has a bad minute, turning a degradation into an outage.
  • Readiness asks whether this instance should receive traffic now. Failure means remove from the load balancer, which is reversible and cheap. Dependency checks, capacity limits and shutdown state all belong here.
async def livez(request):
    return JSONResponse({"alive": True})               # nothing external, ever


async def readyz(request):
    ok = state.ready and state.loop_lag < LAG_BUDGET
    return JSONResponse({"ready": ok, "loop_lag_ms": round(state.loop_lag * 1000)},
                        status_code=200 if ok else 503)

A third probe, startup, tells the orchestrator that a slow boot — cache warming, migrations, model loading — is still in progress, so the other probes are not held to their normal deadlines yet. Where the platform supports it, use it rather than inflating the liveness timeout.

Verify: stopping the database makes readiness fail and liveness keep passing.

2. Measure event-loop lag

Lag is the difference between how long a sleep was supposed to take and how long it actually took. It is the single most useful health signal an async service has, and the measurement costs one task:

class Health:
    def __init__(self):
        self.loop_lag = 0.0

    async def measure_lag(self, interval: float = 0.1) -> None:
        while True:
            start = time.perf_counter()
            await asyncio.sleep(interval)
            self.loop_lag = (time.perf_counter() - start) - interval

Verified against a handler that called time.sleep(0.6): the probe reported 587 ms of lag immediately afterwards and returned 503, then 1 ms and 200 once the loop recovered. That is exactly the behaviour you want from a readiness probe — the instance stops receiving traffic while it cannot serve it, and returns automatically.

Pick the budget from your latency target. A service promising p99 under 200 ms cannot afford 250 ms of lag, so that is the threshold; a batch service can tolerate seconds. Smooth it over a few samples if your workload is bursty, so a single garbage-collection pause does not flap the instance out of the pool.

Lag also belongs in your metrics, not only the probe — see exporting Prometheus metrics from asyncio. The probe answers "now"; the metric shows the trend that predicted it.

Verify: a deliberate time.sleep() in a handler makes readiness fail and recover on its own.

Event-loop lag as a readiness signal 3 bars comparing idle with the others. Event-loop lag as a readiness signal idle 0 ms: ready just after a 0.6 s block 587 ms: not ready after recovery 1 ms: ready Lag is measured by a task that sleeps for a known interval and reports the overshoot. A loop that cannot answer within its budget should not be sent more traffic.

3. Bound every dependency check

Readiness may check dependencies, but a check that hangs is worse than one that fails — the probe times out, the orchestrator treats it as failure, and you have coupled your restart behaviour to someone else's latency. Every check gets its own timeout, and they run concurrently:

async def check(name: str, probe) -> dict:
    started = time.perf_counter()
    try:
        async with asyncio.timeout(0.5):
            await probe()
        return {"ok": True, "ms": round((time.perf_counter() - started) * 1000, 1)}
    except TimeoutError:
        return {"ok": False, "error": "timeout"}
    except Exception as exc:
        return {"ok": False, "error": type(exc).__name__}

Measured across three dependencies: a healthy one returned {"ok": True, "ms": 10.2}, a hanging one returned {"ok": False, "error": "timeout"} after 500.6 ms — bounded, as designed — and a refusing one returned {"ok": False, "error": "ConnectionError"} immediately.

Two refinements matter in production. Cache the results for a few seconds: probes run every few seconds per instance, and an uncached check multiplies that by your fleet size into real load on the dependency. And distinguish required from optional dependencies — a missing cache should not remove an instance that can still serve from the database.

Verify: the probe's own response time stays under its timeout even when every dependency hangs.

4. Flip readiness first during shutdown

Readiness is not only an input to the load balancer; it is the first step of a graceful shutdown. On SIGTERM, report not-ready immediately and keep serving:

async def shutdown() -> None:
    state.ready = False                                # stop new traffic
    await asyncio.sleep(DRAIN_DELAY)                   # let the LB notice
    await server.shutdown()                            # stop accepting
    await finish_in_flight()

The DRAIN_DELAY exists because the load balancer learns about the change on its own schedule — typically one or two probe intervals. Exiting as soon as SIGTERM arrives means requests routed in that window are refused, which is the most common cause of 502s during an otherwise clean deploy. Five to fifteen seconds covers most platforms; match it to your probe interval times the failure threshold.

The liveness probe must keep passing throughout, or the orchestrator will kill the process mid-drain and undo the whole exercise. Graceful shutdown and signals covers the rest of the sequence.

Verify: during a rolling deploy, no request is refused by an instance that is shutting down.

Readiness during a rolling deploy 4 lanes over time. Readiness during a rolling deploy readiness ready reporting 503 new requests still arriving none in-flight work finishing process running exits time → The gap between readiness turning false and traffic stopping is the load balancer, not you.

5. Keep the probe cheap and separate

Probe endpoints run constantly, so they must be trivial: no authentication, no logging per request, no database query on the request path, no ORM. Read pre-computed state — the lag sampler and the cached dependency results — and serialise a small JSON object.

Where the platform allows it, serve probes on a separate port from application traffic. That keeps them working when the main port is saturated or drained, keeps them off your public routing, and lets you apply a different timeout. Where it does not, at least exclude them from access logs and request metrics, or your p50 latency becomes a measurement of your health endpoint.

Verify: probe requests do not appear in application latency metrics and add no measurable load.

Where does this check belong? A decision on What does failing mean with 3 outcomes. Where does this check belong? What does failing mean? only a restart fixes it liveness loop lag, deadlock traffic elsewhere is better readiness dependencies, capacity it is still booting startup probe caches, migrations If a restart would not help, the check does not belong in the liveness probe.

Verification

Probes are correct when:

  • Liveness depends on nothing external and fails only for conditions a restart fixes.
  • Readiness reflects loop lag against a budget derived from your latency target.
  • Every dependency check is bounded and the results are cached.
  • Readiness turns false on SIGTERM, with a drain delay before the process stops accepting.
  • Probes are cheap: no queries, no logs, ideally a separate port.
  • Both probes are exercised in staging, including the failure paths.

Pitfalls & edge cases

  • Checking dependencies in liveness. One dependency outage restarts the entire fleet, usually while it is the least able to recover.
  • A probe that cannot run. If the loop is fully blocked, the probe itself does not respond — which is why liveness has a timeout and is allowed to fail that way.
  • Probe timeouts shorter than the check timeout. The orchestrator gives up while your handler is still waiting; keep the internal budget well inside the external one.
  • Flapping on a single sample. One GC pause should not remove an instance; average a few lag samples or require consecutive failures.
  • Exiting immediately on SIGTERM. Without a drain delay, in-flight and just-routed requests are refused.
  • Reporting ready during startup. Until caches and connections are warm, an instance receiving full traffic will fail; use the startup probe or a ready flag set at the end of initialisation.

Frequently Asked Questions

What is the difference between a liveness and a readiness probe?

Liveness asks whether the process is unrecoverable, and failing it restarts the container, so it must not depend on anything external. Readiness asks whether this instance should receive traffic now, and failing it just removes the instance from the load balancer — that is where dependency checks, capacity limits and shutdown state belong.

How do I measure event loop lag for a health check?

Run a task that records the time, sleeps a known interval, and reports the overshoot: (elapsed - interval). Verified here, a handler blocking for 0.6 s produced 587 ms of lag and the readiness probe returned 503, recovering to 1 ms and 200 on its own once the loop was free.

Should a readiness probe check the database?

Usually yes, with a short timeout and a cached result, since an instance that cannot reach its database should not receive traffic. Never put the same check in the liveness probe, where a database outage would restart every instance at once.

Why do I get 502s during deploys even with graceful shutdown?

Because the process stops accepting connections before the load balancer notices it is unready. Set readiness to false on SIGTERM, then wait a drain delay of one or two probe intervals before shutting the server down, so in-flight and just-routed requests still complete.

Should health endpoints be on a separate port?

Where the platform supports it, yes. A separate port keeps probes working when the application port is saturated or draining, keeps them off public routing, and keeps their requests out of your application's latency metrics and access logs.