Skip to content

Draining In-Flight Requests Before Shutdown

Every rollout produces the same small spike of 502s, always in the first seconds, always on the pods being replaced. The application logs look clean: it received SIGTERM, it shut down, it exited zero. The errors are real all the same, and they come from a race nobody owns — the load balancer is still routing to a process that has already closed its listener. Meanwhile a second, quieter failure is happening on the other side of the same window: requests that were accepted get cancelled mid-flight, leaving a payment charged but not recorded, or a message acknowledged but not processed.

Draining is the fix for both, and it is mostly about ordering and deadlines rather than clever code. This guide covers flipping readiness before touching the listener, waiting out the load balancer's de-registration, tracking what is genuinely in flight, bounding the drain so it cannot outlive the grace period, and deciding what happens to work that does not finish in time.

Prerequisites

The load-balancer race that makes 502s Three lanes contrasting wrong and right shutdown ordering against arriving requests. The load-balancer race that makes 502s wrong order LB still routing: 502s right order LB de-registers listener closed requests still arriving — and served seconds after SIGTERM The de-registration wait is the whole fix; everything else is bookkeeping.

1. Fail readiness before you close anything

The load balancer stops sending traffic when its health check fails, and it learns that on its own schedule. Flip readiness first, then wait for that schedule.

import asyncio

class Readiness:
    def __init__(self) -> None:
        self._ready = True

    def not_ready(self) -> None:
        self._ready = False

    async def probe(self, request):                 # GET /readyz
        return 200 if self._ready else 503


async def begin_shutdown(ready: Readiness, lb_interval: float = 5.0) -> None:
    ready.not_ready()                               # 1. probes start failing
    await asyncio.sleep(lb_interval)                # 2. let the LB notice

Keep liveness (/healthz) passing throughout: a failing liveness probe tells Kubernetes to kill the pod, which is precisely what you are trying to avoid. Only readiness should fail. The sleep is the de-registration wait; size it from the health check's periodSeconds × failureThreshold, not from intuition. Verify: during a rolling deploy, the count of requests arriving after SIGTERM should fall to zero within the configured wait; if requests still arrive later, the wait is too short.

2. Stop accepting, keep serving

Closing the listener refuses new connections while existing ones continue.

async def stop_accepting(server: asyncio.Server) -> None:
    server.close()                     # stop accepting; established connections stay
    await server.wait_closed()

Server.close() closes the listening sockets only — connections already established are untouched, which is exactly the semantic a drain needs. For a broker consumer the equivalent is unsubscribing or pausing the fetch loop while leaving in-flight message handlers running. Verify: after this call, new connections are refused (ECONNREFUSED) while an in-flight request still completes normally.

3. Track what is actually in flight

You cannot drain what you cannot count. A single counter, incremented at entry and decremented in finally, is enough — and its finally is what makes it correct under cancellation.

import asyncio


class InFlight:
    def __init__(self) -> None:
        self._count = 0
        self._idle = asyncio.Event()
        self._idle.set()

    def __enter__(self) -> "InFlight":
        self._count += 1
        self._idle.clear()
        return self

    def __exit__(self, *exc) -> None:
        self._count -= 1
        if self._count == 0:
            self._idle.set()

    async def wait_idle(self) -> None:
        await self._idle.wait()

    @property
    def count(self) -> int:
        return self._count


INFLIGHT = InFlight()


async def handle(request):
    with INFLIGHT:                       # sync context manager: no await inside
        return await process(request)

An Event beats polling count in a loop: wait_idle() costs nothing while work is running and resolves the instant the last handler exits. Use a synchronous context manager so the decrement runs even if the handler is cancelled between awaits. Verify: with 50 concurrent requests, INFLIGHT.count matches your server's own concurrency metric and returns to zero when the load stops.

4. Drain under a deadline

An unbounded drain is how a single stuck request turns into SIGKILL and a truncated write.

import asyncio
import logging

log = logging.getLogger("drain")


async def drain(inflight: InFlight, queue: asyncio.Queue | None = None,
                budget: float = 15.0) -> bool:
    """Return True if everything finished within the budget."""
    try:
        async with asyncio.timeout(budget):
            await inflight.wait_idle()               # requests already accepted
            if queue is not None:
                await queue.join()                   # queued background work
        log.info("drain complete")
        return True
    except TimeoutError:
        log.warning("drain incomplete: %d request(s) in flight, %s queued",
                    inflight.count, queue.qsize() if queue else 0)
        return False

The budget is a slice of the orchestrator's grace period, not a guess: total every phase and keep several seconds in reserve. Returning a boolean rather than raising lets the caller decide whether to cancel the stragglers or (for a queue backed by a broker) simply exit and let redelivery handle them. Verify: hold one request open artificially and confirm the drain returns False after exactly budget seconds with the straggler counted in the log.

What each phase removes A pipeline of four drain phases and what each eliminates. What each phase removes readiness fails removes new routing listener closed removes new connections drain removes accepted work cancel removes the stragglers Each phase eliminates one source of work; the order is what makes the drain terminate.

5. Decide what happens to work that does not finish

The stragglers need a policy, and it depends on whether the work is replayable.

async def finish_or_cancel(inflight: InFlight, tasks: set[asyncio.Task],
                           drained: bool) -> None:
    if drained:
        return
    for task in tasks:
        task.cancel()                                # cooperative: their finally runs
    _done, pending = await asyncio.wait(tasks, timeout=5.0)
    for task in pending:
        log.error("task %s ignored cancellation — data may be inconsistent",
                  task.get_name())

For broker-backed work, cancelling is safe provided you have not acknowledged the message: redelivery replays it elsewhere. For HTTP requests there is no replay, so the honest options are to cancel (the client sees a dropped connection and retries, if it is idempotent) or to extend the budget. What you must not do is acknowledge before processing — that converts a cancelled task into silent data loss regardless of how careful the shutdown is. Verify: kill a pod mid-request against a stub client and confirm the client's retry produces exactly one effect, not zero and not two.

6. Prove it with a load test through a real deploy

The only trustworthy verification is an end-to-end one.

# Terminal 1: constant load through the service's real ingress
hey -z 120s -c 50 https://service.example.internal/health-ish

# Terminal 2: roll the deployment mid-test
kubectl rollout restart deployment/my-service

# Success criteria for the whole window:
#   - zero non-2xx responses attributable to the rollout
#   - shutdown duration well under terminationGracePeriodSeconds
#   - exit code 0 (not 137, which means SIGKILL landed)

Watch three things together: the client's error count, the pod's termination reason, and your shutdown-duration log. A clean rollout has zero errors and exits voluntarily; exit code 137 means the grace period expired and every guarantee above it is void. Verify: repeat with the readiness flip removed and confirm errors appear — proof that the step you added is what prevents them.

The drain timed out — now what? A decision diamond about replayability with three outcome cards. The drain timed out — now what? Can this work be replayed? yes, broker-backed cancel safely redelivery replays it elsewhere no, HTTP in flight cancel and rely on retry only if the request is idempotent no, and not idempotent extend the budget or make the operation idempotent Replayability, not politeness, decides what a straggler deserves.

Verification

  • No rollout errors. A sustained load test across a rolling restart shows zero 502s or connection resets attributable to the deploy.
  • Voluntary exit. Pods terminate with exit code 0, and shutdown duration stays well below the grace period.
  • Nothing lost or duplicated. For queue-backed work, the count of processed items across a restart matches the count produced, with at-least-once semantics preserved.

Pitfalls and edge cases

  • Closing the listener before readiness fails. The load balancer keeps routing for one health-check interval, and every one of those requests is refused.
  • Failing liveness instead of readiness. That instructs the orchestrator to kill the pod immediately — the opposite of a drain.
  • Unbounded queue.join(). One poisoned item that never calls task_done() blocks shutdown until SIGKILL.
  • Counting in-flight work with an async context manager. If the decrement can be skipped by cancellation between awaits, the count never returns to zero.
  • Acknowledging broker messages before processing. Cancelled work then vanishes silently; acknowledge after the effect is durable.
  • Keep-alive connections holding requests open. Long-lived HTTP/1.1 or WebSocket connections do not drain by themselves; send a close frame or Connection: close during shutdown.
  • A drain budget larger than the grace period. Total every phase and compare with the orchestrator's setting before the next deploy, not after.

Frequently Asked Questions

Why do I still get 502s during a rolling deploy?

Almost always the ordering: the process closes its listener before the load balancer has stopped routing to it. Fail the readiness probe first, wait at least one health-check interval (periodSeconds times failureThreshold), and only then close the listener. Keep liveness passing throughout, since failing it tells the orchestrator to kill the pod immediately.

How do I know when all in-flight requests have finished?

Track them explicitly with a counter incremented on entry and decremented in a finally block, paired with an asyncio.Event that is set when the count reaches zero. Awaiting that event costs nothing while work is running and resolves the moment the last handler exits — far better than polling a counter in a sleep loop.

What should happen to requests that do not finish in the drain window?

It depends on replayability. Broker-backed work can be cancelled safely as long as the message has not been acknowledged, because redelivery replays it elsewhere. HTTP requests have no replay, so you either cancel (and rely on the client retrying an idempotent request) or extend the budget. Never acknowledge before the work is durable.

How long should the load-balancer de-registration wait be?

At least one full health-check cycle — periodSeconds multiplied by failureThreshold — plus a small margin. For a probe every 5 seconds with a threshold of 1, wait about 5-7 seconds after flipping readiness before closing the listener. Verify with a load test: requests arriving after that window mean the wait is too short.