Graceful Shutdown & Signal Handling

Every deployment is a shutdown. Kubernetes sends SIGTERM, waits out a grace period, then sends SIGKILL; systemd does the same with TimeoutStopSec; a developer presses Ctrl-C. What happens in that window decides whether a rollout is invisible to users or shows up as a spike of 502s, half-written database rows, and messages acknowledged but never processed. In an async service the window is particularly easy to get wrong, because "in flight" means dozens of tasks in arbitrary states, and the naive response — cancel everything immediately — is barely better than the SIGKILL you were trying to avoid.

A correct shutdown is a sequence, and the order is not negotiable: stop accepting new work, let existing work finish under a deadline, cancel whatever remains, release external resources, and only then exit. This section covers installing signal handlers that run on the loop, structuring the drain so it cannot hang, sizing the deadline against your orchestrator's grace period, and closing the resources — connection pools, executors, async generators — that keep a process alive after the interesting work is done. The parent overview, Resilience, Cancellation & Error Handling, covers cancellation semantics; this section applies them to the one event every process experiences.

Scope of this section:

• Installing SIGTERM/SIGINT handlers with loop.add_signal_handler() and why signal.signal() is wrong here.
• The shutdown sequence: stop intake, drain, cancel, close, exit — and what breaks when it is reordered.
• Budgeting the drain against the orchestrator's grace period, with a hard cap.
• Closing pools, executors and async generators so the process actually exits.
• Readiness versus liveness during shutdown, and the load-balancer race that causes deploy-time 502s.

Architectural principles

• Stop intake first. Draining while still accepting requests never terminates. Close listeners, stop consuming from the broker, and fail readiness probes before awaiting anything.
• The drain must be bounded. Every wait during shutdown needs a deadline. An unbounded await queue.join() turns one stuck item into a SIGKILL and a corrupted deploy.
• Cancellation is the fallback, not the plan. Tasks that finish on their own leave consistent state; cancelled tasks leave whatever their finally blocks manage. Give work a chance to finish, then cancel with intent.
• The grace period is a hard budget. Your total shutdown must fit inside it with margin. If the orchestrator allows 30 s, plan for 20 and cap every phase.
• Signal handlers belong on the loop. loop.add_signal_handler() schedules the callback as a loop callback; signal.signal() interrupts arbitrary bytecode and cannot safely touch loop state.

Execution model: what a signal does to a running loop

At the OS level a signal interrupts the main thread. CPython's default handling sets a flag and runs the Python-level handler at the next bytecode boundary — which, inside a running event loop, could be in the middle of a callback, halfway through mutating a queue's internals. That is why signal.signal() handlers may only do trivial, reentrancy-safe work, and why calling asyncio APIs from one is unsafe.

loop.add_signal_handler() avoids the problem entirely. On Unix, asyncio uses signal.set_wakeup_fd() to turn a signal into a byte written to a self-pipe that the selector is already watching. The signal itself does almost nothing; on the next loop iteration the selector reports the pipe readable and your callback runs as an ordinary loop callback, with the loop in a consistent state and full access to tasks, queues and create_task(). The cost is one file descriptor and the requirement that the loop be running.

From there, shutdown is ordinary async code. The usual shape is a shutdown Event that the handler sets and that long-lived tasks select on, plus a coordinator that runs the phases. What must not happen is the handler doing the work itself: a handler that awaits cannot (it is a plain callback), and a handler that cancels every task immediately skips the drain phase entirely — the single most common cause of "we lose in-flight requests on every deploy".

Pattern catalogue

The shutdown event

The simplest coordination: one Event that every long-lived task watches.

import asyncio
import signal

shutdown = asyncio.Event()


def install_handlers() -> None:
    loop = asyncio.get_running_loop()
    for sig in (signal.SIGTERM, signal.SIGINT):
        loop.add_signal_handler(sig, shutdown.set)      # runs as a loop callback


async def consumer(queue: asyncio.Queue) -> None:
    while not shutdown.is_set():
        try:
            async with asyncio.timeout(1.0):
                item = await queue.get()
        except TimeoutError:
            continue                                     # re-check the flag
        await process(item)

The timeout inside the loop is what makes the flag observable: without it, a consumer parked on queue.get() never notices the event. On Windows, add_signal_handler() is not supported — use signal.signal() with a handler that only calls loop.call_soon_threadsafe(shutdown.set).

Stop intake, then drain

The phase order that makes shutdown terminate.

async def shutdown_sequence(server: asyncio.Server, queue: asyncio.Queue,
                            grace: float = 20.0) -> None:
    server.close()                       # 1. stop accepting new connections
    await server.wait_closed()
    READY.clear()                        # 2. fail readiness so the LB stops routing
    try:
        async with asyncio.timeout(grace):
            await queue.join()           # 3. drain what is already accepted
    except TimeoutError:
        log.warning("drain incomplete after %.0fs: %d item(s) left", grace, queue.qsize())

Every step before the drain removes a source of new work; every step after it releases resources. Reversing the first two produces the classic deploy-time 502: the process stops serving while the load balancer is still routing to it.

Cancel the remainder, deliberately

Whatever has not finished by the deadline gets cancelled — with a bounded wait for the cancellation itself.

async def cancel_remaining(exclude: set[asyncio.Task] | None = None,
                           budget: float = 5.0) -> None:
    exclude = (exclude or set()) | {asyncio.current_task()}
    tasks = [t for t in asyncio.all_tasks() if t not in exclude]
    for task in tasks:
        task.cancel()
    if not tasks:
        return
    done, pending = await asyncio.wait(tasks, timeout=budget)
    for task in pending:                 # ignored the cancellation: shielded or blocked
        log.error("task %s did not stop within %.0fs", task.get_name(), budget)

A task can refuse cancellation — by shielding, by catching CancelledError, or by blocking the loop — so the second timeout is essential. Logging the offenders by name is how the next release fixes them; naming tasks pays for itself here, as described in inspecting pending tasks.

TaskGroup-scoped shutdown

Structured concurrency makes the drain implicit: leaving the TaskGroup block waits for its children.

async def main() -> None:
    install_handlers()
    async with asyncio.TaskGroup() as tg:
        tg.create_task(serve(), name="server")
        tg.create_task(worker(), name="worker")
        await shutdown.wait()                    # wake on SIGTERM
        for task in tg._tasks:                   # ask children to wind down
            task.cancel()
    # the TaskGroup has now awaited every child, including cancellation
    await close_resources()

The group's exit is the drain, which is why TaskGroup-based structure simplifies shutdown so much. The caveat: TaskGroup waits without a deadline, so a child that ignores cancellation blocks the exit — keep the outer asyncio.run() under a supervisor timeout if that risk is real.

Releasing external resources

Work finishing is not the same as the process being able to exit.

async def close_resources(pool, client, executor) -> None:
    async with asyncio.timeout(10):
        await asyncio.shutdown_asyncgens()      # close suspended async generators
        await client.aclose()                   # HTTP connections
        await pool.close()                      # database pool
    executor.shutdown(wait=True, cancel_futures=True)   # threads/processes

asyncio.run() calls shutdown_asyncgens() for you; a hand-rolled loop must do it or suspended generators' finally blocks never run. A ThreadPoolExecutor with wait=True is the classic reason a "finished" process hangs: non-daemon threads keep the interpreter alive, and a thread blocked on a socket read holds shutdown for as long as its own timeout allows.

Resource boundaries

Shutdown is a budget-allocation problem: the orchestrator gives you a fixed window, and every phase must fit inside it.

Phase	Typical budget (30 s grace)	What overruns it
Fail readiness, wait for the LB	2–5 s	LB health-check interval longer than the wait
Stop listeners / broker consumption	< 1 s	Broker client without an async close
Drain in-flight work	10–15 s	One long request; a join() that never completes
Cancel and await cancellation	3–5 s	Tasks that swallow CancelledError
Close pools, clients, executors	3–5 s	executor.shutdown(wait=True) on a blocked thread
Margin	3–5 s	—

Two rules keep it honest. Total the phases and compare with the grace period before deploying — if the sum exceeds it, the orchestrator's SIGKILL will land mid-drain. And subtract the load balancer's de-registration delay from the front: a 5 s health-check interval means up to 5 s of new requests arriving after SIGTERM, which is exactly the window the readiness flip covers.

Integrated production example

An HTTP-ish service with a queue-backed worker pool, a two-phase signal handler (second signal forces exit), a bounded drain, and full resource cleanup.

import asyncio
import contextlib
import logging
import signal
import time

log = logging.getLogger("service")

GRACE = 20.0            # must fit inside the orchestrator's grace period
CANCEL_BUDGET = 5.0
CLOSE_BUDGET = 5.0


class Service:
    def __init__(self) -> None:
        self.shutdown = asyncio.Event()
        self.forced = False
        self.ready = True
        self.queue: asyncio.Queue = asyncio.Queue(maxsize=1000)
        self.server: asyncio.Server | None = None
        self.inflight = 0

    # ---- lifecycle ------------------------------------------------------
    def install_signals(self) -> None:
        loop = asyncio.get_running_loop()
        for sig in (signal.SIGTERM, signal.SIGINT):
            loop.add_signal_handler(sig, self._on_signal, sig)

    def _on_signal(self, sig: signal.Signals) -> None:
        if self.shutdown.is_set():                 # second signal: stop waiting
            log.warning("second %s — forcing immediate shutdown", sig.name)
            self.forced = True
            for task in asyncio.all_tasks():
                task.cancel()
            return
        log.info("%s received — starting graceful shutdown", sig.name)
        self.ready = False                         # readiness probe fails from now on
        self.shutdown.set()

    async def worker(self, name: str) -> None:
        while True:
            try:
                async with asyncio.timeout(0.5):
                    item = await self.queue.get()
            except TimeoutError:
                if self.shutdown.is_set() and self.queue.empty():
                    return                          # nothing left for us to do
                continue
            self.inflight += 1
            try:
                await self.handle(item)
            except asyncio.CancelledError:
                log.warning("item %s cancelled mid-flight", item)
                raise
            except Exception:
                log.exception("item %s failed", item)
            finally:
                self.inflight -= 1
                self.queue.task_done()

    async def handle(self, item) -> None:
        await asyncio.sleep(0.05)                   # stand-in for real work

    # ---- shutdown -------------------------------------------------------
    async def drain(self) -> None:
        started = time.monotonic()
        if self.server is not None:
            self.server.close()                     # stop accepting connections
            await self.server.wait_closed()
        try:
            async with asyncio.timeout(GRACE):
                await self.queue.join()             # let accepted work finish
            log.info("drained cleanly in %.1fs", time.monotonic() - started)
        except TimeoutError:
            log.warning("drain timed out: %d queued, %d in flight",
                        self.queue.qsize(), self.inflight)

    async def stop_tasks(self, keep: set[asyncio.Task]) -> None:
        tasks = [t for t in asyncio.all_tasks() if t not in keep and t is not asyncio.current_task()]
        for task in tasks:
            task.cancel()
        if tasks:
            _done, pending = await asyncio.wait(tasks, timeout=CANCEL_BUDGET)
            for task in pending:
                log.error("task %s ignored cancellation", task.get_name())

    async def close(self, pool=None, client=None, executor=None) -> None:
        with contextlib.suppress(TimeoutError):
            async with asyncio.timeout(CLOSE_BUDGET):
                await asyncio.shutdown_asyncgens()
                if client is not None:
                    await client.aclose()
                if pool is not None:
                    await pool.close()
        if executor is not None:
            executor.shutdown(wait=True, cancel_futures=True)


async def main() -> None:
    svc = Service()
    svc.install_signals()
    workers = {asyncio.create_task(svc.worker(f"w{i}"), name=f"worker.{i}")
               for i in range(8)}
    producer = asyncio.create_task(ingest(svc.queue), name="ingest")

    await svc.shutdown.wait()                       # blocks until SIGTERM/SIGINT
    producer.cancel()                               # 1. stop intake
    with contextlib.suppress(asyncio.CancelledError):
        await producer
    if not svc.forced:
        await svc.drain()                           # 2. finish accepted work
    await svc.stop_tasks(keep=set())                # 3. cancel the remainder
    await svc.close()                               # 4. release resources
    log.info("shutdown complete")


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(main())

Three properties make this survivable in production. The second signal is a distinct, documented behaviour — operators expect a second Ctrl-C to mean "now", and without it they reach for SIGKILL. Every phase is bounded, so the total is predictable and can be checked against the grace period. And readiness flips first, before any awaiting, which is what stops the load balancer sending requests into a process that has begun winding down.

Diagnostic Hook — prove the drain in staging, not in production

Log four things at shutdown: time from signal to readiness=false, time to drain (and items left if it timed out), the names of any tasks that ignored cancellation, and total shutdown duration. Export the total as a histogram — it is the number that must stay comfortably under the orchestrator's grace period, and it grows silently as the service gains dependencies. In staging, deploy under load and check for a single dropped or duplicated request; in production, alert if shutdown duration exceeds 70% of the grace period, because at that point you are one slow dependency away from SIGKILL mid-write.

Failure modes

Failure mode	Root cause	Detection	Fix
502s during every deploy	Listener closed before the load balancer de-registered	Errors cluster in the first seconds of a rollout	Fail readiness first, wait one health-check interval, then close
Shutdown never completes, SIGKILL follows	Unbounded queue.join() or await in the drain	Process exits with 137; no "shutdown complete" log	Wrap every phase in asyncio.timeout()
In-flight work lost	Handler cancels all tasks immediately on SIGTERM	Duplicate or missing messages after a deploy	Cancel only after the drain deadline expires
Process hangs after work finishes	ThreadPoolExecutor with wait=True on a blocked thread	Loop is idle; process still alive	Give thread work its own timeout; cancel_futures=True
Async generator finally never runs	Custom loop without shutdown_asyncgens()	Files or connections left open after exit	Use asyncio.run(), or call it explicitly
Signal ignored under load	Handler installed with signal.signal() touching loop state	Occasional corruption or missed shutdown	Use loop.add_signal_handler()
Second Ctrl-C does nothing	Handler only sets the flag, with no escalation	Operators reach for kill -9	Make the second signal force cancellation

Frequently Asked Questions

How do I handle SIGTERM in an asyncio application?

Install the handler on the loop with loop.add_signal_handler(signal.SIGTERM, callback). The callback runs as an ordinary loop callback, so it can safely touch loop state — typically setting an asyncio.Event that the rest of the service watches. Avoid signal.signal() here: it interrupts arbitrary bytecode and cannot safely call asyncio APIs. On Windows, where add_signal_handler is unavailable, use signal.signal() with a handler that only calls loop.call_soon_threadsafe.

What is the correct order for graceful shutdown?

Fail readiness so the load balancer stops routing, stop accepting new work (close listeners, stop consuming from the broker), drain in-flight work under a deadline, cancel whatever remains and wait a bounded time for it, then close pools, clients and executors. Reversing the first two steps causes deploy-time 502s; skipping the deadline causes SIGKILL mid-write.

How long should the drain timeout be?

Shorter than the orchestrator's grace period minus every other phase. With a 30 second Kubernetes terminationGracePeriodSeconds, budget roughly 5 seconds for load-balancer de-registration, 10-15 for the drain, 5 for cancellation, and keep 3-5 in reserve. Measure real shutdown duration as a histogram and alert when it exceeds about 70% of the grace period.

Why does my asyncio process hang after all the work is finished?

Usually a non-daemon thread or an unclosed resource. ThreadPoolExecutor.shutdown(wait=True) blocks until every worker thread returns, and a thread blocked on a socket read holds the process for as long as its own timeout allows. Suspended async generators and open connection pools have the same effect. Close them explicitly under a timeout, and pass cancel_futures=True to the executor.

Should SIGINT and SIGTERM be handled the same way?

Broadly yes — both should trigger the same graceful sequence — but treat a repeated signal as escalation. The first sets the shutdown event; the second cancels everything immediately. Operators expect a second Ctrl-C to mean "stop now", and providing it keeps them from reaching for SIGKILL, which gives you no cleanup at all.

Related

• Resilience, Cancellation & Error Handling — up to the overview for cancellation semantics and error propagation.
• Handling SIGTERM in asyncio services — the handler itself, including the Windows path and container specifics.
• Draining in-flight requests before shutdown — the load-balancer race and how to bound the drain.
• Shutting down async generators and executors cleanly — the cleanup phase that decides whether the process exits.
• Cancellation Patterns — what cancelling a task actually does to work in progress.