Skip to content

Handling SIGTERM in Asyncio Services

The container is killed 30 seconds into every rollout and the logs show nothing after "starting". No shutdown message, no drain, no cleanup — the process simply stops existing. The usual cause is not a bug in the shutdown logic but a signal that never reached it: a signal.signal() handler that cannot safely touch the loop, a shell wrapper that swallowed SIGTERM, or a Python process running as PID 1 in a container, where the kernel's default-action protection means an unhandled SIGTERM is silently discarded and the process runs until SIGKILL.

This guide installs signal handling that actually works: loop.add_signal_handler() for the Unix path, the call_soon_threadsafe bridge for Windows, escalation when a second signal arrives, PID 1 and entrypoint pitfalls in containers, and a test that proves the handler runs before you find out during a deploy.

Prerequisites

  • Python 3.11+; standard library only.
  • A way to send signals to the process — a shell for local testing, kubectl delete pod or docker stop for the container path.
  • The graceful shutdown and signal handling overview for the full shutdown sequence this handler starts.
Two ways to catch a signal Two columns contrasting the two signal-handling mechanisms. Two ways to catch a signal signal.signal() interpreter context Runs between bytecodes Loop state may be mid-mutation Only call_soon_threadsafe is safe Required on Windows loop.add_signal_handler() loop context Signal writes to a self-pipe Callback runs as a loop callback Full access to tasks and events Unix only Both deliver the signal; only one leaves the loop in a state you can touch.

1. Install the handler on the loop

loop.add_signal_handler() routes the signal through asyncio's self-pipe, so your callback runs as an ordinary loop callback with the loop in a consistent state.

import asyncio
import signal

shutdown = asyncio.Event()


async def main() -> None:
    loop = asyncio.get_running_loop()
    for sig in (signal.SIGTERM, signal.SIGINT):
        loop.add_signal_handler(sig, shutdown.set)
    await serve_until(shutdown)


asyncio.run(main())

The handler must be installed inside the running loop — get_running_loop() fails otherwise, and a handler registered against a different loop never fires. Note that the callback is a plain function, not a coroutine: to start async work from it, use loop.create_task() (and keep a reference to the task). Verify: run the service, send kill -TERM <pid> from another shell, and confirm the shutdown path logs before the process exits.

2. Escalate on a second signal

Operators expect a second Ctrl-C to mean "now". Without escalation, they reach for SIGKILL and you lose all cleanup.

import asyncio
import logging
import signal

log = logging.getLogger("shutdown")


class SignalState:
    def __init__(self) -> None:
        self.event = asyncio.Event()
        self.forced = False

    def handle(self, sig: signal.Signals) -> None:
        if self.event.is_set():
            log.warning("second %s — cancelling everything now", sig.name)
            self.forced = True
            for task in asyncio.all_tasks():
                task.cancel()
            return
        log.info("%s — graceful shutdown started", sig.name)
        self.event.set()


def install(state: SignalState) -> None:
    loop = asyncio.get_running_loop()
    for sig in (signal.SIGTERM, signal.SIGINT):
        loop.add_signal_handler(sig, state.handle, sig)

Passing sig as an argument (rather than a lambda per signal) keeps the log accurate about which signal arrived, which matters when diagnosing whether an orchestrator or a human stopped the process. Verify: press Ctrl-C twice; the first should start the drain, the second should terminate within a second or two.

3. Handle the Windows path

add_signal_handler() raises NotImplementedError on Windows. The portable fallback marshals the signal onto the loop by hand.

import asyncio
import signal
import sys


def install_portable(state: SignalState) -> None:
    loop = asyncio.get_running_loop()
    signals = (signal.SIGTERM, signal.SIGINT)
    if sys.platform == "win32":
        for sig in signals:
            signal.signal(                       # runs in the interpreter's signal context
                sig,
                lambda s, _frame: loop.call_soon_threadsafe(state.handle, signal.Signals(s)),
            )
    else:
        for sig in signals:
            loop.add_signal_handler(sig, state.handle, sig)

call_soon_threadsafe() is the only asyncio API safe to call from a signal.signal() handler; it appends to the loop's callback queue and wakes the loop through its self-pipe. Do not touch events, queues or tasks directly in that handler. Note also that Windows delivers CTRL_C_EVENT/CTRL_BREAK_EVENT rather than Unix signals for console interrupts, so a service that must stop cleanly under Windows service management usually needs a platform-specific path anyway. Verify: on Windows, Ctrl-C in a console should run the same shutdown path as SIGTERM on Linux.

Signals as an escalation ladder Four states showing graceful and forced shutdown paths. Signals as an escalation ladder running graceful forced exited first SIGTERM second signal tasks cancelled drain completes The second signal is what stops operators from reaching for SIGKILL.

4. Make sure the signal reaches Python in a container

Three container-level problems stop the handler from ever running, and all three look identical from inside.

# WRONG: the shell is PID 1, and `sh -c` does not forward SIGTERM to the child
CMD python -m myservice

# RIGHT: exec form — python is PID 1 and receives the signal directly
CMD ["python", "-m", "myservice"]

# PID 1 has no default signal disposition: an unhandled SIGTERM is DISCARDED.
# Installing a handler is what makes SIGTERM meaningful for PID 1 at all.

The shell form (CMD python ...) runs /bin/sh -c, which becomes PID 1 and does not forward signals to its child, so SIGTERM reaches the shell and never the interpreter. The exec form fixes that. Separately, the kernel does not apply default actions to PID 1, so a process without an explicit handler ignores SIGTERM entirely and only dies at SIGKILL — the exact "nothing in the logs, killed after the grace period" symptom. If you need reaping of child processes as well, use an init shim such as tini or docker run --init. Verify: docker stop (which sends SIGTERM, then SIGKILL after 10 s) should produce your shutdown logs and exit well before the kill.

5. Turn the signal into the shutdown sequence

The handler's only job is to start the sequence; the sequence itself is ordinary async code.

import asyncio
import contextlib


async def main() -> None:
    state = SignalState()
    install_portable(state)
    ready = Readiness()                          # your probe endpoint reads this

    async with asyncio.TaskGroup() as tg:
        server = tg.create_task(serve(), name="server")
        workers = [tg.create_task(worker(i), name=f"worker.{i}") for i in range(8)]

        await state.event.wait()                 # ---- signal arrives here ----
        ready.set_not_ready()                    # 1. stop the load balancer
        server.cancel()                          # 2. stop accepting
        with contextlib.suppress(asyncio.CancelledError):
            await server
        if not state.forced:
            with contextlib.suppress(TimeoutError):
                async with asyncio.timeout(15):  # 3. bounded drain
                    await drain()
        for task in workers:                     # 4. cancel the remainder
            task.cancel()
    await close_resources()                      # 5. release externals

Keeping the sequence in main() rather than the handler is what makes it testable and bounded — a callback cannot await, so any handler that "does the shutdown" is either doing it wrong or spawning an untracked task. Verify: the log shows readiness cleared, then the drain, then cancellation, in that order, on every signal.

6. Test it without deploying

Signals are testable in-process, which is the difference between knowing the handler works and hoping.

import asyncio
import os
import signal


async def test_sigterm_triggers_shutdown() -> None:
    state = SignalState()
    install_portable(state)
    assert not state.event.is_set()

    os.kill(os.getpid(), signal.SIGTERM)         # deliver to ourselves
    await asyncio.sleep(0)                       # let the loop run the callback
    assert state.event.is_set(), "handler did not run"

    os.kill(os.getpid(), signal.SIGTERM)         # escalation path
    await asyncio.sleep(0)
    assert state.forced


asyncio.run(test_sigterm_triggers_shutdown())

The single await asyncio.sleep(0) is the point: it yields to the loop so the queued signal callback runs, which is exactly the mechanism that makes loop signal handlers safe. Add a second test that runs the full sequence with a stubbed drain and asserts the phase order from the log. Verify: the test fails if you swap add_signal_handler for a signal.signal() handler that mutates the event directly — proof it is testing the plumbing, not just the flag.

The signal never arrived — why? A decision diamond on container PID 1 with three outcome cards. The signal never arrived — why? Who is PID 1 in the container? /bin/sh via shell CMD shell does not forward use the exec form of CMD python, no handler kernel discards SIGTERM install an explicit handler python with a handler signal is delivered look at the handler itself Two of the three failures are in the Dockerfile, not the Python.

Verification

  • The handler runs. kill -TERM produces a shutdown log line within milliseconds, locally and in the container.
  • Escalation works. A second signal shortens shutdown to about a second, with cancellation logged.
  • The container exits early. docker stop / pod deletion completes well inside the grace period, with exit code 0 rather than 137.

Pitfalls and edge cases

  • signal.signal() touching loop state. It runs between bytecodes and can observe half-mutated structures. Only call_soon_threadsafe() is safe from there.
  • PID 1 with no handler. The kernel discards SIGTERM for PID 1 without an explicit handler; the process survives until SIGKILL.
  • Shell-form CMD. /bin/sh -c becomes PID 1 and does not forward signals. Use the exec form or an init shim.
  • Installing handlers before the loop runs. get_running_loop() requires a running loop; handlers bound to a different loop never fire.
  • Doing the work in the handler. A callback cannot await; spawning an untracked task from it means shutdown can be garbage collected mid-flight.
  • Forgetting SIGINT in production. Some orchestrators and process supervisors send SIGINT; handle both.
  • Signals in a non-main thread. Only the main thread receives signals; a loop running in a worker thread must be signalled indirectly through call_soon_threadsafe.
  • SIGKILL/SIGSTOP are not catchable. Nothing you write runs for those — which is why the graceful path must fit inside the grace period.

Frequently Asked Questions

Why is loop.add_signal_handler better than signal.signal in asyncio?

add_signal_handler routes the signal through asyncio's self-pipe, so the callback runs as a normal loop callback with the loop in a consistent state and full access to tasks, events and create_task. signal.signal handlers run between arbitrary bytecodes, possibly mid-mutation of loop internals, where the only safe asyncio call is loop.call_soon_threadsafe.

Why does my containerised Python service ignore SIGTERM?

Two common causes. The Dockerfile uses the shell form of CMD, so /bin/sh is PID 1 and does not forward the signal to Python — switch to the exec form. Or Python is PID 1 with no handler installed, in which case the kernel discards SIGTERM entirely because default signal dispositions do not apply to PID 1; installing an explicit handler makes it meaningful.

How do I handle SIGTERM on Windows where add_signal_handler is unavailable?

Fall back to signal.signal() with a handler that does nothing but call loop.call_soon_threadsafe() to schedule your real callback on the loop. Be aware that Windows consoles deliver CTRL_C_EVENT and CTRL_BREAK_EVENT rather than Unix signals, and that services managed by the Windows service control manager need a platform-specific stop path.

Should a second SIGTERM force an immediate exit?

Yes, and it should be documented. The first signal starts the graceful sequence; the second cancels every task immediately. Operators expect a repeated interrupt to mean "stop now", and providing that escalation keeps them from sending SIGKILL, which gives you no cleanup at all.