Skip to content

Restarting Crashed Workers with a Supervisor Task

Throughput halves overnight and nobody notices until morning. The queue is draining, no errors are logged, the service is "up" — but four of the eight workers died hours ago, each on an unhandled exception inside its loop, and nothing was watching. This is the default behaviour of a fire-and-forget worker pool: create_task() returns a task nobody awaits, so when it raises, the exception sits unretrieved until garbage collection produces a warning that most log pipelines drop.

A supervisor fixes it: one task whose only job is to notice a worker finishing and start a replacement. The interesting parts are the guards around it — backoff so a systematically broken worker does not spin, a crash-loop detector so a poison message does not consume your CPU forever, and metrics so a pool that is quietly restarting every few seconds is visible rather than merely surviving.

Prerequisites

How a worker dies unnoticed Four lifelines tracing an unobserved worker failure. How a worker dies unnoticed queue worker task object logs get() an item handler raises exception stored, task done warning at GC — maybe Nothing awaits the task, so the failure surfaces late, quietly, or never.

1. Make worker death observable

Before restarting anything, ensure a dying worker is loud.

import asyncio
import logging

log = logging.getLogger("pool")


def report_death(task: asyncio.Task) -> None:
    if task.cancelled():
        return                                     # expected during shutdown
    exc = task.exception()
    if exc is not None:
        log.error("worker %s died: %r", task.get_name(), exc, exc_info=exc)


worker = asyncio.create_task(consume(queue), name="worker.0")
worker.add_done_callback(report_death)             # nothing is silent any more

add_done_callback retrieves the exception, which both logs it and suppresses the "exception was never retrieved" warning at GC time. Note the cancelled() check: shutdown cancels workers deliberately, and treating that as a crash produces a burst of false alarms on every deploy. Verify: raise inside a worker and confirm the error appears immediately, not minutes later at collection time.

2. Supervise from one place

A supervisor waits for any worker to finish and replaces it.

import asyncio


class Pool:
    def __init__(self, size: int, work) -> None:
        self.size, self.work = size, work
        self.workers: dict[asyncio.Task, int] = {}
        self.restarts = 0
        self.stopping = False

    def _spawn(self, index: int) -> asyncio.Task:
        task = asyncio.create_task(self.work(index), name=f"worker.{index}")
        self.workers[task] = index
        return task

    async def supervise(self) -> None:
        for i in range(self.size):
            self._spawn(i)
        while not self.stopping:
            done, _pending = await asyncio.wait(self.workers, timeout=1.0,
                                                return_when=asyncio.FIRST_COMPLETED)
            for task in done:
                index = self.workers.pop(task)
                if self.stopping or task.cancelled():
                    continue
                exc = task.exception()
                log.warning("worker %d exited (%r) — restarting", index, exc)
                self.restarts += 1
                self._spawn(index)

The timeout=1.0 keeps the supervisor responsive to stopping even when no worker is finishing, and re-waiting on a changed set each iteration is what allows replacements to be supervised too. Keep the supervisor itself trivial: it must never be the thing that crashes. Verify: kill a worker with task.cancel() from outside and confirm a replacement appears within a second and the restart counter increments.

3. Back off, and detect a crash loop

A worker that dies immediately on start will otherwise be restarted thousands of times per second.

import asyncio
import collections
import time


class BackoffPool(Pool):
    def __init__(self, size: int, work, window: float = 60.0, max_restarts: int = 10):
        super().__init__(size, work)
        self.window, self.max_restarts = window, max_restarts
        self.recent: collections.deque[float] = collections.deque()
        self.delay = 0.0

    def _crash_looping(self) -> bool:
        now = time.monotonic()
        self.recent.append(now)
        while self.recent and now - self.recent[0] > self.window:
            self.recent.popleft()
        return len(self.recent) > self.max_restarts

    async def _respawn(self, index: int) -> None:
        if self._crash_looping():
            log.critical("worker %d crash-looping: %d restarts in %.0fs — giving up",
                         index, len(self.recent), self.window)
            self.stopping = True                    # let the service fail visibly
            return
        self.delay = min(max(self.delay * 2, 0.1), 10.0)
        await asyncio.sleep(self.delay)             # backoff before replacing
        self._spawn(index)

The crash-loop guard is the part people skip and then regret: without it, a poison message or a missing environment variable produces a process that is technically alive, pinning a core and filling the log, which is worse than a clean crash the orchestrator can restart. Reset delay to zero after a worker survives some threshold (say a minute) so a single transient failure does not slow future recovery. Verify: make the worker raise immediately; restarts should slow to one every 10 s and then stop entirely with a critical log line.

Restart delay as a worker keeps failing Six bars showing restart backoff growth and the crash-loop cutoff. Restart delay as a worker keeps failing 1st restart 0.1 s 2nd 0.2 s 3rd 0.4 s 4th 0.8 s capped 10 s 11 in 60 s stop: crash loop Delays double up to a 10 s cap; more than ten restarts in a 60 s window halts the service. Backoff buys time for a transient fault; the ceiling stops a permanent one.

4. Do not lose the item the worker died holding

A crash between get() and task_done() leaves an item in flight forever.

import asyncio


async def consume(queue: asyncio.Queue, dlq: asyncio.Queue, index: int) -> None:
    while True:
        item = await queue.get()
        try:
            await handle(item)
        except asyncio.CancelledError:
            await queue.put(item)                  # shutdown: give it back
            raise
        except Exception as exc:
            item.attempts += 1
            if item.attempts >= 3:
                await dlq.put((item, repr(exc)))   # poison: stop retrying it
            else:
                await queue.put(item)              # transient: let a peer retry
            log.exception("item %s failed on worker %d", item.id, index)
        finally:
            queue.task_done()

Catching the exception inside the loop is strictly better than letting the worker die, because the worker keeps its place and the item is routed explicitly. Reserve worker death for failures that invalidate the worker itself — a lost database connection it owns, a corrupted local buffer — and let the supervisor handle those. The attempt cap plus a dead-letter queue is what stops one bad item from cycling through every worker in turn, as covered in implementing a dead-letter queue with asyncio. Verify: a handler that always raises should send its item to the dead-letter queue after three attempts, with queue.join() still able to complete.

5. Export restart metrics and alert on them

A pool that restarts constantly looks healthy from outside.

# Per pool:
#   pool_workers_alive          gauge     — should equal the configured size
#   pool_worker_restarts_total  counter   — rate() is the signal, not the total
#   pool_items_dead_lettered    counter   — poison messages detected
#
# Alert: rate(pool_worker_restarts_total[5m]) > 0.1   (one restart every 10 min)
#        pool_workers_alive < configured_size for 1m

The alive gauge catches the original scenario — a pool silently running at half strength — and the restart rate catches the opposite one, where the pool is nominally full but churning. Together they cover both ways a supervised pool can be quietly broken. Verify: kill workers deliberately in staging and confirm both signals move as expected and recover.

Two gauges, four situations A two-by-two matrix of pool health signals. Two gauges, four situations restarts flat restarts rising workers alive = size healthy churning — find the recurring fault workers alive < size silent deaths, no supervisor crash loop — check the guard Alive-below-size catches the silent failure; restart rate catches the noisy one.

Verification

  • A killed worker is replaced. The alive gauge dips and recovers within seconds, with a logged reason.
  • A broken worker does not spin. A worker that fails on start backs off to a bounded interval and then trips the crash-loop guard.
  • No work is lost. With handlers failing randomly, the queue still drains completely and failed items land in the dead-letter queue.

Pitfalls and edge cases

  • Fire-and-forget workers. Without a done-callback or a supervisor, a dead worker is silent until throughput tells you.
  • Restarting without backoff. A worker that fails at startup burns CPU restarting thousands of times a second.
  • No crash-loop ceiling. An alive-but-useless process is worse than a clean exit the orchestrator can act on.
  • Counting shutdown cancellation as a crash. Every deploy then produces a burst of false restart alerts.
  • Losing the in-flight item. A crash between get() and task_done() leaves join() hanging forever; use finally.
  • The supervisor itself unsupervised. Put it in the same TaskGroup as the rest of the service so its death is not silent either.
  • Restarting a worker whose dependency is dead. A pool that restarts while the database is down just multiplies connection attempts; back off on the dependency instead.

Frequently Asked Questions

Why do my asyncio workers die silently?

Because nothing retrieves their exception. A task created with create_task and never awaited holds its exception until the object is garbage collected, at which point asyncio logs a warning that most pipelines drop. Attach add_done_callback to log it immediately, and supervise the pool so a dead worker is replaced rather than merely mourned.

How should a supervisor restart a crashed worker?

Wait on the set of worker tasks with FIRST_COMPLETED, identify which finished, log the reason, and spawn a replacement with the same index. Apply exponential backoff between restarts and a crash-loop guard — more than N restarts within a window should stop the service rather than spin forever, so the orchestrator can act.

Should a worker die on an exception or catch it?

Catch it in the loop for item-level failures: the worker keeps its place and you can route the item explicitly to a retry or a dead-letter queue. Let the worker die only when the failure invalidates the worker itself, such as a lost connection it owns, and let the supervisor create a fresh one.

What happens to the item a worker was processing when it crashed?

It is lost unless you handle it. Wrap the work so task_done() is always called in a finally block, and decide explicitly whether to requeue the item, dead-letter it after an attempt cap, or drop it. Without that, queue.join() waits forever for an item nobody is processing.