Skip to content

Recovering from BrokenProcessPool in Async Services

At 02:40 a worker process is killed by the kernel's OOM killer while thumbnailing an unusually large upload. From that moment every CPU-bound request in the service fails with BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending — not only the request that triggered it, and not only for a moment. A ProcessPoolExecutor that loses a worker unexpectedly declares itself broken permanently: pending futures fail, new submissions fail, and the only cure is a new executor. Services that do not handle this stay broken until someone restarts them, which is why the incident is usually described as "the service stopped rendering thumbnails overnight". This guide reproduces the failure, builds a pool wrapper that rebuilds itself, decides which jobs may be retried, prevents the common causes with worker recycling and payload limits, and turns the event into an alert with a cause attached.

Prerequisites

  • Python 3.11+; examples need the if __name__ == "__main__": guard because workers re-import the main module. Standard library only.
  • Executor basics from CPU-Bound Task Offloading.
  • Retry rules from Retry & Backoff Strategies: a job that crashed a worker may have had side effects.
One crashed worker, one dead executor 5 stages from worker dies to rebuild. One crashed worker, one dead executor worker dies OOM or segfault its jobs fail BrokenProcessPool executor broken permanently new submissions raise immediately rebuild only cure Nothing in the executor recovers on its own; the object has to be replaced.

1. Reproduce the blast radius

One crashing job takes down every other job in flight and the executor itself. os._exit(1) in a worker stands in for a segfault in a C extension or an OOM kill.

import asyncio
import os
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures.process import BrokenProcessPool


def double(x: int) -> int:
    return x * 2


def crash() -> None:
    os._exit(1)                                          # like a segfault or an OOM kill


async def main() -> None:
    loop = asyncio.get_running_loop()
    pool = ProcessPoolExecutor(max_workers=2)
    print("before:", await loop.run_in_executor(pool, double, 21))

    in_flight = [loop.run_in_executor(pool, double, i) for i in range(4)]
    try:
        await loop.run_in_executor(pool, crash)
    except BrokenProcessPool as exc:
        print("crashing job ->", type(exc).__name__)

    outcomes = await asyncio.gather(*in_flight, return_exceptions=True)
    print("jobs in flight ->", [type(o).__name__ if isinstance(o, BaseException) else o
                                for o in outcomes])
    try:
        await loop.run_in_executor(pool, double, 1)
    except BrokenProcessPool:
        print("new submissions -> BrokenProcessPool (the executor is unusable)")
    pool.shutdown(wait=False, cancel_futures=True)


if __name__ == "__main__":
    asyncio.run(main())

In this run the four in-flight jobs happened to complete before the crash landed, but the executor was still poisoned: every later submission raised BrokenProcessPool. Under load, in-flight jobs on the dead worker fail with the same exception. BrokenProcessPool subclasses BrokenExecutor and RuntimeError, so a broad except Exception around job code will swallow it — and then silently return errors for every request.

Verify: submissions after the crash raise BrokenProcessPool, and the pool never recovers by itself.

2. Wrap the pool so it can rebuild itself

Treat the executor as a replaceable resource behind a small facade. When a submission raises BrokenProcessPool, discard the executor, build a new one, and let the caller decide about retrying.

import asyncio
import logging
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures.process import BrokenProcessPool

log = logging.getLogger("cpu.pool")


class ResilientPool:
    def __init__(self, workers: int, max_tasks_per_child: int | None = 100) -> None:
        self._workers = workers
        self._max_tasks = max_tasks_per_child
        self._pool: ProcessPoolExecutor | None = None
        self._lock = asyncio.Lock()                       # one rebuild at a time
        self.generation = 0
        self.breaks = 0

    def _ensure(self) -> ProcessPoolExecutor:
        if self._pool is None:
            self._pool = ProcessPoolExecutor(max_workers=self._workers,
                                             max_tasks_per_child=self._max_tasks)
            self.generation += 1
        return self._pool

    async def _rebuild(self, broken: ProcessPoolExecutor) -> None:
        async with self._lock:
            if self._pool is broken:                      # another caller may have rebuilt already
                self.breaks += 1
                log.error("process pool broken; rebuilding (generation %d)", self.generation + 1)
                broken.shutdown(wait=False, cancel_futures=True)
                self._pool = None
                self._ensure()

    async def run(self, fn, *args):
        pool = self._ensure()
        loop = asyncio.get_running_loop()
        try:
            return await loop.run_in_executor(pool, fn, *args)
        except BrokenProcessPool:
            await self._rebuild(pool)
            raise                                         # the caller decides about retrying

    def shutdown(self) -> None:
        if self._pool is not None:
            self._pool.shutdown(wait=False, cancel_futures=True)
            self._pool = None


async def main() -> None:
    pool = ResilientPool(workers=2)
    try:
        await pool.run(crash)
    except BrokenProcessPool:
        pass
    print("after rebuild:", await pool.run(double, 21), "| generation:", pool.generation)
    pool.shutdown()


if __name__ == "__main__":
    logging.basicConfig(level=logging.ERROR)
    asyncio.run(main())

The lock matters: without it, ten concurrent callers all see BrokenProcessPool and each builds a replacement, leaving nine orphaned executors and a burst of process creation. Comparing self._pool is broken makes the rebuild idempotent — the callers that lost the race simply use the new pool.

Verify: the first call raises, the next call succeeds, and generation is 2 — one rebuild, not one per caller.

3. Decide which jobs may be retried

A rebuilt pool is useless if the caller's request already failed. Retrying is right when the job is pure — rendering, parsing, hashing — and wrong when it had side effects or when the job itself is what killed the worker. Retry once, on a fresh pool, and only for jobs marked safe.

import asyncio
import logging
from concurrent.futures.process import BrokenProcessPool

log = logging.getLogger("cpu.pool")


async def run_with_recovery(pool: ResilientPool, fn, *args, idempotent: bool = True):
    try:
        return await pool.run(fn, *args)
    except BrokenProcessPool:
        if not idempotent:
            log.error("worker crashed running a non-idempotent job; not retrying")
            raise
        log.warning("worker crashed; retrying once on the rebuilt pool")
        return await pool.run(fn, *args)                  # a second crash propagates


async def main() -> None:
    pool = ResilientPool(workers=2)
    try:
        await run_with_recovery(pool, crash, idempotent=True)
    except BrokenProcessPool:
        print("second crash propagated: the job itself is the problem")
    print("healthy job after recovery:", await run_with_recovery(pool, double, 7))
    pool.shutdown()


if __name__ == "__main__":
    logging.basicConfig(level=logging.WARNING)
    asyncio.run(main())

A job that reliably crashes the worker must not be retried forever: one retry distinguishes "unlucky" — an OOM kill caused by another process — from "poison", and the second failure sends the job to a dead-letter path instead of a third worker. Where a crash may have left partial external effects, treat it exactly like a timeout and use idempotency keys.

Verify: a job that always crashes propagates after one retry, and ordinary jobs succeed on the rebuilt pool.

The pool broke — what now? A decision on What kind of job was running with 3 outcomes. The pool broke — what now? What kind of job was running? idempotent, first failure retry once on the rebuilt pool has side effects fail the request or use idempotency keys crashed twice treat as poison dead-letter the job The rebuild is automatic; the retry is a decision per job.

4. Prevent the common causes

Most BrokenProcessPool incidents come from three causes: memory growth in long-lived workers, oversized payloads, and jobs that call os._exit or sys.exit. Worker recycling and input limits remove the first two.

import asyncio
import os
from concurrent.futures import ProcessPoolExecutor

MAX_PAYLOAD_BYTES = 64 * 1024 * 1024


class PayloadTooLarge(ValueError):
    pass


def guarded_render(image_bytes: bytes) -> int:
    if len(image_bytes) > MAX_PAYLOAD_BYTES:             # reject before allocating more
        raise PayloadTooLarge(f"{len(image_bytes)} bytes exceeds the limit")
    return len(image_bytes)


async def main() -> None:
    loop = asyncio.get_running_loop()
    # max_tasks_per_child recycles workers, so leaked memory cannot accumulate forever
    pool = ProcessPoolExecutor(max_workers=1, max_tasks_per_child=2)
    pids = [await loop.run_in_executor(pool, os.getpid) for _ in range(6)]
    print("distinct worker pids over 6 jobs:", len(set(pids)))     # 3

    try:
        await loop.run_in_executor(pool, guarded_render, b"x" * (MAX_PAYLOAD_BYTES + 1))
    except PayloadTooLarge as exc:
        print("rejected cleanly:", exc)                   # an exception, not a dead worker
    pool.shutdown()


if __name__ == "__main__":
    asyncio.run(main())

Recycling after max_tasks_per_child jobs replaced the worker twice over six jobs, which bounds the memory any single worker can leak — at the cost of re-importing modules in each new worker, so set it high enough that start-up is a small share of job time. Validating payload size inside the job turns "the worker was OOM-killed" into an ordinary exception the caller can handle, and keeping large data out of the payload entirely is better still, as described in reducing pickle overhead in ProcessPoolExecutor payloads.

Verify: six jobs run in three distinct worker processes, and the oversized payload raises PayloadTooLarge without breaking the pool.

5. Alert with the cause attached

"Pool broken" alone sends an engineer to read kernel logs. Record what the worker was doing, and check the system's view of why it died, so the alert names the cause.

import asyncio
import logging
import os
import resource
import subprocess

log = logging.getLogger("cpu.pool")


def worker_memory_limit_mib() -> int:
    soft, _hard = resource.getrlimit(resource.RLIMIT_AS)
    return -1 if soft == resource.RLIM_INFINITY else soft // (1024 * 1024)


def recent_oom_kills() -> int:
    """Count OOM kills the kernel reported recently (best effort, Linux)."""
    try:
        out = subprocess.run(["dmesg", "--since", "-5min"], capture_output=True,
                             text=True, timeout=2).stdout
    except (OSError, subprocess.SubprocessError):
        return -1                                         # not permitted in this container
    return out.lower().count("killed process")


def report_break(job_name: str, payload_bytes: int, generation: int) -> dict:
    event = {
        "event": "process_pool_broken",
        "job": job_name,
        "payload_bytes": payload_bytes,
        "pool_generation": generation,
        "worker_memory_limit_mib": worker_memory_limit_mib(),
        "recent_oom_kills": recent_oom_kills(),
    }
    log.error("process pool broken", extra={"pool": event})
    return event


if __name__ == "__main__":
    logging.basicConfig(level=logging.ERROR)
    print(report_break("thumbnail", payload_bytes=143_000_000, generation=3))

Two signals separate the usual causes: a positive recent_oom_kills with a large payload points at memory, while breaks that cluster on one job type point at a crashing extension. Alert on rate — more than one break per hour, or any break followed by a second within a minute — rather than on single events, since one rebuild is a recovery, not an outage. Feed the same counter into a circuit breaker so a pool that breaks repeatedly stops accepting CPU work instead of thrashing.

Verify: the event includes the job name, payload size and pool generation, and running it inside your container shows whether dmesg is readable there.

Causes and preventions A grid of 4 rows by 2 columns. Causes and preventions cause symptom prevention memory growth in workers OOM kill after hours max_tasks_per_child oversized payload breaks on big inputs validate size in the job C extension segfault breaks on one job kind isolate, try spawn job calls sys.exit breaks immediately raise instead of exiting Each prevention turns a dead worker into an ordinary exception.

Verification

Process-pool failures are handled when:

  • A crash does not end the service: after BrokenProcessPool, the next request succeeds on a rebuilt pool.
  • Rebuilds are single: concurrent failures produce one new executor, not one per caller.
  • Retries are deliberate: only idempotent jobs retry, once, and a repeat failure propagates.
  • Causes are prevented: workers recycle after a bounded number of jobs, and oversized payloads are rejected as exceptions.
  • Breaks are observable: each break emits a structured event with job, payload size and generation, and alerts fire on rate.

Pitfalls & edge cases

  • Catching Exception around job code. BrokenProcessPool is a RuntimeError, so a broad handler hides it and the service returns errors indefinitely. Catch it explicitly.
  • Rebuilding inside the failing call path only. Futures already submitted to the broken pool also fail; callers must handle the exception rather than assume the rebuild fixes their job.
  • max_tasks_per_child too low. Every recycle pays interpreter start-up and module imports; a value of one turns a pool into per-job processes.
  • Fork-related crashes. Workers created with fork inherit locks and threads from the parent; if crashes correlate with third-party libraries, try the spawn start method.
  • Ignoring container memory limits. The kernel kills the biggest process, which is usually a worker holding a large payload. Size limits so the whole pool fits within the container's memory.

Frequently Asked Questions

What causes BrokenProcessPool in Python?

A worker process died unexpectedly — a segfault in a C extension, an OOM kill, or code calling os._exit or sys.exit — while jobs were running or pending. The executor then marks itself broken: pending futures fail and every new submission raises BrokenProcessPool, because the pool cannot know what state its workers are in.

Can a ProcessPoolExecutor recover after BrokenProcessPool?

Not by itself. The executor is permanently unusable and must be replaced. Wrap it so that on BrokenProcessPool you shut the old executor down with cancel_futures=True and create a new one, guarding the rebuild with a lock so concurrent failures create only one replacement.

Should I retry a job that broke the process pool?

Only if it is idempotent, and only once. One retry distinguishes an unlucky job, for example one killed because another process exhausted memory, from a job that reliably crashes the worker. If the second attempt also breaks the pool, treat the job as poison and route it to a dead-letter path.

How does max_tasks_per_child help with process pool stability?

It recycles a worker after it has run the given number of jobs, so memory leaked by a job or a C library cannot accumulate indefinitely and trigger an OOM kill. The trade-off is interpreter start-up and imports for each new worker, so choose a value where recycling cost is small relative to job duration.