Skip to content

Running Subprocesses with asyncio.create_subprocess_exec

An upload service converts each file with an external tool. The first version calls subprocess.run() inside the request handler, and during a batch of large uploads every other endpoint's latency climbs to the duration of one conversion. The second version switches to asyncio.create_subprocess_exec() and wraps it in asyncio.timeout(), and a week later the host has four hundred orphaned converter processes, because the timeout stopped waiting but never stopped the process. The third version adds proc.kill() and still leaks — the converter is a shell script whose real worker is a grandchild. Running external commands from asyncio is straightforward once, and subtle at scale. This guide builds a single run() helper with the semantics of subprocess.run(check=True), adds a timeout that escalates from SIGTERM to SIGKILL, cleans up entire process trees on timeout and cancellation, and bounds how many children run at once.

Prerequisites

  • Python 3.11+ on Linux or macOS; process-group handling in steps 3–4 is POSIX-specific, with Windows notes in the pitfalls. Standard library only.
  • The subprocess model from Subprocesses & File I/O: pipes are served by the loop, and a timeout does not terminate a child by itself.
  • Timeout and cancellation semantics from choosing asyncio.timeout vs wait_for.
The life of one child process 5 stages from acquire slot to reaped. The life of one child process acquire slot semaphore exec argument list communicate drain both pipes check code raise on non-zero reaped no zombie left Every exit path — success, failure, timeout, cancellation — ends in the last box.

1. Start the child with an argument list

create_subprocess_exec() takes the program and each argument as separate strings and executes the program directly, with no shell in between. That makes untrusted values — file names from uploads, branch names from webhooks — safe to pass, because nothing interprets spaces, quotes, ; or $(...).

import asyncio
import os


async def start(filename: str) -> asyncio.subprocess.Process:
    return await asyncio.create_subprocess_exec(
        "convert-tool", "--input", filename, "--format", "webp",   # each argument separate
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.DEVNULL,        # never inherit the service's stdin
        cwd="/srv/work",
        env={**os.environ, "OMP_NUM_THREADS": "1"},   # explicit, minimal changes
        start_new_session=True,                  # own process group: see step 4
    )

Compare this with create_subprocess_shell(f"convert-tool --input {filename}"): a file named x; rm -rf /srv becomes two commands. Set stdin=DEVNULL unless you intend to write to the child, so a tool that unexpectedly prompts for input fails immediately instead of waiting forever on a pipe no one will write to.

Verify: call start() with a filename containing spaces and a semicolon; the tool receives it as one literal argument, which you can confirm by running sys.executable -c "import sys; print(sys.argv)" in place of the tool.

2. Collect output and check the exit code

For commands whose output is bounded, communicate() reads stdout and stderr concurrently until both close and the process exits. Wrap that in a helper that returns a result object and, like subprocess.run(check=True), raises CalledProcessError on failure so callers cannot forget to check.

import asyncio
import signal
import subprocess
from dataclasses import dataclass


@dataclass(frozen=True)
class Completed:
    argv: tuple[str, ...]
    returncode: int
    stdout: bytes
    stderr: bytes


def describe(returncode: int) -> str:
    """Negative return codes mean the child was killed by a signal."""
    if returncode < 0:
        return f"killed by {signal.Signals(-returncode).name}"
    return f"exited with {returncode}"


async def run_simple(*argv: str, check: bool = True) -> Completed:
    proc = await asyncio.create_subprocess_exec(
        *argv, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.DEVNULL,
    )
    out, err = await proc.communicate()
    result = Completed(tuple(argv), proc.returncode, out, err)
    if check and result.returncode != 0:
        raise subprocess.CalledProcessError(result.returncode, argv, out, err)
    return result

Reusing subprocess.CalledProcessError keeps error handling identical to synchronous code, and its stderr attribute carries the tool's own explanation into your logs. A return code of -9 or -15 is not a tool failure at all: it means something — your timeout, the OOM killer, an operator — sent a signal. describe() makes that distinction visible in logs.

Verify: a child that writes boom to stderr and exits with 2 raises CalledProcessError with returncode == 2 and stderr == b"boom"; describe(-9) returns killed by SIGKILL.

3. Add a timeout that escalates from SIGTERM to SIGKILL

A timeout must end with the child gone. Send SIGTERM first so well-behaved tools can remove temporary files and flush output, wait a short grace period, then send SIGKILL, and always await proc.wait() so the exit is reaped. Doing this in finally covers cancellation as well as the timeout.

import asyncio
import os
import signal
import subprocess


async def terminate_tree(proc: asyncio.subprocess.Process, grace: float) -> None:
    try:
        os.killpg(proc.pid, signal.SIGTERM)            # polite: let it clean up
    except ProcessLookupError:
        pass
    try:
        async with asyncio.timeout(grace):
            await proc.wait()
            return
    except TimeoutError:
        pass
    try:
        os.killpg(proc.pid, signal.SIGKILL)            # not negotiable
    except ProcessLookupError:
        pass
    await proc.wait()                                  # reap: no zombies


async def run(*argv: str, check: bool = True, timeout: float | None = None,
              grace: float = 3.0, cwd: str | None = None,
              env: dict[str, str] | None = None) -> Completed:
    proc = await asyncio.create_subprocess_exec(
        *argv,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.DEVNULL,
        cwd=cwd,
        env=env,
        start_new_session=True,
    )
    try:
        async with asyncio.timeout(timeout):
            out, err = await proc.communicate()
    finally:
        if proc.returncode is None:                    # timed out or cancelled
            await terminate_tree(proc, grace)
    result = Completed(tuple(argv), proc.returncode, out, err)
    if check and result.returncode != 0:
        raise subprocess.CalledProcessError(result.returncode, argv, out, err)
    return result

When the timeout fires, TimeoutError propagates after cleanup, so callers see a timeout rather than a confusing negative return code. When the calling task is cancelled — a client disconnect, a service shutdown — the same finally block terminates the child before the CancelledError continues upward.

Verify: a child that ignores SIGTERM and sleeps for 30 seconds, run with timeout=0.5, grace=0.5, raises TimeoutError after about one second — half a second of timeout plus half a second of grace before SIGKILL. Cancelling a task running run() on a 30-second sleeper leaves no process behind in ps.

Timeout, grace, then SIGKILL 2 lanes over time. Timeout, grace, then SIGKILL caller await communicate() grace period reaped child group running ignores SIGTERM KILL start timeout +grace time → SIGTERM gives cleanup a chance; SIGKILL guarantees the process is gone.

4. Clean up the whole process tree

proc.kill() signals exactly one process. Tools launched through wrapper scripts, npm, make or shells spawn children of their own, and those grandchildren survive — still consuming CPU, still holding the stdout pipe open. Because asyncio considers a subprocess finished only when it has exited and its pipes are closed, the parent's wait() stays pending until the orphaned grandchild exits too.

import asyncio
import sys

GRANDCHILD = ("import subprocess, sys, time;"
              "subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(3)']);"
              "time.sleep(30)")


async def kill_only_the_child() -> float:
    proc = await asyncio.create_subprocess_exec(
        sys.executable, "-c", GRANDCHILD, stdout=asyncio.subprocess.PIPE)
    await asyncio.sleep(0.5)
    loop = asyncio.get_running_loop()
    started = loop.time()
    proc.kill()                                  # the grandchild still holds stdout
    await proc.wait()
    return loop.time() - started                 # about 2.5 s: until the grandchild exits


async def kill_the_group() -> float:
    proc = await asyncio.create_subprocess_exec(
        sys.executable, "-c", GRANDCHILD, stdout=asyncio.subprocess.PIPE,
        start_new_session=True)
    await asyncio.sleep(0.5)
    loop = asyncio.get_running_loop()
    started = loop.time()
    await terminate_tree(proc, grace=1.0)        # SIGTERM to the whole group
    return loop.time() - started                 # immediate


print(asyncio.run(kill_only_the_child()), asyncio.run(kill_the_group()))

In this measurement the grandchild sleeps for only three seconds, so wait() returned after about 2.5 seconds; with a real long-running worker it does not return at all. start_new_session=True makes the child the leader of a new session and process group whose ID equals its PID, so os.killpg(proc.pid, ...) reaches every descendant that has not deliberately left the group.

Time for wait() after killing a child 2 bars comparing proc.kill() on the child only with the others. Time for wait() after killing a child proc.kill() on the child only 2.53 s os.killpg() on the group about 0 s Measured on Linux, Python 3.14; the grandchild slept 3 s, so a real worker would never exit. wait() resolves only when the process has exited and its pipes have closed.

Verify: the first call returns roughly 2.5 seconds and the second returns almost immediately; ps -o pid,pgid,cmd during the first run shows the grandchild outliving its parent.

5. Bound how many children run at once

Every child is a full process with its own memory, up to three pipes, and a slot in the host's process table. Launching one per incoming request turns a traffic spike into a fork storm. Put a semaphore in front of the helper, sized to what the host and the tool can actually sustain.

import asyncio
import os
import sys

# CPU-heavy tools: roughly one per core. I/O-heavy tools: the tool's own limits apply.
CHILD_SLOTS = asyncio.Semaphore(max(1, os.process_cpu_count() or 1))


async def run_bounded(*argv: str, timeout: float) -> Completed:
    async with CHILD_SLOTS:
        return await run(*argv, timeout=timeout)


async def main() -> None:
    results = await asyncio.gather(
        *(run_bounded(sys.executable, "-c", f"print({i})", timeout=10) for i in range(50))
    )
    print(sum(1 for r in results if r.returncode == 0), "succeeded")


asyncio.run(main())

Acquire the slot outside the timeout, as shown, so time spent waiting for a slot does not count against the child's own budget; if the caller has an overall deadline, bound the wait for the slot separately. The same queueing principles apply as for limiting concurrent requests with asyncio.Semaphore.

Verify: while 50 jobs run, ps --ppid <service pid> | wc -l never exceeds the slot count plus the header line, and all 50 succeed.

Verification

The subprocess helper is production-ready when:

  • No blocking calls remain: a search for subprocess.run, check_output and os.system in async code finds nothing.
  • Failures are explicit: non-zero exits raise CalledProcessError with stderr attached, and signal deaths are logged as signals.
  • Timeouts end processes: after a timeout, the child and its descendants are gone and reaped within the grace period.
  • Cancellation ends processes: cancelling the calling task leaves no children behind.
  • Concurrency is bounded: the number of live children never exceeds the configured slots, even under burst load.

Pitfalls & edge cases

  • Windows process groups. start_new_session and os.killpg do not exist on Windows. Use creationflags=subprocess.CREATE_NEW_PROCESS_GROUP and send CTRL_BREAK_EVENT, or run taskkill /T /F /PID to terminate a tree.
  • Children that leave the group. Daemonising tools call setsid() themselves and escape killpg. Run such tools under a supervisor, or in a cgroup or container that can be stopped as a unit.
  • Large output through communicate(). It buffers everything in memory. For output that can reach megabytes, stream it as described in streaming subprocess output without deadlocks.
  • Environment leakage. Passing env=None inherits the service's full environment, including secrets. Build a minimal environment for untrusted or third-party tools.
  • Blocking preexec_fn. Arbitrary Python code in preexec_fn runs between fork and exec and is unsafe with threads. Prefer the dedicated arguments — start_new_session, user, group, umask — which do not run Python in the child.

Frequently Asked Questions

What is the difference between create_subprocess_exec and create_subprocess_shell?

create_subprocess_exec runs a program directly with an explicit argument list, so arguments are never interpreted by a shell and untrusted values are safe. create_subprocess_shell passes a single command string to the shell, which enables pipes and globbing but also command injection if any part of the string comes from input.

How do I check the exit code of an asyncio subprocess?

After await proc.communicate() or await proc.wait(), read proc.returncode. Zero means success, a positive value is the program's own exit status, and a negative value means the process was terminated by that signal number, for example -9 for SIGKILL. Raise subprocess.CalledProcessError for non-zero codes to match subprocess.run(check=True).

How do I kill an asyncio subprocess and all of its children?

Start it with start_new_session=True so it leads its own process group, then signal the group with os.killpg(proc.pid, signal.SIGTERM), wait a grace period, send SIGKILL to the group if it is still running, and await proc.wait(). Killing only the child leaves grandchildren running and holding its pipes open.

How many subprocesses can asyncio run at the same time?

The event loop can supervise many, but the host limits how many are sensible: each child is a process with its own memory and file descriptors. Guard creation with an asyncio.Semaphore sized to cores for CPU-heavy tools, or to the tool's own limits, and keep the file descriptor limit well above the number of pipes.