Skip to content

Running Subprocesses with AnyIO

AnyIO's subprocess API — run_process to run a command and collect its output, open_process to stream it — works the same on asyncio and trio, and it ties the child's lifetime to the cancel scope around it. That last property is the main reason to prefer it over asyncio.create_subprocess_exec. Measured with AnyIO 4.15 on Python 3.14: when move_on_after(0.5) expired around run_process or open_process running sleep 30, the child was killed with SIGKILL (return code −9) on both backends; the same timeout with asyncio.wait_for(proc.wait(), 0.5) left the child still running. Two hundred concurrent run_process(["true"]) calls finished in 0.14 s on asyncio and 0.11 s on trio. A non-zero exit raised CalledProcessError with the captured stderr. Collecting 200 MiB of output with run_process peaked at 222 MiB of memory; streaming it from open_process peaked at 23 MiB. And cancellation kills only the direct child: a sleep started in the background by a shell survived the shell's death, until the command was started in a new session and its whole process group was killed. This guide covers those behaviours.

Prerequisites

1. Run a command and collect its output

run_process starts the command, waits for it, and returns a CompletedProcess with stdout and stderr as bytes. With the default check=True, a non-zero exit raises:

import subprocess
import anyio

async def git_head(repo: str) -> str:
    try:
        result = await anyio.run_process(["git", "-C", repo, "rev-parse", "HEAD"])
    except subprocess.CalledProcessError as exc:
        raise RuntimeError(f"git failed ({exc.returncode}): {exc.stderr.decode().strip()}") from exc
    return result.stdout.decode().strip()

Measured: a command that wrote "oops" to stderr and exited 3 raised CalledProcessError with returncode=3 and stderr=b'oops\n' on both backends. Pass the command as a list, not a string — a string is run through the shell, with the quoting and injection risks that brings. Two hundred concurrent run_process(["true"]) calls in one task group took 0.14 s on asyncio and 0.11 s on trio, so starting processes is cheap; bound their number anyway, since each holds file descriptors and a process slot, as in limiting concurrent subprocesses.

Verify: a failing command surfaces its exit code and stderr, and a successful one returns its output.

2. Rely on cancellation to stop the child

The cancel scope around a subprocess call owns the process. When the scope is cancelled — by a deadline, by a sibling failing, by shutdown — AnyIO kills the child and waits for it:

async def render_thumbnail(src: str, dst: str) -> bool:
    with anyio.move_on_after(10) as scope:
        await anyio.run_process(["convert", src, "-resize", "256x256", dst])
    return not scope.cancelled_caught

Measured on both backends: with a 0.5 s deadline around sleep 30, the call returned after 0.60 s and the child was gone, killed with SIGKILL (return code −9). The equivalent plain-asyncio code behaves differently:

proc = await asyncio.create_subprocess_exec("sleep", "30")
try:
    await asyncio.wait_for(proc.wait(), 0.5)
except TimeoutError:
    pass                      # the child is still running here

Measured: after the timeout, the sleep was still alive. wait_for cancelled the waiting, not the process; plain asyncio code must kill the child itself, as covered in timing out subprocesses in asyncio. With AnyIO the guarantee is structural — if the scope is gone, so is the child.

Verify: after a deadline expires around a subprocess call, the child's PID no longer exists.

A 0.5 s deadline around sleep 30 A grid of 4 rows by 3 columns. A 0.5 s deadline around sleep 30 code child after deadline grandchild after deadline anyio.run_process, asyncio backend killed (-9) survived anyio.open_process, trio backend killed survived asyncio.wait_for(proc.wait(), 0.5) still running still running anyio + start_new_session + killpg killed killed AnyIO 4.15, Python 3.14.

3. Stream large output with open_process

run_process buffers everything the child writes. For large or unbounded output, open the process and read its streams as they arrive:

async def count_lines(path: str) -> int:
    lines = 0
    async with await anyio.open_process(["zcat", path]) as proc:
        async for chunk in proc.stdout:              # bytes, as the child writes them
            lines += chunk.count(b"\n")
    if proc.returncode:
        raise subprocess.CalledProcessError(proc.returncode, ["zcat", path])
    return lines

Measured with a child writing 200 MiB: run_process peaked at 222 MiB of resident memory, holding the whole output; streaming from open_process peaked at 23 MiB. open_process does not check the exit code, so check returncode after the block. If the child writes to both stdout and stderr, read both concurrently in a task group, or redirect stderr — a child blocked on a full stderr pipe while you wait on stdout is the classic deadlock covered in streaming subprocess output without deadlocks.

Verify: memory while processing a large output stays flat, and a failing child still raises.

4. Kill the whole process group, not only the child

Cancellation kills the process AnyIO started. If that process started others — a shell running a pipeline, a build tool spawning compilers, a script launching a server — they are not killed with it:

await anyio.run_process(["sh", "-c", "sleep 30 & wait"])     # the shell dies, sleep 30 lives on

Measured: when the deadline expired, the shell was killed (return code −9) and its background sleep kept running, re-parented to init. Start the command in a new session, which makes it the leader of its own process group, and kill the group when the scope ends:

import os, signal

async def run_group(cmd: list[str], timeout: float) -> None:
    pgid = None
    try:
        with anyio.move_on_after(timeout):
            async with await anyio.open_process(cmd, start_new_session=True) as proc:
                pgid = proc.pid                    # session leader: its PID is the group ID
                await proc.wait()
    finally:
        if pgid is not None:
            try:
                os.killpg(pgid, signal.SIGKILL)    # everything it left behind
            except ProcessLookupError:
                pass

Measured on both backends: the grandchild was gone after the deadline. Killing the group after AnyIO's own cleanup, rather than inside the async with, avoids racing AnyIO for the child's exit status — doing it the other way round produced an "exit status already read" warning and a bogus return code of 255. On Windows, process groups work differently; use a job object or a process-tree utility there.

Verify: after cancellation, no process from the command's group remains (pgrep -g <pgid> finds nothing).

Running a command that may spawn children A flow of 5 stages. Running a command that may spawn children open_process start_new_session=True Wait or stream inside move_on_after Cancelled AnyIO SIGKILLs the child finally os.killpg(pgid, SIGKILL) Check returncode, stderr The cancel scope owns the child; your code owns the rest of its group.

5. Send input and choose between collect and stream

Both functions accept input. run_process(cmd, input=b"...") writes the bytes to the child's stdin and closes it, which suits filters such as jq or gzip; open_process gives you proc.stdin as a stream to write incrementally, which suits feeding a long-running tool. A summary of the choice:

# small input, small output: collect
result = await anyio.run_process(["jq", ".items | length"], input=payload)

# large or incremental: stream both ways in a task group
async with await anyio.open_process(["gzip", "-c"]) as proc:
    async with anyio.create_task_group() as tg:
        tg.start_soon(write_all, proc.stdin, source)      # closes stdin when done
        tg.start_soon(copy_to_file, proc.stdout, dst)

Writing and reading in separate tasks is what keeps a process with large input and output from deadlocking: the child cannot finish writing until you read, and you cannot finish writing until it reads. The same code runs unchanged on trio, which is the point of AnyIO; test both backends with the pytest plugin as in testing async code with the AnyIO pytest plugin.

Verify: a round trip of a large payload through gzip -c and gunzip -c completes without hanging and matches the input.

Which AnyIO subprocess call? A decision on What does the command do with 4 outcomes. Which AnyIO subprocess call? What does the command do? small output run_process(check=True) CalledProcessError with stderr large output open_process, stream stdout 23 MiB vs 222 MiB spawns children new session + killpg grandchild otherwise survives large input and output write and read in a task group no pipe deadlock In every case the cancel scope bounds the child's lifetime.

Verification

Subprocesses run with AnyIO are well-behaved when:

  • Commands are lists, and failures raise with the exit code and stderr.
  • Every call sits inside a cancel scope with a deadline, relying on AnyIO to kill the child.
  • Large output is streamed from open_process, with the return code checked.
  • Commands that spawn children run in a new session, and their group is killed afterwards.

Diagnostic Hook: periodically count processes whose parent is PID 1 and whose command line matches the tools your service runs. Orphans accumulating there are grandchildren that outlived a cancelled parent — the case AnyIO does not cover on its own.

Pitfalls & edge cases

  • Assuming asyncio timeouts kill the child. Measured: wait_for left it running.
  • Assuming AnyIO kills grandchildren. Measured: a shell's background job survived.
  • Collecting huge output. Measured: 222 MiB for 200 MiB of output.
  • Killing the group inside the async with. It races AnyIO for the exit status.

Frequently Asked Questions

How do I run a subprocess with AnyIO?

await anyio.run_process(["cmd", "arg"]) runs it and returns a CompletedProcess with stdout and stderr; a non-zero exit raises CalledProcessError. Use anyio.open_process to stream output.

Does AnyIO kill the subprocess when cancelled?

Yes: a 0.5 s move_on_after around sleep 30 killed it with SIGKILL on both asyncio and trio. Plain asyncio.wait_for around proc.wait() left the child running.

Why do child processes survive after my AnyIO task is cancelled?

AnyIO kills only the process it started. Processes spawned by it, such as a shell's background jobs, survive; start the command with start_new_session=True and kill its process group with os.killpg.

Should I use run_process or open_process for large output?

open_process: streaming 200 MiB peaked at 23 MiB of memory, while run_process held it all and peaked at 222 MiB.