Skip to content

Subprocesses & File I/O in Asyncio

Not all I/O is a network socket. Services shell out to ffmpeg, git, pg_dump and in-house CLIs; they write uploads to disk, tail log files, and produce reports. In asyncio these are the operations most likely to be done wrong, because the synchronous versions look harmless. subprocess.run() blocks the event loop for the entire life of the child process. open().read() on a network filesystem can block for seconds. And the async versions carry their own traps: a timeout around a subprocess stops waiting for the child but leaves it running, a pipe that nobody drains deadlocks the child once its buffer fills, and wrapping every line of file output in an await can be two hundred times slower than the blocking code it replaced.

This section covers both halves. For processes, asyncio offers a genuinely asynchronous API — asyncio.create_subprocess_exec() — built on non-blocking pipes and child-exit notification, so hundreds of children can be supervised from one loop. For regular files there is no such thing: operating systems do not offer readiness notifications for disk files the way they do for sockets, so every "async file" library is a thread pool in disguise, and the design question becomes how to use that thread pool efficiently. The parent section, Network I/O & Protocol Handling, covers transports and streams for sockets; the stream API here is the same one, pointed at pipes.

Scope of this section:

  • The asyncio subprocess model: transports, pipes, exit notification and the Process object.
  • Timeouts and cancellation that actually terminate children and their process trees.
  • Reading output incrementally without pipe deadlocks or unbounded memory.
  • Why file I/O always ends up in a thread, and how to batch it so that costs little.
  • Bounding concurrent children and file operations so neither exhausts the host.

Architectural principles

  • Never run a child process synchronously on the loop. subprocess.run(), os.system() and check_output() block the loop until the child exits. Use asyncio.create_subprocess_exec(), or push the synchronous call into a thread only when a library forces it.
  • Stopping a wait is not stopping a process. Timeouts and cancellation interrupt the coroutine awaiting the child; the child keeps running. Every code path that gives up on a child must terminate it and then wait for it to exit.
  • Every pipe you open must be drained. A child blocks when a pipe buffer fills. Read stdout and stderr concurrently — or merge them — for the entire life of the process, or use communicate() when output is small.
  • Files are blocking I/O; batch the blocking. Move whole file operations into one asyncio.to_thread() call rather than paying a thread hop per line or per small chunk.
  • Bound children and file work explicitly. Each child costs a process, file descriptors and memory; each file operation holds an executor thread. Cap both with semaphores sized to the host, not to demand.
Two kinds of I/O, two execution paths 4 stacked layers from Event loop selector to Kernel. Two kinds of I/O, two execution paths Event loop selector stdout / stderr pipes stdin pipe child-exit pidfd Stream API StreamReader StreamWriter proc.wait() Executor threads open / read / write aiofiles calls to_thread jobs Kernel 64 KiB pipe buffers page cache process groups Pipes are real non-blocking I/O; disk files are blocking I/O in a thread.

Execution model: pipes on the loop, files in threads

asyncio.create_subprocess_exec() forks and executes the child, then connects its standard streams to the loop through non-blocking pipes. On the loop side, each pipe becomes a transport feeding an asyncio.StreamReader — proc.stdout and proc.stderr — or a StreamWriter for proc.stdin. Reading from them is exactly like reading a socket: await proc.stdout.readline() suspends until data arrives, and the loop's selector wakes it. Child exit is observed separately; on current Linux versions asyncio uses a pidfd, which the selector can watch like any file descriptor, so waiting for exit costs no thread. await proc.wait() resolves when the process has exited and its pipes have closed.

Two resources sit outside the loop's control. The first is the kernel pipe buffer — typically 64 KiB on Linux. When the child writes more than that and nobody reads, the child's write() blocks; if the parent is simultaneously waiting for the child to exit, neither side can progress. The second is the child's own process tree: if the child spawns grandchildren, killing the child does not kill them, and they keep the pipes open, so wait() never finishes. Starting children in their own session with start_new_session=True puts the whole tree in one process group that can be signalled at once.

Regular files follow a different path entirely. epoll and kqueue report disk files as always "ready", so a non-blocking read on a file still blocks inside the kernel while the page cache is filled from disk. asyncio therefore has no file API, and libraries such as aiofiles run each operation on a thread pool. That is correct, but every await f.write(line) becomes a submission to the executor, a thread wake-up, and a future resolved back on the loop — overhead measured in tens of microseconds per call, which dominates when the actual write takes nanoseconds. The same thread-pool mechanics are described in running blocking SDK calls with asyncio.to_thread.

How an undrained pipe deadlocks 4 lanes over time. How an undrained pipe deadlocks child writes output blocked: pipe full pipe buffer filling 64 KiB full parent, wait() waiting for exit that never comes parent, draining reads while child writes exit time → Read the pipes for the whole life of the child, or merge them, or use communicate().

Pattern catalogue

Run a command and collect its output

For short-lived commands with modest output, communicate() reads stdout and stderr concurrently, waits for exit, and returns both. Always use the exec form with an argument list.

import asyncio


async def git_head(repo: str) -> str:
    proc = await asyncio.create_subprocess_exec(
        "git", "-C", repo, "rev-parse", "HEAD",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    stdout, stderr = await proc.communicate()
    if proc.returncode != 0:
        raise RuntimeError(f"git failed ({proc.returncode}): {stderr.decode().strip()}")
    return stdout.decode().strip()

communicate() buffers everything in memory, so use it only when output size is known to be small. create_subprocess_shell() exists, but passing untrusted input through a shell invites command injection; reserve it for fixed command strings. The full lifecycle, including exit codes and environment handling, is in running subprocesses with asyncio.create_subprocess_exec.

Timeouts that terminate the child

A timeout around communicate() or wait() must be followed by killing the process and reaping it.

import asyncio


async def run_with_timeout(argv: list[str], seconds: float) -> tuple[int, bytes]:
    proc = await asyncio.create_subprocess_exec(*argv, stdout=asyncio.subprocess.PIPE)
    try:
        async with asyncio.timeout(seconds):
            out, _ = await proc.communicate()
            return proc.returncode, out
    except TimeoutError:
        proc.kill()                        # without this the child keeps running
        await proc.wait()                  # reap it: no zombie, pipes closed
        raise

Run against a child that sleeps for five seconds with a 200 ms timeout, the TimeoutError arrives on time — and without the kill(), os.kill(proc.pid, 0) confirms the process is still alive afterwards. Put the termination in a finally block when the same cleanup must also run on cancellation.

Stream output line by line

For long-running children or large output, read incrementally and keep only what you need.

import asyncio


async def follow(argv: list[str]) -> int:
    proc = await asyncio.create_subprocess_exec(
        *argv,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.STDOUT,     # merge: one stream, one reader, natural ordering
        limit=1024 * 1024,                    # longest line accepted by readline()
    )
    async for raw in proc.stdout:
        print("child:", raw.decode(errors="replace").rstrip())
    return await proc.wait()

Merging stderr into stdout removes the two-reader problem entirely at the cost of not being able to tell the streams apart. The default limit is 64 KiB; a single longer line raises ValueError: Separator is not found, and chunk exceed the limit. Keeping the streams separate, parsing progress, and feeding stdin are covered in streaming subprocess output without deadlocks.

Kill the whole process tree

Children that spawn their own children need group-level signals.

import asyncio
import os
import signal


async def start_isolated(argv: list[str]) -> asyncio.subprocess.Process:
    return await asyncio.create_subprocess_exec(*argv, start_new_session=True)


async def stop_tree(proc: asyncio.subprocess.Process, grace: float = 5.0) -> None:
    if proc.returncode is not None:
        return
    os.killpg(proc.pid, signal.SIGTERM)          # the session leader's pid is the group id
    try:
        async with asyncio.timeout(grace):
            await proc.wait()
    except TimeoutError:
        os.killpg(proc.pid, signal.SIGKILL)
        await proc.wait()

A shell wrapper script that launches a worker, killed with proc.kill(), dies while the worker survives and keeps the stdout pipe open; proc.wait() then hangs. With start_new_session=True and os.killpg(), both disappear. This is POSIX-only; on Windows, use job objects or taskkill /T.

Batch file I/O into one thread hop

Treat a file operation as one blocking unit of work and offload it once.

import asyncio
import json
from pathlib import Path


def _write_ndjson(path: Path, records: list[dict]) -> int:
    with path.open("w", encoding="utf-8") as f:
        for record in records:
            f.write(json.dumps(record) + "\n")
    return len(records)


async def export(path: Path, records: list[dict]) -> int:
    return await asyncio.to_thread(_write_ndjson, path, records)   # one hop, not one per line

Writing 20,000 short lines took about 2 ms as a single to_thread call and about 545 ms with aiofiles awaiting each line — the same thread pool, used 20,000 times instead of once. The measurements and when a streaming approach is still necessary are in async file I/O with aiofiles vs asyncio.to_thread.

Loop stall while writing 256 MiB 4 bars comparing write + fsync on the loop, run 1 with the others. Loop stall while writing 256 MiB write + fsync on the loop, run 1 139 ms lag write + fsync on the loop, run 2 171 ms lag same work via to_thread, run 1 1.7 ms lag same work via to_thread, run 2 1.1 ms lag Worst lateness of a 5 ms heartbeat, local disk, Python 3.14; the write itself took 140-190 ms each time. Offloading does not make the write faster; it stops everyone else waiting for it.

Resource boundaries

Resource What consumes it How to size and bound it
Child processes One per concurrent create_subprocess_exec Semaphore sized to cores for CPU-heavy tools, or to the tool's own concurrency limit
File descriptors Up to three pipes per child, plus pidfds Keep ulimit -n well above children × 4; close unused pipes by not requesting them
Pipe buffers ~64 KiB each in the kernel Drain continuously; never wait() with an undrained PIPE
Parent memory communicate() buffers all output Stream and keep a bounded tail for anything that can exceed a few megabytes
Executor threads Each to_thread or aiofiles call holds one Default pool is min(32, cpus + 4); give heavy file work its own executor
Disk and page cache Large sequential writes, fsync Limit concurrent writers; batch writes; call fsync once per file, not per record

The executor row is easy to miss: file work and blocking SDK calls share the default thread pool with asyncio.to_thread(), so a burst of slow NFS reads can delay unrelated offloaded calls. Separate them with a dedicated ThreadPoolExecutor when file work is heavy, following worker pool sizing for mixed workloads.

Integrated production example

A job runner that executes external commands concurrently: a semaphore bounds the number of children, each child runs in its own process group, output is streamed line by line with only a bounded tail retained, timeouts and cancellation terminate the entire tree, and the final report is written to disk with a single thread hop.

import asyncio
import json
import logging
import os
import signal
import sys
import time
from dataclasses import asdict, dataclass
from pathlib import Path

log = logging.getLogger("jobs")
MAX_PARALLEL = max(1, (os.process_cpu_count() or 2) // 2)
_slots = asyncio.Semaphore(MAX_PARALLEL)


@dataclass
class JobResult:
    name: str
    returncode: int | None
    seconds: float
    timed_out: bool
    tail: list[str]


async def _pump(stream: asyncio.StreamReader, name: str, tail: list[str]) -> None:
    async for raw in stream:                               # line by line, never buffering it all
        line = raw.decode(errors="replace").rstrip()
        log.info("[%s] %s", name, line)
        tail.append(line)
        del tail[:-20]                                     # keep the last 20 lines only


async def _stop_group(proc: asyncio.subprocess.Process, grace: float = 2.0) -> None:
    if proc.returncode is not None:
        return
    try:
        os.killpg(proc.pid, signal.SIGTERM)                # the child and everything it spawned
    except ProcessLookupError:
        return
    try:
        async with asyncio.timeout(grace):
            await proc.wait()
    except TimeoutError:
        os.killpg(proc.pid, signal.SIGKILL)
        await proc.wait()


async def run_job(name: str, argv: list[str], timeout_s: float) -> JobResult:
    async with _slots:                                     # bound concurrent children
        started = time.monotonic()
        proc = await asyncio.create_subprocess_exec(
            *argv,
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.STDOUT,              # one ordered stream
            start_new_session=True,                        # own process group for clean kills
            limit=1024 * 1024,                             # tolerate long log lines
        )
        tail: list[str] = []
        timed_out = False
        try:
            async with asyncio.timeout(timeout_s):
                await _pump(proc.stdout, name, tail)
                await proc.wait()
        except TimeoutError:
            timed_out = True
        finally:
            await _stop_group(proc)                        # runs on timeout AND cancellation
        return JobResult(name, proc.returncode, round(time.monotonic() - started, 2), timed_out, tail)


def _write_report(path: Path, results: list[JobResult]) -> None:
    path.write_text(json.dumps([asdict(r) for r in results], indent=2))


async def main() -> None:
    logging.basicConfig(level=logging.INFO, format="%(message)s")
    py = sys.executable
    jobs = [
        ("ok", [py, "-c", "import time\nfor i in range(3): print('step', i, flush=True); time.sleep(0.1)"], 5),
        ("fails", [py, "-c", "import sys; print('bad input'); sys.exit(3)"], 5),
        ("hangs", [py, "-c", "import time; print('working', flush=True); time.sleep(60)"], 0.5),
    ]
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(run_job(n, argv, t)) for n, argv, t in jobs]
    results = [t.result() for t in tasks]
    await asyncio.to_thread(_write_report, Path("job-report.json"), results)   # file I/O off the loop
    for r in results:
        print(r.name, r.returncode, r.timed_out, r.seconds)


asyncio.run(main())

Running it interleaves the three jobs' output in the log, then prints ok 0 False 0.31, fails 3 False 0.01 and hangs -15 True 0.5: the successful job exits cleanly, the failing job's exit code is preserved, and the hanging job is terminated by SIGTERM exactly at its half-second budget rather than running for its full minute. Cancelling main() — for example during graceful shutdown — runs the same finally block, so no child outlives the runner.

Diagnostic Hook — are children and files hurting the loop?

Export four numbers. Live children (processes started minus processes reaped): it should return to zero between batches; a steady climb means a path gives up on a child without killing and waiting for it. Child duration and timeout rate per command: a rising timeout rate usually means the host is oversubscribed, not that the tool got slower. Default-executor queue depth or to_thread wait time: file work and blocking calls share it, and a growing wait shows file I/O crowding out everything else. Loop lag during file-heavy periods: any spike means a synchronous open()/read() is still on the loop. Alert on live children above MAX_PARALLEL for more than a minute, and on loop lag above 100 ms correlated with export or upload jobs.

Signals for children and file work A grid of 4 rows by 2 columns. Signals for children and file work signal healthy unhealthy trend means live children returns to zero children not reaped timeout rate flat and low host oversubscribed to_thread wait time near zero files crowd the executor loop lag during exports unchanged sync file I/O on loop Live children is the one to alert on first: leaked processes outlast restarts of the runner.

Failure modes

Failure mode Root cause Detection Fix
Loop freezes while a command runs subprocess.run() or check_output() called in a coroutine Loop lag equal to the command's duration create_subprocess_exec, or to_thread for library-mandated sync calls
Child processes accumulate Timeout or cancellation stops the wait but not the child Process count grows; orphans reparented to init Kill and await proc.wait() in finally
Child hangs with large output Parent awaits wait() with an undrained PIPE Child blocked in write; parent in wait Drain pipes concurrently, merge stderr, or use communicate()
wait() never returns after kill Grandchildren keep the pipes open Killed pid gone, but pipe readers still pending start_new_session=True and os.killpg()
ValueError: ... chunk exceed the limit A line longer than the stream reader limit Exception from readline() or async for Raise limit=, or read with read(n) chunks
Memory spike on large output communicate() buffering megabytes Parent RSS tracks child output size Stream and keep a bounded tail
File exports far slower than expected One await per line through aiofiles Export time scales with line count, CPU in thread handoffs Batch the whole operation into one to_thread call
Unrelated offloaded calls slow down File I/O saturating the default executor to_thread wait time rises during exports Dedicated executor for file work

Frequently Asked Questions

How do I run a subprocess without blocking the asyncio event loop?

Use asyncio.create_subprocess_exec with the program and arguments as separate strings, then await proc.communicate() for small outputs or read proc.stdout incrementally for large ones. The child's pipes and exit notification are handled by the event loop, so other tasks keep running. Avoid subprocess.run inside coroutines, because it blocks the loop.

Does asyncio.timeout kill a subprocess when it expires?

No. The timeout cancels the coroutine that is waiting on the process, but the child keeps running. Catch the TimeoutError, call proc.kill() or proc.terminate(), and then await proc.wait() so the process is reaped. Put that cleanup in a finally block so cancellation is handled the same way.

Why does my asyncio subprocess hang when it produces a lot of output?

The child's stdout or stderr pipe buffer, usually around 64 KiB, filled up because nothing was reading it, so the child blocked on write while the parent waited for it to exit. Read the pipes while the process runs, merge stderr into stdout, or use communicate(), which reads both streams concurrently.

Is there true asynchronous file I/O in Python asyncio?

Not for regular files. Operating system readiness APIs such as epoll treat disk files as always ready, so reads and writes still block in the kernel. aiofiles and asyncio.to_thread both run file operations in a thread pool. Batch whole file operations into a single thread call to keep the per-call overhead small.

Should I use aiofiles or asyncio.to_thread for file operations?

Both use threads. aiofiles gives an async file-like interface, which is convenient for streaming large files chunk by chunk, but awaiting many tiny operations is slow. asyncio.to_thread around a synchronous function that does the whole read or write is usually far faster and has no dependency. Choose based on how many awaits the operation needs.