Skip to content

Async File I/O with aiofiles vs asyncio.to_thread

A service that exports audit logs was "made async" by replacing open() with aiofiles.open() and adding await in front of every write(). Loop lag during exports disappeared — and the export itself became hundreds of times slower, so a job that used to take a second now holds a request open for minutes. The opposite mistake is just as common: plain open().write() inside a coroutine, harmless on a laptop SSD, blocking every request on the node for a quarter of a second when the file lives on network storage or an fsync hits a busy disk. Both come from the same misunderstanding. Operating systems do not provide readiness-based asynchronous I/O for regular files, so asyncio cannot do file I/O "natively"; every approach runs the blocking system calls on a thread. The only real questions are how many thread hand-offs an operation costs and which thread pool pays for them. This guide measures both and turns the results into rules.

Prerequisites

Thread hand-offs per operation 4 lanes over time. Thread hand-offs per operation loop, per line await await await await await await thread, per line w w w w w w loop, one call await to_thread thread, one call all writes time → The work is the same; the number of trips between loop and thread is not.

1. Measure what synchronous file I/O does to the loop

Start by proving the problem on your own storage. A heartbeat task that sleeps 5 ms and records how late it wakes up shows exactly how long the loop was blocked while a file operation ran.

import asyncio
import os
import tempfile
import time

PAYLOAD = os.urandom(1 << 20)                     # 1 MiB


def write_and_sync(path: str, mib: int) -> None:
    with open(path, "wb") as f:
        for _ in range(mib):
            f.write(PAYLOAD)
        f.flush()
        os.fsync(f.fileno())                      # force it to disk, as durable writers must


async def measure(label: str, work) -> None:
    loop = asyncio.get_running_loop()
    worst, stop = 0.0, False

    async def heartbeat() -> None:
        nonlocal worst
        while not stop:
            started = loop.time()
            await asyncio.sleep(0.005)
            worst = max(worst, loop.time() - started - 0.005)

    hb = asyncio.create_task(heartbeat())
    await asyncio.sleep(0.02)
    started = time.perf_counter()
    await work()
    elapsed = time.perf_counter() - started
    stop = True
    await hb
    print(f"{label:>18}: took {elapsed * 1000:5.0f} ms, worst loop lag {worst * 1000:6.1f} ms")


async def main() -> None:
    path = os.path.join(tempfile.mkdtemp(dir=os.path.expanduser("~")), "big.bin")

    async def on_loop() -> None:
        write_and_sync(path, 256)

    async def in_thread() -> None:
        await asyncio.to_thread(write_and_sync, path, 256)

    await measure("sync on the loop", on_loop)
    await measure("asyncio.to_thread", in_thread)
    os.remove(path)


asyncio.run(main())

Writing and syncing 256 MiB to a local disk took 140–170 ms either way, but on the loop the heartbeat was late by the full duration — 139 and 171 ms in two runs — while in a thread the worst lag was 1–2 ms. On network filesystems or saturated disks the same operation can take seconds, and every request on the process waits with it.

Verify: run the script on the storage your service actually uses (a mounted volume, not /tmp if production writes elsewhere). The "sync on the loop" lag should track the operation's duration; the thread version's lag should stay in single-digit milliseconds.

2. Compare per-call aiofiles with a batched thread call

aiofiles wraps each file method in a coroutine that runs the real method on the loop's default executor. That is correct, and it keeps the loop free — but every await is a separate hand-off to a thread and back. When the individual operations are tiny, the hand-offs dominate.

# pip install aiofiles
import asyncio
import os
import tempfile
import time

import aiofiles

LINES = [f"event {i} " + "x" * 80 + "\n" for i in range(20_000)]


def write_sync(path: str) -> None:
    with open(path, "w") as f:
        for line in LINES:
            f.write(line)


async def aiofiles_per_line(path: str) -> None:
    async with aiofiles.open(path, "w") as f:
        for line in LINES:
            await f.write(line)                   # one thread hand-off per line


async def aiofiles_batched(path: str) -> None:
    async with aiofiles.open(path, "w") as f:
        await f.write("".join(LINES))             # one hand-off for the data


async def to_thread_whole(path: str) -> None:
    await asyncio.to_thread(write_sync, path)     # one hand-off for everything


async def main() -> None:
    path = os.path.join(tempfile.mkdtemp(), "events.log")
    for label, fn in [("aiofiles, per line", aiofiles_per_line),
                      ("aiofiles, batched", aiofiles_batched),
                      ("to_thread, whole op", to_thread_whole)]:
        best = min([await timed(fn, path) for _ in range(3)])
        print(f"{label:>20}: {best * 1000:7.1f} ms")


async def timed(fn, path: str) -> float:
    started = time.perf_counter()
    await fn(path)
    return time.perf_counter() - started


asyncio.run(main())
Writing 20,000 lines to a local disk 4 bars comparing synchronous, blocks loop with the others. Writing 20,000 lines to a local disk synchronous, blocks loop 1.7 ms asyncio.to_thread, whole op 1.9 ms aiofiles, one batched write 2.7 ms aiofiles, await per line 545 ms Measured with the step 2 script, Python 3.13, aiofiles 25.1, best of three runs. Per-line awaits cost hundreds of times more than one thread call.

Best of three runs with Python 3.13 on a local disk: writing the 20,000 lines synchronously took 1.7 ms, a single to_thread call 1.9 ms, a single batched aiofiles write 2.7 ms, and aiofiles awaiting each line 545 ms. Reading the same file line by line with async for line in f through aiofiles took 426–554 ms, against 1.4 ms for one to_thread call that counted the lines. The disk did the same work in every case; the difference is 20,000 thread round trips.

Verify: the per-line variant should be two orders of magnitude slower than the others on your machine too. If it is not, your lines are large enough that the write itself dominates — which is exactly the case where per-call awaits are acceptable.

3. Batch whole file operations into one thread call

The rule that falls out of the measurement: write the file logic as an ordinary synchronous function, and cross into a thread once per logical operation. The function can open, loop, serialise, flush and fsync in plain Python with no awaits at all.

import asyncio
import json
import os
from pathlib import Path


def _atomic_write_ndjson(path: Path, records: list[dict]) -> int:
    tmp = path.with_suffix(path.suffix + ".tmp")
    with tmp.open("w", encoding="utf-8") as f:
        for record in records:
            f.write(json.dumps(record, separators=(",", ":")) + "\n")
        f.flush()
        os.fsync(f.fileno())                      # durable before it becomes visible
    os.replace(tmp, path)                         # atomic rename on POSIX
    return len(records)


def _read_ndjson(path: Path) -> list[dict]:
    with path.open(encoding="utf-8") as f:
        return [json.loads(line) for line in f]


async def export_events(path: Path, records: list[dict]) -> int:
    return await asyncio.to_thread(_atomic_write_ndjson, path, records)


async def import_events(path: Path) -> list[dict]:
    return await asyncio.to_thread(_read_ndjson, path)

This shape also makes correctness easier. The temporary-file-plus-os.replace pattern, the fsync, and the serialisation all live in one synchronous function that can be unit-tested without an event loop, and asyncio.to_thread() copies context variables so log lines written inside it keep their request IDs.

Verify: exporting 20,000 records costs one executor job — confirm by wrapping the default executor's submit with a counter in a test — and a reader never sees a partially written file, even if the process is killed mid-export.

4. Stream large files with large chunks

Batching the whole operation is not possible when the data does not fit in memory: hashing a multi-gigabyte upload, streaming a file to a client, copying between storage tiers. Then per-call awaits are unavoidable, and chunk size decides the number of hand-offs. Use large chunks.

import asyncio
import hashlib

import aiofiles


async def sha256_file(path: str, chunk_size: int = 1 << 20) -> str:
    digest = hashlib.sha256()
    async with aiofiles.open(path, "rb") as f:
        while chunk := await f.read(chunk_size):  # one hand-off per MiB
            digest.update(chunk)
    return digest.hexdigest()


def _sha256_sync(path: str, chunk_size: int = 1 << 20) -> str:
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        while chunk := f.read(chunk_size):
            digest.update(chunk)
    return digest.hexdigest()


async def sha256_file_one_hop(path: str) -> str:
    return await asyncio.to_thread(_sha256_sync, path)    # when nothing else needs the chunks

Hashing a 64 MiB file took 463 ms with aiofiles and 4 KiB chunks, 43 ms with 1 MiB chunks, and 33 ms as a single to_thread call. The aiofiles form still earns its place when each chunk must be handed to something asynchronous between reads — writing it to a socket, uploading it with an async client — because a single thread call cannot await in between. For sending a file to a socket, loop.sendfile() goes further and lets the kernel copy the data without passing it through Python at all.

Verify: time your streaming path at 4 KiB, 64 KiB and 1 MiB chunks; the gain flattens once chunk handling rather than hand-offs dominates, which marks the chunk size to use.

Hashing a 64 MiB file 3 bars comparing aiofiles, 4 KiB chunks with the others. Hashing a 64 MiB file aiofiles, 4 KiB chunks 463 ms aiofiles, 1 MiB chunks 43 ms one asyncio.to_thread call 33 ms Measured with the step 4 functions, Python 3.13, local disk, best of three runs. Chunk size, not the library, decides the cost of a streamed file operation.

5. Give file work its own bounded executor

Both asyncio.to_thread() and aiofiles use the loop's default executor, whose size is min(32, os.cpu_count() + 4) threads. File work shares those threads with every other blocking call in the process. A burst of slow reads from network storage can occupy them all, and unrelated to_thread calls — a blocking SDK, a DNS lookup — queue behind it. Put heavy file work on its own executor and bound how much of it can queue.

import asyncio
import contextvars
import functools
from concurrent.futures import ThreadPoolExecutor

FILE_IO = ThreadPoolExecutor(max_workers=8, thread_name_prefix="file-io")
_file_slots = asyncio.Semaphore(32)                     # max jobs running or queued


async def run_file_io(fn, /, *args, **kwargs):
    async with _file_slots:                              # backpressure before the executor queue
        loop = asyncio.get_running_loop()
        ctx = contextvars.copy_context()                 # keep request context in the thread
        return await loop.run_in_executor(FILE_IO, functools.partial(ctx.run, fn, *args, **kwargs))


async def shutdown_file_io() -> None:
    await asyncio.to_thread(FILE_IO.shutdown, wait=True)  # finish in-flight writes on exit

Size the file executor for the storage, not the CPU: a local disk handles many parallel operations, while a single NFS mount or a rate-limited cloud volume may do better with a handful. The semaphore stops a flood of export requests from queueing thousands of jobs in memory, the same backpressure applied to any other queue.

Verify: during a synthetic burst of file jobs, the wait time of an unrelated asyncio.to_thread() call stays near zero, and memory stays flat as file requests beyond the semaphore wait in the application rather than the executor queue.

Verification

File I/O is handled well when:

  • The loop never touches files directly: loop lag does not rise during exports, uploads or log rotation, on the storage used in production.
  • Hand-offs are proportional to work, not to lines: whole operations run in one thread call, and streamed operations use chunks of hundreds of kilobytes or more.
  • Writes are durable and atomic where it matters: exports use a temporary file, fsync and os.replace.
  • File work is isolated: heavy file I/O runs on its own executor, and blocking calls elsewhere are not delayed by it.
  • Intake is bounded: a semaphore caps queued file jobs, and shutdown waits for in-flight writes.

Pitfalls & edge cases

  • os.path.exists() and stat() on slow storage. Metadata calls block too, and on network filesystems they can be the slowest operations of all. Fold them into the same thread call as the read or write they guard.
  • Logging handlers writing to files. logging.FileHandler writes synchronously on the calling thread — the loop. Under heavy logging, use a QueueHandler with a QueueListener so file writes happen on a background thread.
  • Closing files in __del__. An aiofiles object that is garbage-collected without await f.close() may close on an arbitrary thread or not flush. Always use async with.
  • Temporary directories on different filesystems. os.replace() is atomic only within one filesystem. Create the temporary file next to the destination, not in /tmp.
  • Assuming the page cache hides slow disks. Writes appear instant until the kernel starts throttling writers under dirty-page pressure, at which point even small writes block. Measure with fsync enabled, as step 1 does.

Frequently Asked Questions

Is aiofiles truly asynchronous file I/O?

No. Operating systems do not provide readiness-based non-blocking I/O for regular files, so aiofiles runs each file operation on a thread pool and awaits the result. That keeps the event loop responsive, but every awaited call costs a thread hand-off, which becomes very expensive when operations are small and numerous.

Is asyncio.to_thread faster than aiofiles?

For many small operations, yes, when the whole operation is done in a single to_thread call. Writing 20,000 short lines took about 2 ms as one to_thread call and about 545 ms with aiofiles awaiting each line. With large chunks or a single batched call, aiofiles performs similarly, because both use threads underneath.

Does reading a file in a coroutine block the asyncio event loop?

Yes. open, read, write, fsync and even os.stat are blocking system calls, and running them directly in a coroutine stalls every other task until they return. On fast local disks this can be brief, but on network filesystems or busy disks it can take hundreds of milliseconds or more.

What chunk size should I use when streaming files in asyncio?

Use large chunks, typically 256 KiB to 1 MiB, so that the per-chunk thread hand-off is small compared with the work on each chunk. In our measurement, hashing a 64 MiB file through aiofiles took 463 ms with 4 KiB chunks and 43 ms with 1 MiB chunks.