Skip to content

Watching Files for Changes in asyncio

Reloading configuration without a restart, picking up dropped files from a spool directory, rebuilding on save — all of them need an answer to "has this file changed?", and the obvious answer, a thread that calls os.stat in a loop, is both wasteful and wrong. Wasteful because most of the time nothing changed; wrong because a file rewritten twice within one poll interval looks like one change, and a file replaced by rename changes inode without necessarily changing size or mtime in a way your comparison catches. Linux has offered a push-based alternative since 2005, and asyncio can consume it directly through loop.add_reader without any extra thread. This guide builds that, adds the debouncing every real watcher needs, and covers the cross-platform option for when Linux-only is not acceptable.

Prerequisites

  • Python 3.11+. The inotify section is Linux-specific and uses ctypes; the watchfiles section works on Linux, macOS and Windows (pip install watchfiles).
  • File I/O in async code from Subprocesses & File I/O, because reading the changed file still blocks.
  • Bounded queues from Async Queue Management, which is where events land.
Three ways to notice a file changed A grid of 4 rows by 2 columns. Three ways to notice a file changed approach latency notes stat polling in a thread half the interval portable, misses fast rewrites inotify + add_reader under a millisecond Linux only, no extra thread watchfiles (awatch) tens of milliseconds cross-platform, batches for you no watching at all next restart often the right answer Measured here: 0.29 ms for inotify, 101 ms for a watchfiles batch at its default step.

1. Understand what polling actually costs

Polling is not expensive in CPU terms: 1,000 os.stat calls on a directory took 1.5 ms on this machine, so a one-second poll is free. Its real costs are latency — a change is noticed after half the interval on average — and correctness. The signature you compare matters:

async def changed(path: str) -> bool:
    st = await asyncio.to_thread(os.stat, path)
    return (st.st_mtime_ns, st.st_size, st.st_ino)     # all three, not just mtime

st_mtime_ns rather than st_mtime because the float loses precision on rapid rewrites, and st_ino because an atomic replace gives the path a new inode — which is how most editors and every well-behaved config manager write files. Even with all three, two changes inside one interval collapse into one, which is fine for "reload config" and fatal for "process every file that appears".

Note the to_thread: os.stat blocks, and a stat on a network filesystem can block for seconds. Polling from the loop thread directly is a slow-callback waiting to happen.

Verify: rewrite the file twice within one interval and confirm the poller reports a single change.

2. Read inotify events straight from the event loop

inotify gives you a file descriptor that becomes readable when something happens. That is exactly what loop.add_reader wants, so no thread is required. The descriptor comes from libc through ctypes — about fifteen lines, and the payload format is fixed:

import ctypes
import os
import struct

IN_MODIFY, IN_CLOSE_WRITE, IN_MOVED_TO = 0x2, 0x8, 0x80
IN_CREATE, IN_DELETE = 0x100, 0x200
_libc = ctypes.CDLL("libc.so.6", use_errno=True)
_HEADER = struct.Struct("iIII")                        # wd, mask, cookie, name length


def inotify_init() -> int:
    fd = _libc.inotify_init1(os.O_NONBLOCK | os.O_CLOEXEC)
    if fd < 0:
        raise OSError(ctypes.get_errno(), "inotify_init1")
    return fd


def add_watch(fd: int, path, mask: int) -> int:
    wd = _libc.inotify_add_watch(fd, os.fsencode(str(path)), mask)
    if wd < 0:
        raise OSError(ctypes.get_errno(), f"inotify_add_watch {path}")
    return wd


def read_events(fd: int):
    data = os.read(fd, 65536)                          # non-blocking: readable means ready
    offset = 0
    while offset < len(data):
        wd, mask, cookie, length = _HEADER.unpack_from(data, offset)
        offset += _HEADER.size
        name = data[offset:offset + length].split(b"\0", 1)[0].decode()
        offset += length
        yield wd, mask, name

Registering it with the loop is three lines, and the callback does nothing but move events onto a queue:

fd = inotify_init()
add_watch(fd, directory, IN_CLOSE_WRITE | IN_MOVED_TO | IN_CREATE | IN_DELETE)
queue: asyncio.Queue = asyncio.Queue()
loop.add_reader(fd, lambda: [queue.put_nowait((m, n)) for _, m, n in read_events(fd)])

Writing a file in that directory delivered the event 0.29 ms later — three orders of magnitude faster than a one-second poll, with no thread and no work while idle. Remember to loop.remove_reader(fd) and os.close(fd) on shutdown; the descriptor is a real resource and the watch persists until it is closed.

Verify: creating a file in the watched directory produces an event within a millisecond.

An inotify event on its way to your reload 5 stages from kernel event to one reload. An inotify event on its way to your reload kernel event on the watch descriptor add_reader fires no thread involved parse and queue never blocks the loop debounce window collect the burst one reload validated before swap The reader callback is synchronous: read, parse, enqueue, return.

3. Watch for the right events

The event mask is where most watchers go wrong. A direct write to a new file produced two events — IN_CREATE (0x100) then IN_CLOSE_WRITE (0x8) — and writing five chunks with a flush after each produced the same two, because IN_MODIFY was not in the mask. That is the behaviour you want for "the file is now complete".

An atomic replace, the way editors and configuration managers save, produced three events:

[('0x100', '.config.yaml.tmp'), ('0x8', '.config.yaml.tmp'), ('0x80', 'config.yaml')]

The first two name the temporary file; only the third, IN_MOVED_TO (0x80), names the file you care about. A watcher that only looks for IN_CLOSE_WRITE on config.yaml will never fire for an atomically saved config — the single most common "my file watcher does not work" report.

The practical mask is IN_CLOSE_WRITE | IN_MOVED_TO for "a complete file is now at this path", plus IN_CREATE | IN_DELETE if you track the directory's contents. Reserve IN_MODIFY for tailing a file that is appended to in place, such as a log.

One more structural point: watch the directory, not the file. An inotify watch follows the inode, so a watch on config.yaml stops reporting the moment an atomic replace gives that name a different inode.

Verify: save the watched file with your editor and confirm an IN_MOVED_TO event arrives for the real filename.

What a save actually looks like 2 columns contrasting direct write, write temp, then rename. What a save actually looks like direct write open, write, close IN_CREATE for the name IN_CLOSE_WRITE when closed one event per flush if you watch IN_MODIFY readers can see a half-written file write temp, then rename what editors and config tools do events name the temp file IN_MOVED_TO for the real name the inode changes readers see old or new, never partial Watch IN_CLOSE_WRITE and IN_MOVED_TO; ignore IN_MODIFY unless you are tailing.

4. Debounce the burst into one reload

Every save produces several events, and a build tool or a git checkout produces hundreds. Reloading per event is both wasteful and racy — you may read the file between two writes. The fix is a quiet window: after the first event, keep collecting until nothing has happened for a short interval.

async def debounced(queue: asyncio.Queue, *, quiet: float = 0.2):
    """Yield once per burst: one event, then `quiet` seconds of silence."""
    while True:
        names = {await queue.get()}                    # block until something happens
        while True:
            try:
                names.add(await asyncio.wait_for(queue.get(), quiet))
            except TimeoutError:
                break                                  # the burst is over
        yield names

Six events 30 ms apart collapsed into a single reload, 354 ms after the first event — 150 ms of burst plus the 200 ms quiet window. Tune quiet against your writers: 200 ms is comfortable for editors, while a slow build may need a second. If bursts can be continuous, add a maximum delay so a busy directory still reloads eventually.

watchfiles does this for you. awatch() batches changes and yields sets, and its defaults produced one batch of five changes 101 ms after the first write:

from watchfiles import awatch

async for changes in awatch("/etc/myservice", debounce=200, step=50):
    print(changes)      # {(Change.added, '/etc/myservice/f0.txt'), ...}

It is Rust-backed, uses inotify on Linux, FSEvents on macOS and ReadDirectoryChangesW on Windows, and it costs one dependency and one background thread. For anything that must run on more than Linux, it is the right default.

Verify: a burst of writes produces exactly one reload, and the reload happens after the last write, not the first.

Six events, one reload 3 lanes over time. Six events, one reload file events e1 e2 e3 e4 e5 e6 quiet window restarted by each event reload runs once time → Measured: six events 30 ms apart produced one reload 354 ms after the first.

5. Validate before swapping

A watcher that reloads on every change will eventually read a file someone is halfway through editing, or one with a typo in it. Since the whole point is to avoid a restart, the reload must never be able to take the service down:

def load(text: str) -> dict:
    data = dict(line.split("=") for line in text.strip().splitlines())
    workers = int(data["workers"])
    if not 1 <= workers <= 64:
        raise ValueError(f"workers out of range: {workers}")
    return {"workers": workers}


async def reload(path, current: dict) -> dict:
    text = await asyncio.to_thread(pathlib.Path(path).read_text)
    try:
        new = load(text)                               # parse and validate first
    except Exception:
        logging.exception("config reload rejected, keeping previous")
        return current                                 # atomic: swap only on success
    return new

Running that over three inputs: workers=8 was applied, and both workers=999 and workers=oops were rejected with the previous value retained. The swap is a single assignment, so readers never observe a half-built configuration — the same reasoning as reloading configuration on SIGHUP, which is the other half of this pattern.

Verify: write an invalid config and confirm the service logs a rejection and keeps serving with the old values.

Verification

A file watcher is production-ready when:

  • The loop is never blocked: reads and stats happen in threads or in a non-blocking reader callback.
  • Atomic saves are seen: replacing the file by rename triggers a reload.
  • Bursts collapse: a rapid series of writes produces one reload, after the last write.
  • Bad input is survivable: an invalid file is rejected and the previous state is kept.
  • Descriptors are released: remove_reader and close run on shutdown, and the watch count does not grow with reloads.

Pitfalls & edge cases

  • Watching the file rather than its directory. The watch follows the inode; an atomic replace silently orphans it.
  • inotify limits. fs.inotify.max_user_watches caps recursive watches, and exceeding it raises OSError: [Errno 28] No space left on device — a confusing message for a watch limit.
  • Network and container filesystems. NFS, SMB and some overlay mounts do not generate inotify events for changes made elsewhere; polling is the only option there.
  • Recursive watching. inotify is not recursive: every subdirectory needs its own watch, added as directories appear. This is most of what watchfiles does for you.
  • Reading inside the reader callback. The callback is synchronous on the loop; parse only the event structure there and read the file from a task.
  • Reloading things that cannot be reloaded. Listening sockets, pool sizes and worker counts usually need a restart; make the watcher change only what is genuinely swappable.

Frequently Asked Questions

How do I watch a file for changes in asyncio without polling?

On Linux, create an inotify descriptor with inotify_init1 through ctypes, add a watch on the containing directory, and register the descriptor with loop.add_reader. The callback parses events and puts them on a queue — no thread, and events arrive in well under a millisecond. For cross-platform code, use the watchfiles library's awatch.

Why doesn't my file watcher fire when I save in my editor?

Because the editor wrote a temporary file and renamed it over the original. The events name the temporary file, and the final one is IN_MOVED_TO for the real name. Watch the directory rather than the file, and include IN_MOVED_TO in the mask alongside IN_CLOSE_WRITE.

How do I debounce file change events in asyncio?

Await the first event, then keep collecting with asyncio.wait_for(queue.get(), quiet) until it times out, and act once on the collected set. A 200 ms quiet window turned six events 30 ms apart into a single reload. The watchfiles library applies the same idea with its debounce and step parameters.

Is polling with os.stat good enough for config reloads?

Often, yes. A thousand stat calls take about 1.5 ms, so a one-second poll costs nothing measurable, and configuration rarely needs sub-second reaction. Compare st_mtime_ns, st_size and st_ino rather than mtime alone, and run the stat in a thread so a slow filesystem cannot stall the loop.

Does inotify work inside Docker containers and on network filesystems?

Inside a container, yes, for changes made in the same kernel namespace — including bind-mounted host directories on Linux. It does not work for changes made on the far side of NFS, SMB or some virtualised filesystem drivers, because no local kernel event is generated. Fall back to polling there.