Skip to content

Memory & Resource Leaks in Asyncio Services

A leaking async service does not degrade: it runs perfectly until the container is killed, restarts, and runs perfectly again. That pattern — a sawtooth in the memory graph and a restart count nobody looks at — hides four different problems with four different tools. Python objects accumulating in a cache are found with tracemalloc. Tasks that never complete are found by counting and naming them; each costs only 1,936 bytes, but holds its entire coroutine frame alive. File descriptors leak two at a time per unclosed HTTP client, measured, and hit a limit that is often 1,024 in a container. And native memory from C extensions is invisible to tracemalloc entirely.

The first question in any investigation is therefore which of those four you are in, and it is answered by comparing two numbers you should already export: resident memory and tracemalloc's own total. If they rise together, it is Python objects. If RSS rises alone, it is native or fragmentation. If neither moves while the service still dies, it is descriptors or tasks. This section covers each in turn, measured. The parent section, Resilience, Cancellation & Error Handling, covers the ownership patterns that prevent most of them.

Scope of this section:

  • Locating Python-object growth with tracemalloc snapshot diffs.
  • Counting and diagnosing file descriptor leaks, including ResourceWarning.
  • The unclosed-client-session warnings, what they mean and how to fix them.
  • Tracking task growth, and naming tasks so the leak identifies itself.
  • Giving every resource an owner, so the leak cannot be written in the first place.

Architectural principles

  • Every resource has exactly one owner. A client, a task, a pool or a cache entry either lives in an async with, in the lifespan, or in a tracked registry that is cancelled at shutdown. Anything with no answer to "who closes this?" is the thing that leaks.
  • Bound every collection. A dictionary keyed by user input with no maxsize is a memory leak with a schema. Caches, in-flight maps, lock tables and retry state all need limits.
  • Export the four counters from day one. Resident memory, traced memory, descriptor count and live task count. Each is one line, and each turns a 3 a.m. crash into a graph that was rising for a week.
  • Trends matter, thresholds do not. A healthy service has hundreds of tasks and dozens of descriptors. The signal is a count that never returns to its baseline.
  • Make leaks fail tests. filterwarnings = error::ResourceWarning and an assertion that asyncio.all_tasks() is empty after teardown catch the leak at the commit that introduces it.
Four things an async service leaks, and how each is seen A grid of 4 rows by 2 columns. Four things an async service leaks, and how each is seen leaks signal tool Python objects RSS and traced memory rise together tracemalloc snapshots tasks all_tasks() climbs monotonically names and get_stack() descriptors fd count rises with traffic /proc/self/fd, ResourceWarning native memory RSS rises, traced memory flat a native profiler The first question in any leak investigation is which row you are in.

Execution model: why async leaks differ

Three properties of asyncio change what leaking looks like.

A suspended coroutine is a live object graph. A task waiting on a socket holds its frame, and the frame holds every local — the request, the parsed body, the buffer, the connection. So a task leak is a memory leak whose size is set by what the handler happened to have in scope, and 10,000 leaked tasks held 18.5 MiB in testing before counting anything their frames referenced.

Cleanup is cooperative. A garbage-collected client does not close its sockets promptly; a cancelled task does not release its resources until its finally runs; a close() that is never awaited does nothing. Every resource release in an async service is a step someone must write, and the failures are silent by design.

Everything is shared. One event loop, one process, one descriptor table. A leak in a rarely used code path consumes the same limits as the hot one, and the symptom appears in whichever request happens to need a descriptor when the table is full — which is almost never the request that leaked it.

There is a fourth property that makes async leaks harder to reason about than threaded ones: the absence of a stack. A leaked thread can be found in a stack dump, with a traceback showing exactly where it is stuck. A leaked task is an object in a set, and its traceback exists only if you ask for it with get_stack(). That is why naming tasks matters so much more here than naming threads does elsewhere — the name is frequently the only human-readable thing attached to a leak until someone writes the introspection code, and writing it during an incident is the wrong time.

What each leaked thing actually costs 4 stacked layers from a pending task to the limits. What each leaked thing actually costs a pending task 1,936 bytes each, measured plus its whole coroutine frame an unclosed HTTP client 2 file descriptors, measured plus its connection pool an unbounded cache entry the object, forever until the container is killed the limits RLIMIT_NOFILE, often 1,024 the container memory limit Verified end to end: 800 stuck tasks and 4,000 cached buffers produced a 31.7 MiB rise the watchdog explained by itself.

Pattern catalogue

Diff two snapshots to find the growing line

tracemalloc.start(10)
await warm_up()
gc.collect()
baseline = tracemalloc.take_snapshot()
await run_workload()
gc.collect()
for stat in tracemalloc.take_snapshot().compare_to(baseline, "lineno")[:5]:
    print(stat)

The leaking line appeared at +19,658 KiB against noise measured in fractions of a kilobyte. See finding memory leaks with tracemalloc.

Count descriptors, and read them by kind

OPEN_FDS.set(len(os.listdir("/proc/self/fd")))

A healthy process showed {'socket': 4, 'file': 3, 'anon_inode': 1}; 50 unclosed clients added 100 sockets. See detecting leaked sockets and file descriptors.

Name tasks so the histogram names the bug

task = asyncio.create_task(consume(queue), name=f"consumer:{queue_name}")
...
collections.Counter(t.get_name().rsplit(":", 1)[0] for t in asyncio.all_tasks())

Which produced {'leak': 10000, 'Task': 1} — the spawner, with no code reading. See tracking task growth.

Give background work a self-cleaning registry

def spawn(coro, name: str) -> asyncio.Task:
    task = asyncio.create_task(coro, name=name)
    _background.add(task)
    task.add_done_callback(_background.discard)
    task.add_done_callback(_log_failure)
    return task

Fifty tracked tasks left the registry at zero once they completed — the size is itself a metric.

Make the warnings fatal where it is cheap

# pytest.ini
filterwarnings = error::ResourceWarning

An unclosed aiohttp session reports Unclosed client session and Unclosed connector; httpx reports the socket with its remote address. See fixing unclosed client session warnings.

Bound the collections that grow with input

self.entries: OrderedDict[str, tuple[float, bytes]] = OrderedDict()
...
while len(self.entries) > maxsize:
    self.entries.popitem(last=False)                   # LRU eviction, always

The same applies to in-flight maps, per-key lock dictionaries, retry state and deduplication sets. Each is keyed by something a user controls, so each grows without limit unless something removes entries — and the removal is easy to forget precisely because these structures work perfectly in testing, where the key space is small.

Close in the right order at shutdown

    finally:
        health.ready = False                           # stop new traffic
        await drain_in_flight()                        # let running work finish
        for task in list(_background):
            task.cancel()
        await asyncio.gather(*_background, return_exceptions=True)
        await client.aclose()                          # then close what they were using
        await pool.close()

Closing clients before the tasks that use them produces a second class of error — failed in-flight work and "connection closed" exceptions during every deploy — so the order is as much a correctness property as the closing itself.

From a rising graph to a line of code 5 stages from a gauge trends up to fix ownership. From a rising graph to a line of code a gauge trends up memory, tasks or fds pick the category compare RSS and traced apply the tool snapshots, names, fds name the site file, line, spawner fix ownership bound, close, cancel Each category has exactly one tool; the cost is knowing which category you are in.

Resource boundaries

Resource What consumes it How to size and bound it
Heap Cached objects, buffers, suspended frames maxsize on every cache; measured diffs
Tasks One per spawn that never completes A registry with done callbacks; TaskGroup where possible
Descriptors Clients, files, pipes, event loops One client per process; async with elsewhere
Connection pools max_size per worker process Divide the server's limit by the worker count
Executor threads to_thread and sync dependencies A dedicated executor for heavy blocking work
Event loops One per process; more if created manually asyncio.run, or an explicit close()
tracemalloc overhead Frames kept per allocation Temporary only: 0.05 µs → 5.96 µs per allocation at 25 frames

The pool row is the one that turns a scaling change into an outage: max_size=20 with eight workers is 160 connections, and the database's limit is reached during the deploy that added the workers. The executor row is the quietest: threads held by blocking calls are invisible in every metric except the latency of unrelated work that was queued behind them, which is why a dedicated executor for blocking work is worth its configuration. And the event-loop row matters in exactly one situation — code that creates loops manually, in a bridge or a test helper — where each unclosed loop is an eventpoll descriptor and a set of pools that nothing will ever reclaim.

Integrated production example

A watchdog task that samples the four counters, and — when any of them grows beyond a threshold — explains the growth itself with a tracemalloc diff, a task-name histogram and a descriptor breakdown.

class ResourceWatchdog:
    """One task that watches memory, descriptors and tasks, and explains growth itself."""

    def __init__(self, *, interval=30.0, task_growth=500, fd_growth=100, mem_growth_mb=50):
        self.interval = interval
        self.task_growth, self.fd_growth, self.mem_growth = task_growth, fd_growth, mem_growth_mb
        self.baseline = None
        self.snapshot = None

    def sample(self) -> dict:
        usage = resource.getrusage(resource.RUSAGE_SELF)
        return {
            "tasks": len(asyncio.all_tasks()),
            "fds": len(os.listdir("/proc/self/fd")),
            "rss_mb": usage.ru_maxrss / 1024,
            "traced_mb": tracemalloc.get_traced_memory()[0] / 1024 / 1024,
        }

    async def run(self, stop: asyncio.Event) -> None:
        tracemalloc.start(10)                                  # temporary, for this investigation
        gc.collect()
        self.baseline, self.snapshot = self.sample(), tracemalloc.take_snapshot()
        while not stop.is_set():
            with contextlib.suppress(TimeoutError):
                async with asyncio.timeout(self.interval):     # the sleep doubles as the stop wait
                    await stop.wait()
            now = self.sample()
            if (now["tasks"] - self.baseline["tasks"] > self.task_growth
                    or now["fds"] - self.baseline["fds"] > self.fd_growth
                    or now["traced_mb"] - self.baseline["traced_mb"] > self.mem_growth):
                log.warning("resource growth", **self.explain(now))
                self.baseline = now                            # re-baseline after reporting
                gc.collect()
                self.snapshot = tracemalloc.take_snapshot()

    def explain(self, now: dict) -> dict:
        gc.collect()
        current = tracemalloc.take_snapshot()
        top = current.compare_to(self.snapshot, "lineno")[:2]
        memory = [f"+{s.size_diff / 1024:.0f} KiB "
                  f"{os.path.basename(s.traceback[0].filename)}:{s.traceback[0].lineno}"
                  for s in top if s.size_diff > 0]
        names = collections.Counter(
            t.get_name().rsplit("-", 1)[0] for t in asyncio.all_tasks())
        kinds = collections.Counter()
        for fd in os.listdir("/proc/self/fd"):
            try:
                target = os.readlink(f"/proc/self/fd/{fd}")
            except OSError:
                continue                                       # closed under us: ignore
            kinds[target.split(":")[0] if ":" in target else "file"] += 1
        return {"delta": {k: round(now[k] - self.baseline[k], 1) for k in now},
                "memory": memory, "tasks": dict(names.most_common(2)), "fds": dict(kinds)}

Run against a service leaking both ways — 800 tasks stuck on an event that is never set, and 4,000 buffers added to an unbounded cache — the watchdog emitted a single report:

delta:  {'tasks': 800, 'fds': 0, 'rss_mb': 33.4, 'traced_mb': 31.7}
memory: ['+31078 KiB leaktopic.py:65 LEAKY_CACHE[i] = bytearray(20_000)', '+594 KiB locks.py:168']
tasks:  {'stuck': 800, 'Task': 1}
fds:    {'socket': 3, 'file': 2, 'anon_inode': 1}

The line that leaks, the number of tasks and the spawner that made them, and a descriptor table showing that descriptors were not the problem — in one log entry, without anyone attaching a debugger.

Diagnostic Hook — the four counters, and what their shapes mean

Export RSS, traced memory, open descriptors and live tasks, sampled every 15–30 seconds. Then read them as a set rather than individually. RSS and traced rising together is a Python object leak, and tracemalloc will name the line. RSS rising while traced is flat is native memory or fragmentation, and no Python tool will help — look at C extensions. Descriptors rising with traffic is an unclosed client, and the /proc/self/fd breakdown names the kind. Tasks rising monotonically is a spawner with no owner, and a Counter over task names identifies it. Alert on any of the four failing to return to its baseline over an hour, rather than on absolute values, and include the four deltas in every restart notification so the next occurrence starts with data.

Who owns this resource? A decision on How long should it live with 3 outcomes. Who owns this resource? How long should it live? one operation async with closed on every path the process the lifespan created once, closed once longer than a request a tracked registry cancelled at shutdown Anything with no answer to this question is the thing that will leak.

Failure modes

Failure mode Root cause Detection Fix
Sawtooth memory, periodic restarts Unbounded cache or collection RSS and traced memory rise together maxsize and eviction; verified by snapshot diff
Errno 24: Too many open files Client created per request Descriptor count rises with traffic One client in the lifespan
Growing memory with no traced growth Native allocation or fragmentation RSS rises, tracemalloc flat Native profiler; check C extensions
Task count climbing forever Tasks waiting on something that never happens all_tasks() never returns to baseline Registry plus cancellation at shutdown
Unclosed client session in logs Session garbage collected while open ResourceWarning at collection time async with, or lifespan ownership
Memory held by one stuck request A suspended frame keeping objects alive A task suspended for hours Timeouts on every await
Pool exhaustion after scaling Pool size multiplied by workers Connections ≈ max_size × workers Divide by the worker count
Leak reaches production repeatedly Nothing fails the build No teardown assertions Fatal ResourceWarning; assert no tasks survive

Frequently Asked Questions

How do I tell what kind of leak an async service has?

Compare resident memory with tracemalloc's traced total. Rising together means Python objects, and snapshot diffs will name the line. RSS alone means native memory or fragmentation. Neither rising while the process still fails means descriptors or tasks — count both.

Do leaked asyncio tasks actually use much memory?

The task object is small — 1,936 bytes, measured — but a suspended coroutine holds its entire frame, including the request, buffers and connection it was using. Ten thousand leaked tasks held 18.5 MiB of traced memory before counting what their frames referenced.

Why does my service run out of file descriptors?

Almost always a client created per request instead of once per process. Measured, 50 unclosed httpx clients held 100 descriptors — two each — while 50 requests through one shared client used two. Container limits are often 1,024, so this reaches failure quickly.

Can I leave tracemalloc enabled in production?

Not permanently. It costs 0.05 µs to 5.96 µs per allocation depending on frame depth — over a hundred times the untraced cost. Enable it for an investigation via PYTHONTRACEMALLOC or an admin endpoint, take snapshots, and turn it off again.

How do I stop leaks reaching production at all?

Make them fail the build: filterwarnings = error::ResourceWarning in pytest, and an assertion in teardown that asyncio.all_tasks() contains nothing the test created. Both catch the leak at the commit that introduces it rather than in a restart graph weeks later.