Skip to content

Finding Memory Leaks in asyncio with tracemalloc

A leaking async service looks the same as a healthy one until the container is killed: steady traffic, normal latency, resident memory climbing a few megabytes an hour. tracemalloc turns that into a file and line number in about ten minutes of work, because it records where every Python allocation came from and can diff two points in time. In the verified example below, 2,000 requests through a handler with a three-character mistake produced a top entry of +19,658 KiB on the exact line. The cost of leaving it on is what stops this being the default: allocation went from 0.05 µs to 5.96 µs with 25 frames of traceback, a 113x multiplier on allocation-heavy code.

Prerequisites

  • Python 3.11+; tracemalloc is standard library.
  • A reproducible workload — a load test, or production traffic you can sample.
  • Baseline metrics from exporting Prometheus metrics from asyncio, so you know memory is actually growing.
Finding a leak with two snapshots 5 ordered steps. Finding a leak with two snapshots tracemalloc.start(10) frames decide the detail warm up first so caches are not the diff take a baseline snapshot after a gc.collect() run the workload hundreds of iterations compare_to(snapshot, "lineno") the leak is the top line Verified: 2,000 leaking requests put the offending line at the top with +19.6 MiB.

1. Take two snapshots around the workload

The whole technique is a diff:

import gc
import tracemalloc

tracemalloc.start(10)                                  # keep 10 frames per allocation

await warm_up()                                        # fill caches and pools first
gc.collect()
baseline = tracemalloc.take_snapshot()

await run_workload()                                   # the thing you suspect
gc.collect()
current = tracemalloc.take_snapshot()

for stat in current.compare_to(baseline, "lineno")[:10]:
    print(stat)

The warm_up() matters more than it looks. Without it the diff is dominated by one-time allocations — connection pools, compiled regexes, lazily imported modules — and the real growth is buried. gc.collect() before each snapshot removes cycles that would otherwise appear as growth and then disappear.

On the leaking handler, the top three entries were unambiguous:

+ 19658.3 KiB  memleak.py:6   LEAK.append(bytearray(10_000))
+     0.3 KiB  tracemalloc.py:560
+     0.3 KiB  tracemalloc.py:423

One line, four orders of magnitude above the noise. That is what a real leak looks like in a diff; if the top entries are all small and similar, you probably have fragmentation or a native allocation rather than a Python-object leak.

Verify: the top entry's size difference is a meaningful fraction of the growth you measured in RSS.

2. Choose the grouping that answers your question

compare_to takes a key, and the right one depends on where you are in the investigation:

key groups by use when
"lineno" file and line finding the allocating line — start here
"traceback" the full call path the line is inside a helper that everything calls
"filename" module narrowing to a library first

"traceback" is the one that solves the awkward case. If the top line is data = await response.json() inside a shared HTTP wrapper, the line tells you nothing; the traceback tells you which caller is holding the results:

top = current.compare_to(baseline, "traceback")[0]
for line in top.traceback.format():
    print(line)

The number of frames you get is whatever you passed to tracemalloc.start(). Ten is usually enough to reach your own code through a framework; 25 is the practical maximum before the overhead becomes annoying even for a debugging session.

Verify: the traceback reaches code you own, not just library internals.

What each comparison key tells you A grid of 4 rows by 2 columns. What each comparison key tells you key groups by use when "lineno" file and line finding the allocating line "traceback" the full call path the line is shared by many callers "filename" module narrowing to a library first Snapshot.filter_traces a file pattern excluding noise from the stdlib Start with lineno; switch to traceback when the top line is inside a helper everything calls.

3. Filter out the noise

Real services allocate constantly, and the diff shows all of it. filter_traces narrows the view to your code:

current = current.filter_traces((
    tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
    tracemalloc.Filter(False, tracemalloc.__file__),
    tracemalloc.Filter(True, "/app/*"),                # only our code
))

The inverse is also useful: filtering to a suspected library confirms or eliminates it in one run. And Snapshot.dump() / Snapshot.load() let you take snapshots in production and analyse them somewhere else, which is often the only practical option — the leak reproduces under real traffic and not in your load test.

For a long investigation, take snapshots on a timer and keep the last few:

async def snapshot_loop(interval: float = 300.0) -> None:
    while True:
        await asyncio.sleep(interval)
        snapshot = tracemalloc.take_snapshot()
        snapshot.dump(f"/tmp/snap-{int(time.time())}.dump")

Comparing snapshot N with snapshot N-1 shows what grew in the last interval, which is far more useful than comparing against a start-up baseline hours old.

Verify: after filtering, the top entries are lines you recognise.

4. Budget the overhead

The measurements, over 20,000 small allocations:

configuration per allocation
untraced 0.05 µs
tracemalloc.start(1) 2.20 µs
tracemalloc.start(25) 5.96 µs

A 44x to 113x multiplier — but only on allocation, which is a small part of most request handling. A service doing real I/O may see single-digit percentage latency growth; one building large object graphs per request will notice immediately.

The practical approach is to enable it deliberately and temporarily: an environment variable that starts tracing at boot, or an admin endpoint that starts it, takes a pair of snapshots and stops. PYTHONTRACEMALLOC=10 does the same from outside the code, which is useful when the leak appears only in production.

Verify: with tracing on, latency and loop lag stay within what your service can tolerate for the duration of the investigation.

What tracemalloc costs per allocation 3 bars comparing untraced with the others. What tracemalloc costs per allocation untraced 0.05 µs tracemalloc(1) 2.20 µs tracemalloc(25) 5.96 µs Measured over 20,000 allocations; the multiplier applies to allocation-heavy code only. Over a hundred times the cost per allocation: enable it deliberately, not by default.

5. Know what tracemalloc cannot see

It traces Python's allocator. Three categories of growth are invisible to it:

  • Native allocations. Memory allocated by C extensions through malloc — numpy arrays' data buffers, some database drivers, compression and crypto libraries — does not appear. The tell is RSS growing while tracemalloc.get_traced_memory() stays flat.
  • Fragmentation. Python may hold arenas it cannot return to the OS, so RSS stays high after the objects are freed. Traced memory drops; RSS does not.
  • Non-memory resources. File descriptors, sockets, tasks and threads leak without allocating much — a leaked AsyncClient costs 2 descriptors and a few kilobytes. Those need descriptor counting and task counting instead.

The quickest triage is to export both numbers and compare their shapes:

TRACED = Gauge("python_traced_bytes", "tracemalloc current")
RSS = Gauge("process_rss_bytes", "resident set size")

If they rise together, tracemalloc will find it. If RSS rises alone, reach for a native profiler or a resource counter.

Verify: traced memory and RSS are both exported, so the next investigation starts with the right tool.

What kind of leak is this? A decision on What is growing with 3 outcomes. What kind of leak is this? What is growing? RSS and traced memory a Python object leak tracemalloc finds it RSS but not traced memory native or fragmentation tracemalloc cannot see it neither, but fds or tasks a resource leak count descriptors and tasks Comparing RSS with tracemalloc’s own total is the fastest way to pick the right tool.

Verification

A leak investigation is on track when:

  • The service is warmed up before the baseline snapshot.
  • gc.collect() runs before each snapshot.
  • The top diff entry is large relative to the observed growth.
  • The traceback reaches your code, not only library internals.
  • Tracing is temporary, with its overhead measured and tolerated.
  • RSS and traced memory are compared, so the right tool is chosen.

Pitfalls & edge cases

  • No warm-up. One-time setup allocations dominate the diff and hide the leak.
  • Too few frames. start(1) puts every allocation at the library line that made it.
  • Leaving it on in production. A 113x allocation multiplier is not a monitoring strategy.
  • Chasing fragmentation. Traced memory flat with RSS high means the objects are gone; the arenas are not.
  • Forgetting async lifetimes. A pending task holds its whole frame, so its locals appear as a leak until it completes.
  • Snapshots taken in different processes. With several workers, compare snapshots from the same one.

Frequently Asked Questions

How do I find a memory leak in an asyncio service?

Start tracemalloc with a frame limit, warm the service up, take a baseline snapshot, run the suspect workload, take a second snapshot and compare them with compare_to(baseline, "lineno"). The leaking line appears at the top — in testing, +19,658 KiB against noise measured in fractions of a kilobyte.

How much does tracemalloc slow down a Python service?

A lot per allocation and little per request. Measured over 20,000 allocations: 0.05 µs untraced, 2.20 µs with one frame and 5.96 µs with 25 — a 113x multiplier on allocation itself. Enable it temporarily for an investigation rather than leaving it on.

Why does tracemalloc show no growth when RSS keeps rising?

Because the memory is not being allocated by Python's allocator. Native allocations from C extensions are invisible to tracemalloc, and freed memory can remain in arenas the interpreter has not returned to the OS. Export both numbers so you can tell which case you are in.

Should I compare snapshots to a start-up baseline?

Not for a long-running investigation. Take snapshots periodically and compare each with the previous one, so the diff shows what grew in that interval rather than everything allocated since boot — including all the one-time setup you do not care about.

Can I use tracemalloc in production?

Yes, deliberately and temporarily: start it via PYTHONTRACEMALLOC or an admin endpoint, take a pair of snapshots, dump them to disk and stop. Analyse the dumps elsewhere. Leaving it permanently enabled adds significant cost to every allocation your service makes.