Skip to content

Tracking Task Growth in Long-Running Services

A task that never completes is the async equivalent of a thread that never exits, except it costs almost nothing individually — 1,936 bytes, measured — so thousands can accumulate before anything looks wrong. What makes them expensive is what they keep alive: a suspended coroutine holds its entire frame, which means the request object, the buffers, the database connection it was using and every local variable in scope. Ten thousand pending tasks held 18.5 MiB of traced memory in the measurement below, and that is before counting the objects their frames reference. One gauge makes this visible; one Counter makes it diagnosable in a minute.

Prerequisites

What pending tasks cost 2 bars comparing 1,000 pending tasks with the others. What pending tasks cost 1,000 pending tasks 1.9 MiB 10,000 pending tasks 18.5 MiB Measured with tracemalloc: 1,936 bytes per task, plus whatever each coroutine frame holds. The task object is small; what it keeps alive — buffers, connections, request state — is not.

1. Export the count, and watch its shape

async def sample_tasks(interval: float = 15.0) -> None:
    while True:
        TASKS.set(len(asyncio.all_tasks()))
        await asyncio.sleep(interval)

The number itself is not meaningful — a busy service has hundreds of live tasks at any instant — but its shape is. A healthy service's task count oscillates around a baseline that tracks concurrency. A leaking one climbs monotonically, and the slope tells you how long you have: at ten leaked tasks per second and 2 KB each, a 512 MiB container survives roughly seven hours.

In the verified run, the baseline was 1 task (the main coroutine), 10,000 leaked tasks took it to 10,001, and cancelling them returned it to 1. That return to baseline is the property to alert on: compare the count now with the count an hour ago, rather than against a fixed threshold.

Verify: the gauge returns to its baseline when traffic stops.

2. Name every task

Without names, all_tasks() returns objects called Task-4731 and the histogram tells you nothing. With names, one line finds the leak:

task = asyncio.create_task(consume(queue), name=f"consumer:{queue_name}")

by_prefix = collections.Counter(
    task.get_name().rsplit(":", 1)[0] for task in asyncio.all_tasks())

Verified output from the leaking run: {'leak': 10000, 'Task': 1}. The spawner is named directly — no code reading required.

TaskGroup.create_task() takes the same name argument, and so does loop.create_task(). Making naming a convention costs nothing and is the difference between a five-minute diagnosis and an afternoon.

Verify: a Counter over live task names is readable, with no large Task-N group.

From a rising graph to a fixed leak 5 ordered steps. From a rising graph to a fixed leak export len(all_tasks()) one gauge, sampled group by name prefix a Counter over get_name() inspect one instance coro and current frame find the spawner names make this trivial give it an owner a registry, or a TaskGroup Naming every task is what makes the histogram readable; unnamed ones are all called Task-N.

3. Inspect a stuck task

Three calls identify what a leaked task is doing:

sample = next(t for t in asyncio.all_tasks() if t.get_name().startswith("leak"))
sample.get_name()                    # 'leak-7709'
sample.get_coro().__qualname__       # 'forever'
frame = sample.get_stack()[-1]
f"{frame.f_code.co_name}:{frame.f_lineno}"   # 'forever:4'

That is the whole diagnosis: the coroutine function, and the exact line it is suspended on. get_stack() returns the frames of a suspended task, so the last one is where it is waiting — an await on an event, a queue, a socket read.

For a broader view, task.print_stack() writes the traceback, and in Python 3.14 asyncio.print_call_graph() renders the await chain including which task is waiting on which. A dump endpoint that prints the top few groups on request is a useful thing to have before you need it:

async def tasks_dump(request):
    rows = [{"name": t.get_name(), "coro": t.get_coro().__qualname__,
             "frame": f"{t.get_stack()[-1].f_code.co_name}" if t.get_stack() else None}
            for t in list(asyncio.all_tasks())[:200]]
    return JSONResponse(rows)

Verify: the dump names a coroutine and a line you recognise.

Turning a task count into a diagnosis A grid of 5 rows by 2 columns. Turning a task count into a diagnosis call returns tells you task.get_name() the name you gave it which spawner made it task.get_coro().__qualname__ the coroutine function what it is running task.get_stack()[-1] the current frame the line it is suspended on task.done() / cancelled() its state stuck, or finished and unobserved Counter over get_name() a histogram the leaking spawner, immediately Verified: a leaked task reported name leak-7709, coro forever, suspended at forever:4.

4. Understand the three ways tasks leak

Waiting on something that will never happen. A task blocked on Event.wait(), Queue.get() or a lock nobody releases. Nothing wakes it, and nothing cancels it, because whoever created it forgot about it. This is the common case and the one the count catches.

Awaiting I/O with no timeout. A request to a hung upstream holds a task, a connection and a request's worth of memory indefinitely. The per-attempt timeout is what bounds it; without one, an upstream that stops responding converts your entire connection pool into leaked tasks.

Finished but unobserved. These do not grow the count — the task completes — but its exception is never retrieved, so you get Task exception was never retrieved at some arbitrary later moment, or nothing at all if the object is still referenced. It hides the error that would have explained the other two categories.

def _log_failure(task: asyncio.Task) -> None:
    if not task.cancelled() and (exc := task.exception()) is not None:
        logger.error("task failed", exc_info=exc, extra={"task": task.get_name()})

Verify: for each long-lived task in your service, you can name what cancels it.

Why is this task still alive? A decision on What is it doing with 3 outcomes. Why is this task still alive? What is it doing? waiting on an event or queue nothing will wake it shut it down explicitly awaiting I/O with no timeout a hung dependency add a timeout done, but nobody awaited it an exception is hidden log it in a done callback The third case does not grow the task count, but it hides the error that explains the other two.

5. Give every task an owner

The fix for all three is structural: nothing creates a task without something that will stop it.

_background: set[asyncio.Task] = set()


def spawn(coro, name: str) -> asyncio.Task:
    task = asyncio.create_task(coro, name=name)
    _background.add(task)                              # a strong reference
    task.add_done_callback(_background.discard)        # released when it finishes
    task.add_done_callback(_log_failure)
    return task

Verified: 50 tracked tasks put the registry at 50 immediately after spawning and 0 once they completed — the registry is self-cleaning, and its size is itself a metric worth exporting.

A TaskGroup is the stronger version where the lifetime fits a block, because nothing can outlive the async with. For work that must outlive a request, the registry above plus cancellation in the lifespan teardown is the equivalent:

    finally:
        for task in list(_background):
            task.cancel()
        await asyncio.gather(*_background, return_exceptions=True)

Note one measured subtlety: 100 fire-and-forget short tasks left the count at its baseline, because they completed. Task count only catches tasks that stay alive — which is why the registry, with its done-callback logging, is what catches the rest.

Verify: after shutdown, asyncio.all_tasks() contains only the task running the shutdown.

Verification

Task growth is under control when:

  • The live task count is exported and compared against itself over time.
  • Every task has a name that identifies its spawner.
  • A dump endpoint or log can list coroutines and suspension points.
  • Every long-lived task has an owner that will cancel it.
  • Failures are logged in a done callback.
  • Shutdown leaves no tasks behind.

Pitfalls & edge cases

  • Alerting on an absolute count. Healthy services have hundreds; the trend is the signal.
  • Unnamed tasks. Task-4731 tells you nothing when there are 10,000 of them.
  • create_task with no reference. The loop holds only a weak one; keep a strong reference until completion.
  • Counting inside a worker with several processes. Each has its own loop; the gauge is per process.
  • get_stack() on a running task. It returns the current frame only while suspended; for a running task it is the frame at the moment of the call.
  • Cancelling without awaiting. The task has not finished unwinding when the process exits, so its cleanup does not run.

Frequently Asked Questions

How do I detect leaked asyncio tasks?

Export len(asyncio.all_tasks()) as a gauge and watch its trend. A healthy service oscillates around a baseline; a leak climbs monotonically. In testing, 10,000 leaked tasks took the count from 1 to 10,001 and cancelling them returned it to 1.

How much memory does a pending asyncio task use?

About 1,936 bytes for the task itself — 18.5 MiB for 10,000, measured with tracemalloc. The real cost is what the suspended coroutine frame keeps alive: request objects, buffers and connections that cannot be collected while the task exists.

How do I find out what a stuck task is waiting on?

Use task.get_coro().qualname for the coroutine and task.get_stack()[-1] for the frame it is suspended in. Verified on a leaked task, that gave coro forever suspended at forever:4 — the exact await. task.print_stack() prints the same as a traceback.

Why are my tasks unnamed in all_tasks()?

Because create_task was called without name=. Pass a descriptive name at every creation site — including TaskGroup.create_task — and a Counter over task names then identifies the leaking spawner immediately.

What is the right way to own a background task?

Keep it in a module-level set, discard it in a done callback, log its exception in another, and cancel the whole set during shutdown. Verified, a 50-task registry returned to zero as the tasks completed. Where the lifetime fits a block, a TaskGroup is stronger still.