Skip to content

Inspecting Pending Tasks with asyncio.all_tasks

The service is up, the loop is responsive, and yet a shutdown never completes — or memory climbs all afternoon, or one endpoint hangs while everything else is fine. In a threaded service you would take a stack dump and read it. The async equivalent is a task dump: asyncio.all_tasks() returns every task on the loop that has not finished, and each of those tasks can tell you its name, the coroutine it is running, the exact line that coroutine is suspended on, and the stack of frames beneath it. Ten lines of code turn that into the async equivalent of jstack, triggerable from a running container.

The hard part is not the API — it is producing output you can actually read. A service with 8,000 tasks dumps 8,000 lines that all look alike unless the tasks were named, and the interesting question ("which of these have been alive since before the incident, and where are they stuck?") needs grouping, not a raw listing. This guide builds the dump, makes it signal-triggerable, adds naming and age so it is readable, and walks through interpreting the three patterns that matter: a hang, a leak, and a stuck await.

Prerequisites

What a task can tell you when you ask it A stack of five task introspection accessors annotated with what they answer. What a task can tell you when you ask it get_name() who created it — only if you named it get_coro().__qualname__ which coroutine is running cr_frame.f_lineno the exact await it is parked on print_stack() the frames beneath, or the exception it died with done() / cancelled() whether it is finished, failed, or cancelling The middle two columns are what turn a dump into a diagnosis.

1. Take a basic dump

asyncio.all_tasks() must be called from inside the running loop; it returns the set of tasks belonging to that loop which are not yet done.

import asyncio


def dump_tasks() -> None:
    tasks = asyncio.all_tasks()
    print(f"--- {len(tasks)} live task(s) ---")
    for task in tasks:
        coro = task.get_coro()
        frame = getattr(coro, "cr_frame", None)
        where = f"{frame.f_code.co_filename}:{frame.f_lineno}" if frame else "<no frame>"
        print(f"{task.get_name():<24} {getattr(coro, '__qualname__', coro)!s:<34} {where}")

cr_frame is the coroutine's current frame — the line it is suspended on, which is almost always an await. That single column answers most questions immediately: a hundred tasks parked on queue.get() are healthy consumers; a hundred parked on pool.acquire() are a pool exhaustion incident. Verify: call it from a debug endpoint or a SIGUSR1 handler under normal load and confirm the counts and locations match what you expect the service to be doing.

2. Make every task identifiable before you need to

A dump full of Task-1471 is a dump you cannot act on. Names cost nothing and can be attached in two places: at creation, or centrally through a task factory that stamps every task with its request context.

import asyncio
import contextvars

request_id: contextvars.ContextVar[str] = contextvars.ContextVar("request_id", default="-")


def naming_task_factory(loop, coro, *, context=None, **kwargs):
    task = asyncio.Task(coro, loop=loop, context=context, **kwargs)
    task.set_name(f"{getattr(coro, '__qualname__', 'task')}#{request_id.get()}")
    return task


async def main() -> None:
    asyncio.get_running_loop().set_task_factory(naming_task_factory)
    async with asyncio.TaskGroup() as tg:
        tg.create_task(worker(), name="worker.main")   # explicit names still win

The factory runs on every task creation, including tasks created inside libraries you do not control, so it is the only way to guarantee full coverage. Keep it to attribute assignment — a factory that logs or allocates heavily taxes every create_task() in the process. Verify: dump again and confirm no task is left with a default Task-N name; any that remain were created before the factory was installed, usually during startup.

3. Trigger the dump from a running process

Signals are the standard trigger: they need no HTTP surface, no auth story, and work in any container. Install the handler on the loop so the dump runs as a loop callback, where all_tasks() is meaningful and thread-safe.

import asyncio
import signal


async def main() -> None:
    loop = asyncio.get_running_loop()
    loop.add_signal_handler(signal.SIGUSR1, dump_tasks)      # kill -USR1 <pid>
    await serve()

kubectl exec -it my-pod -- kill -USR1 1        # dump goes to the container's logs

Use loop.add_signal_handler() rather than signal.signal(): the latter runs the handler in a signal context that can interrupt a callback mid-update, and calling all_tasks() from there can observe a torn view of the loop's internals. Verify: send the signal and confirm the dump appears in the logs within one loop iteration and the service keeps serving traffic while it prints.

4. Add age and stack depth so the output ranks itself

Two derived fields make a long dump self-sorting. Age tells you which tasks predate the incident; the stack's deepest frame tells you what the task is actually waiting on rather than which handler created it.

import asyncio
import io
import time

CREATED: "dict[asyncio.Task, float]" = {}


def timing_task_factory(loop, coro, *, context=None, **kwargs):
    task = asyncio.Task(coro, loop=loop, context=context, **kwargs)
    task.set_name(getattr(coro, "__qualname__", "task"))
    CREATED[task] = loop.time()
    task.add_done_callback(CREATED.pop)          # no leak: entry dies with the task
    return task


def dump_tasks(limit: int = 40) -> None:
    loop = asyncio.get_running_loop()
    now = loop.time()
    tasks = sorted(asyncio.all_tasks(), key=lambda t: CREATED.get(t, now))
    print(f"--- {len(tasks)} live task(s); {min(limit, len(tasks))} oldest ---")
    for task in tasks[:limit]:
        age = now - CREATED.get(task, now)
        buf = io.StringIO()
        task.print_stack(limit=2, file=buf)      # deepest frames of the coroutine stack
        tail = buf.getvalue().strip().splitlines()[-1].strip() if buf.getvalue() else "?"
        print(f"[{age:7.1f}s] {task.get_name():<28} {tail}")

Task.print_stack() prints the coroutine's stack if it is suspended, or the traceback if it has already failed — the closest async analogue to a thread dump. Sorting by age puts the suspects first: in a hang, the oldest tasks are the ones holding everything up. Verify: the oldest entries should be your long-lived infrastructure tasks (the server, the probes); anything unexpected at the top of that list is the thing to investigate.

Reading a task dump A decision diamond about parked lines with three outcome cards. Reading a task dump What do the parked lines look like? one old task on join() completion bug a task_done() is missing thousands on one line task leak created per request, never awaited N waiters on acquire() resource contention compare N to the pool size The distribution of parked lines, not any single line, names the failure.

5. Read the three patterns

The same dump answers three different questions depending on what it looks like.

# (a) HANG — shutdown never completes
[ 913.4s] worker.consume#-        File "app/worker.py", line 61, in consume
                                    await self.queue.join()          <- waits forever
[ 913.4s] server.accept           File "app/net.py", line 22, in accept
                                    await server.serve_forever()

# (b) LEAK — task count climbs, memory follows
[ 402.7s] notify#req-91a2         File "app/notify.py", line 14, in notify
[ 402.5s] notify#req-91a3         File "app/notify.py", line 14, in notify
... 4,812 more tasks on the same line, one per request since 09:12

# (c) STUCK AWAIT — one endpoint hangs, the rest is fine
[  61.9s] handle#req-77c1         File "asyncpg/pool.py", line 741, in acquire
[  61.8s] handle#req-77c2         File "asyncpg/pool.py", line 741, in acquire
... 20 tasks parked in pool.acquire(), pool max_size=20

Pattern (a) — a single task on join() or gather() while nothing else progresses — is a completion-signal bug: something never called task_done(), or a producer never exited. Pattern (b) — thousands of tasks all parked on the same line, ages spread evenly — is a leak: work is created per request and never awaited or cancelled. Pattern (c) — a bounded cluster of tasks waiting on the same acquisition — is contention on a limited resource, and the number of waiters against the pool size tells you whether to raise the limit or reduce concurrency; see sizing async connection pools. Verify: match the count in the dump against the configured limit — 20 waiters on a 20-connection pool is not a mystery, it is arithmetic.

6. Keep a bounded, safe version in production

The dump is on-demand for a reason: it is O(tasks), it formats frames, and at 50k tasks the full version can occupy the loop long enough to look like an outage.

def dump_tasks_safe(limit: int = 50) -> None:
    tasks = asyncio.all_tasks()
    if len(tasks) > 20_000:                      # a dump this size IS the incident
        print(f"--- {len(tasks)} tasks; printing names only ---")
        counter: dict[str, int] = {}
        for t in tasks:
            key = t.get_name().split("#", 1)[0]
            counter[key] = counter.get(key, 0) + 1
        for name, count in sorted(counter.items(), key=lambda kv: -kv[1])[:limit]:
            print(f"{count:>7} x {name}")
        return
    dump_tasks(limit=limit)

Above a threshold, print a histogram by task name instead of individual frames: on a leak that is strictly more useful anyway, because the answer is "4,812 × notify" rather than 4,812 nearly identical lines. Verify: synthesise 50k idle tasks in staging, trigger the dump, and confirm loop lag stays under your alert threshold while it runs.

Full dump vs name histogram Two columns contrasting full frame dumps and name histograms for task inspection. Full dump vs name histogram full frame dump under ~20k tasks One line per task Shows the exact await Formats frames — O(tasks) Best for a hang name histogram above ~20k tasks One line per task name Shows the leak's shape No frame formatting Best for a leak Above the threshold the histogram is both cheaper and more readable.

Verification

  • Every task has a meaningful name. A dump under normal load shows no Task-N entries except those created before the factory was installed.
  • The dump is cheap enough. Triggering it under production-like load moves p99 loop lag by less than your warning threshold.
  • Counts reconcile. The number of live tasks matches expectations: infrastructure tasks plus roughly (in-flight requests × tasks per request). A number that does not reconcile is the finding.

Pitfalls and edge cases

  • Calling all_tasks() from another thread. It reads loop internals and only sees the loop it was given; from a worker thread you get the wrong loop or a RuntimeError. Trigger dumps through loop.call_soon_threadsafe() or a loop signal handler.
  • current_task() is excluded from your reading, not from the set. The dumping callback itself appears in all_tasks() when it runs as a task; ignore it rather than being confused by it.
  • Holding tasks in a dict without cleanup. The age map in step 4 must remove entries in a done-callback, or the map itself becomes the leak you are hunting.
  • print_stack() on a failed task prints the exception, not a stack. That is useful — it surfaces an exception nobody retrieved — but do not read it as "the task is running here".
  • Names are not unique. Two tasks may share a name; correlate with the request id embedded by the factory when you need identity.
  • Tasks in a TaskGroup still appear individually. A group's children are ordinary tasks; grouping in the output is your job, which is what naming with a shared prefix gives you.

Frequently Asked Questions

How do I dump asyncio tasks from a running production process?

Install a loop signal handler at startup — loop.add_signal_handler(signal.SIGUSR1, dump_tasks) — and send kill -USR1 to the process when you need it. The handler runs as a loop callback, so asyncio.all_tasks() sees a consistent view, and the output goes to your normal logs without exposing a debug endpoint.

Why do all my tasks show up as Task-1, Task-2 and so on?

Nothing named them. Pass name= to create_task for tasks you own, and install a task factory with loop.set_task_factory() so every task — including ones created inside libraries — is named from the coroutine's qualified name plus the request context. Do it at startup; tasks created earlier keep their default names.

Is asyncio.all_tasks() safe to call in production?

Yes, on demand. It builds a set of every live task, so it is O(number of tasks): negligible at hundreds of tasks, milliseconds at tens of thousands. Keep it out of per-request paths and short timers, sample the count every ten seconds at most, and switch to a name histogram instead of full frames above roughly 20,000 tasks.

How can I tell a task leak from normal load?

Look at the age distribution and the parked line. Under normal load, task ages are short and varied and the count returns to baseline after traffic drops. A leak shows a growing population of tasks that share one source line, with ages spread evenly back to when the leak started, and a count that never returns to baseline.