Skip to content

Reading Await Chains with task.get_stack

"Where is this task stuck?" is the most common question when an asyncio service misbehaves, and the obvious API — task.get_stack() — answers it badly for the most common case. For a task suspended in an await, it returns only the task's own coroutine frame: in a test, a task parked in top → middle → leaf → sleep reported just ['top']. For a task that failed, it does much better, returning the traceback frames down to the raise. Python 3.14 added asyncio.capture_call_graph(), which returns the whole await chain as data — enough to group thousands of tasks by where they are waiting. In a test with 30 request tasks, grouping by innermost user frame showed 20 waiting in leaf and 10 in pool_acquire in one line of output. This guide shows what each API returns and how to turn it into a summary you can read during an incident.

Prerequisites

1. Know what get_stack returns in each state

task.get_stack(limit=None) returns a list of frame objects, and its meaning depends on the task's state:

Task state get_stack() returns
pending, suspended in an await the task's own coroutine frame only
done with an exception the traceback frames, outermost first
done with a result, or cancelled an empty list 
async def inner():
    raise ValueError("bad row")


async def outer():
    await inner()


async def main() -> None:
    t = asyncio.create_task(outer())
    await asyncio.sleep(0)
    print([f.f_code.co_name for f in t.get_stack()])     # ['outer', 'inner']
    t.print_stack()                                       # formatted, like a traceback

For the failed task, print_stack() printed a proper traceback ending in ValueError: bad row. For a pending task, the single frame is the outermost coroutine — which tells you which task is stuck, but not where. The limit argument trims the list from the newest end for tracebacks and from the oldest for stacks, which rarely matters when there is only one frame.

Verify: call get_stack() on a pending task several awaits deep; you get one frame, the top-level coroutine.

What get_stack shows for each task state A grid of 3 rows by 3 columns. What get_stack shows for each task state task state get_stack() capture_call_graph() on 3.14 pending in an await outer coroutine only the full await chain done with an exception traceback frames not needed done or cancelled empty list None For stuck tasks, get_stack identifies the task; the call graph locates the await.

2. Walk the await chain yourself before 3.14

Each suspended coroutine records what it is awaiting in cr_await. Following that attribute from the task's coroutine downward reaches the innermost awaitable, usually a future, where the task is parked:

def await_chain(task: asyncio.Task) -> list[tuple[str, int, str]]:
    out = []
    coro = task.get_coro()
    while coro is not None:
        frame = getattr(coro, "cr_frame", None) or getattr(coro, "ag_frame", None)
        if frame is not None:
            out.append((frame.f_code.co_filename, frame.f_lineno, frame.f_code.co_name))
        coro = getattr(coro, "cr_await", None) or getattr(coro, "ag_await", None)
    return out

On the top → middle → leaf task this returned ['top', 'middle', 'leaf', 'sleep'] — the full chain. It stops at the first object that is not a coroutine or async generator: a future, a custom awaitable, or a __await__ implemented in C. That is usually exactly the boundary you want, since "awaiting a future created by the connection pool" is where the investigation goes next.

cr_await and cr_frame are documented attributes of coroutine objects, so this is not relying on private internals, but it only sees coroutines: a task waiting inside a __await__ generator of a third-party awaitable shows the chain down to that object.

Verify: the last entry of each chain names the coroutine that called the blocking await — sleep, acquire, read, get.

3. Use capture_call_graph on 3.14

Python 3.14 makes the walk official. asyncio.capture_call_graph(task) returns a FutureCallGraph with call_stack — a list of FrameCallGraphEntry objects whose .frame is a real frame — and awaited_by, the tasks waiting on this one. asyncio.print_call_graph(task) formats the same data:

async def leaf():
    await asyncio.sleep(10)


async def middle():
    await leaf()


async def top():
    await middle()


async def main() -> None:
    t = asyncio.create_task(top(), name="handler-42")
    await asyncio.sleep(0.01)
    g = asyncio.capture_call_graph(t)
    print([e.frame.f_code.co_name for e in g.call_stack])     # ['sleep', 'leaf', 'middle', 'top']
    asyncio.print_call_graph(t)

The call stack is innermost first. awaited_by matters in structured code: a child task in a TaskGroup lists the parent blocked in TaskGroup.__aexit__, so you can reconstruct the tree of who is waiting on whom without guessing from names. Calling either function with no argument captures the current task, which is useful for logging "where am I" from deep inside a coroutine.

Verify: for a TaskGroup child, g.awaited_by is non-empty and names the parent task.

The await chain that capture_call_graph returns A sequence of 4 messages between 5 participants. The await chain that capture_call_graph returns Task handler-42 top middle leaf sleep future runs coroutine await middle() await leaf() await sleep(10): parked here get_stack sees only the first box; the call graph and a cr_await walk see the whole chain.

4. Summarise thousands of tasks by where they wait

In a busy service, printing every task is useless. Group them by innermost user frame — the first frame in your code, skipping asyncio and library internals:

import collections
import os

STDLIB = os.path.dirname(asyncio.__file__)


def where_waiting(task: asyncio.Task) -> str:
    graph = asyncio.capture_call_graph(task)
    if graph is None:
        return "<done>"
    for entry in graph.call_stack:                     # innermost first
        code = entry.frame.f_code
        if not code.co_filename.startswith(STDLIB) and "site-packages" not in code.co_filename:
            return f"{code.co_name} ({os.path.basename(code.co_filename)}:{entry.frame.f_lineno})"
    return graph.call_stack[0].frame.f_code.co_name


def summarise_tasks(top: int = 10) -> list[tuple[str, int]]:
    return collections.Counter(where_waiting(t) for t in asyncio.all_tasks()).most_common(top)

With 30 request tasks, 10 of which were stuck acquiring from a pool, the summary was [('leaf', 20), ('pool_acquire', 10)] — the pool problem visible at a glance. During an incident, a summary that says "4,000 tasks waiting in acquire at pool.py:88" is worth more than any individual stack. Expose it on an admin endpoint next to the counters from tracking task growth in long-running services.

Verify: call summarise_tasks() under load; the top entries match where you expect requests to spend their time.

30 request tasks grouped by where they wait 2 horizontal bars comparing leaf (normal work) with the others. 30 request tasks grouped by where they wait leaf (normal work) 20 tasks pool_acquire (stuck) 10 tasks Grouping by innermost frame turns a wall of stacks into one actionable line.

5. Capture creation sites for orphaned tasks

An await chain tells you where a task is now; sometimes the question is who created it. Debug mode records a creation traceback on every task (task._source_traceback, used in asyncio's own error messages), at a large cost — task-heavy code ran about 37× slower in debug mode in a microbenchmark. A cheaper production option records just the caller's location in a custom task factory:

import sys


def factory(loop, coro, **kwargs):
    task = asyncio.Task(coro, loop=loop, **kwargs)
    caller = sys._getframe(1)
    while caller is not None and caller.f_code.co_filename.startswith(STDLIB):
        caller = caller.f_back                   # skip asyncio's own create_task frames
    if caller is not None:
        task.created_at = f"{caller.f_code.co_filename}:{caller.f_lineno}"
    return task


asyncio.get_running_loop().set_task_factory(factory)

Combining "created at" with "waiting at" answers both halves of a leak investigation. The full task-factory approach is in logging task lifecycles with a custom task factory.

Verify: every task in a dump shows a created_at pointing into your code.

Verification

You can locate stuck tasks when:

  • You know which API returns what: one frame for pending tasks from get_stack, a full chain from the call graph or a cr_await walk.
  • A summary by innermost user frame is available on demand.
  • Failed tasks are inspected with print_stack() or their exception's traceback.
  • Creation sites are recorded cheaply for long-lived services.

Diagnostic Hook: export the top five "waiting at" locations and their counts as a periodic log line or metric with the location as a label (bounded to the top few to avoid cardinality blow-ups). A location whose count climbs while throughput is flat is the bottleneck — almost always a pool acquire, a lock, or a downstream call without a timeout.

Pitfalls & edge cases

  • Calling capture_call_graph from another thread. Frames are only safe to inspect on the loop thread; run summaries on the loop.
  • Holding frame references. Storing frames from get_stack keeps their locals alive; extract strings and drop the frames.
  • Expecting a chain through third-party awaitables. The walk stops at objects that are not coroutines.
  • Heavy summaries on huge task sets. Walking 100,000 tasks takes time on the loop; sample or cap the count.

Frequently Asked Questions

Why does asyncio task.get_stack return only one frame?

For a task that is suspended, get_stack returns the task's own coroutine frame rather than the chain of coroutines it is awaiting. Follow cr_await from coroutine to coroutine, or use asyncio.capture_call_graph on Python 3.14, to see where it is actually waiting.

How do I print the stack of an asyncio task?

task.print_stack() prints a traceback-style view. For a failed task it shows the frames down to the raise; for a pending task only the top frame. On Python 3.14, asyncio.print_call_graph(task) prints the full await chain and the tasks awaiting it.

What is asyncio.capture_call_graph?

A Python 3.14 function that returns a FutureCallGraph for a task: its call stack as frame entries, innermost first, and the list of tasks awaiting it. It is the data behind print_call_graph and the asyncio ps and pstree tools.

How do I find where most of my tasks are stuck?

Group asyncio.all_tasks() by the innermost frame from your own code in each task's await chain and count them. The largest group is usually a pool acquire, a lock or a downstream call without a timeout.