Skip to content

Migrating from gather to TaskGroup

asyncio.gather has one behaviour that surprises nearly everyone who relies on it: when a child raises, gather re-raises that exception immediately and leaves the other children running. Measured on three tasks where the first fails after 50 ms, gather raised at 0.05 s — and the two siblings went on to finish 0.4 s later, writing their results into a void. The same three tasks in a TaskGroup produced a cancelled sibling list and no stray work. That difference, plus reporting every failure rather than the first, is the reason to migrate. The cost is that your error handling changes shape, because a TaskGroup raises an ExceptionGroup.

Prerequisites

The same three tasks, one of which fails 2 columns contrasting asyncio.gather, asyncio.TaskGroup. The same three tasks, one of which fails asyncio.gather raises the first exception siblings keep running later failures are lost results list is ordered nothing waits for the strays asyncio.TaskGroup raises an ExceptionGroup siblings are cancelled every failure is reported results read from tasks the block waits for all Measured: gather returned at 0.05 s and the two siblings still finished 0.4 s later.

1. See the difference on one example

Three tasks, the first failing quickly:

# gather
try:
    await asyncio.gather(fail_fast(), slow_a(), slow_b())
except ValueError as exc:
    ...                                                # raised at 0.05 s
# slow_a and slow_b are still running here, and will finish at 0.4 s

# TaskGroup
try:
    async with asyncio.TaskGroup() as tg:
        tg.create_task(fail_fast())
        tg.create_task(slow_a())
        tg.create_task(slow_b())
except* ValueError as eg:
    ...                                                # raised at 0.05 s
# slow_a and slow_b were cancelled; nothing is still running

Measured outcomes, recorded by the tasks themselves:

at the exception 0.5 s later
gather finished: none, cancelled: none finished: slow-a, slow-b
TaskGroup cancelled: slow-b, slow-a finished: none

And with two failures, gather reported ValueError('x failed') alone while the TaskGroup reported ['x failed', 'y failed']. Whether losing the second failure matters depends on your system; it usually does, because correlated failures are what identify a root cause.

Verify: add a completion log to each child and watch whether siblings still log after the error is handled.

Where the work goes after a gather failure 4 lanes over time. Where the work goes after a gather failure failing task raises caller waits handles the error moved on sibling 1 still running, unobserved sibling 2 still running, unobserved time → Their results go nowhere; their side effects happen anyway.

2. Translate the plain call

The mechanical translation keeps a handle on each task so you can read results afterwards:

async def fetch_all(urls):
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(fetch(url)) for url in urls]
    return [task.result() for task in tasks]           # only reached if all succeeded

The list comprehension runs after the async with exits, which is the point: the block does not exit until every task has finished, so task.result() is always safe there. Ordering is preserved because the list is built in submission order, matching gather's contract.

If the block raises, the return is never reached and the ExceptionGroup propagates — so callers that used to catch ValueError need except* ValueError, or except BaseExceptionGroup for a coarser catch. This is the part of the migration that touches code far from the change, and it is worth doing in one pass rather than leaving a mix.

Verify: the happy path returns results in submission order, and a failure raises a group rather than a bare exception.

3. Translate return_exceptions=True

gather(..., return_exceptions=True) returns exceptions as values instead of raising. A TaskGroup has no such switch — any exception reaching the group cancels the siblings — so the capture moves inside each task:

async def capture(coro):
    try:
        return await coro
    except Exception as exc:                           # not BaseException: cancellation must pass
        return exc


async def gather_all(coros):
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(capture(c)) for c in coros]
    return [task.result() for task in tasks]

Verified equivalent: both forms returned ['a', ValueError, 'c'] for the same three coroutines. The except Exception rather than except BaseException matters — catching CancelledError here would make the group's own cancellation impossible, and produce a hang on shutdown.

Honestly assessed, this is more code than return_exceptions=True for the same result, and if you genuinely want every task to run to completion regardless of the others, gather is still the better tool. The migration is worth it where a failure should stop the rest.

Verify: a failing task yields an exception object in the results list and the other tasks still complete.

Translating each gather shape A grid of 4 rows by 2 columns. Translating each gather shape gather form TaskGroup equivalent note gather(*coros) create_task in the block read task.result() after return_exceptions=True capture inside each task the group must not see them gather of background work a long-lived group or a supervisor task partial results wanted wrap each task body failures become values The only form with no direct translation is one that wants siblings to outlive a failure.

4. Handle the errors in the new shape

Every except around a migrated call needs revisiting. Three rules cover it:

try:
    async with asyncio.TaskGroup() as tg:
        ...
except* ConnectionError as eg:                         # handles matching leaves
    logger.warning("upstreams failed: %d", len(eg.exceptions))
except* ValueError as eg:                              # a second clause, also matching
    logger.error("bad data", exc_info=eg)

except* clauses are not mutually exclusive — a group containing both types runs both clauses, each seeing only its own leaves. Anything unmatched re-raises automatically as a new group. And logging needs exc_info, because str() on a group reports only "unhandled errors in a TaskGroup (N sub-exceptions)"; see logging ExceptionGroups with full tracebacks.

For a call site that genuinely only cares whether anything failed, except BaseExceptionGroup as eg is fine — just remember it will also catch a group carrying a CancelledError.

Verify: each except* clause receives only its own leaf types, and unmatched failures still propagate.

5. Know which gathers not to migrate

A TaskGroup is a scope: nothing inside it outlives the block. That makes three patterns a poor fit.

  • Fire-and-forget background work. A group cannot be left open across requests. Long-lived background tasks need a supervisor that owns them for the service's lifetime — a group in a dedicated task, as in Task Scheduling & Lifecycle.
  • Independent work that must all complete. A nightly job running twenty exports where one failure should not stop the others is exactly what return_exceptions=True is for. The capture wrapper achieves it, but adds nothing.
  • Racing for the first result. asyncio.wait(..., FIRST_COMPLETED) is still the right tool — as in hedging requests — because a group by definition waits for everyone.

Note also that tg.create_task() after the block has exited raises RuntimeError: TaskGroup ... is finished, which catches attempts to keep a group handle around for later use.

Verify: no migrated call site needs tasks to outlive the async with block.

Should this gather become a TaskGroup? A decision on What happens when one child fails with 3 outcomes. Should this gather become a TaskGroup? What happens when one child fails? the rest are pointless TaskGroup cancellation is the win every result is wanted either, with capture gather is fine here nothing awaits them neither: supervise a tracked background group A gather whose siblings should keep running is the one case to leave alone.

Verification

A migration is complete when:

  • No sibling outlives a failure: children are cancelled, verified by their own logs.
  • Every failure is reported, not just the first.
  • Results are read after the block, from task handles, in submission order.
  • Error handling uses except* or explicitly catches BaseExceptionGroup.
  • Logging uses exc_info, never string formatting of the group.
  • No group is expected to outlive its block.

Pitfalls & edge cases

  • except Exception left in place. It no longer matches, because the group is raised instead; the error escapes to the caller.
  • Catching BaseException in the capture wrapper. It swallows CancelledError and hangs the group's cancellation.
  • Reading task.result() inside the block. The task may not be done yet; read after the async with.
  • Assuming one failure means one exception. Even a single failing child raises an ExceptionGroup wrapping it.
  • Nested groups. A child that is itself a TaskGroup contributes a nested group; flatten recursively when logging.
  • Cancellation from outside. Cancelling the task that runs the group cancels every child — verified — which is usually what you want during shutdown.

Frequently Asked Questions

What is the difference between asyncio.gather and TaskGroup?

When a child fails, gather raises that exception and lets the other children keep running; a TaskGroup cancels the siblings, waits for them, and raises an ExceptionGroup containing every failure. Measured on three tasks, gather's two siblings still finished 0.4 s after the error was handled.

How do I replace gather(return_exceptions=True) with a TaskGroup?

Wrap each coroutine in a helper that catches Exception and returns it as a value, so no exception ever reaches the group. Then read results from the task handles after the block. Catch Exception rather than BaseException, or you will swallow cancellation.

Does a TaskGroup always raise an ExceptionGroup?

Yes — even for a single failing child. Handle it with except* clauses, which match by leaf type and re-raise anything unmatched, or catch BaseExceptionGroup when you only need to know that something failed.

Can I use create_task on a TaskGroup after the block exits?

No. It raises RuntimeError: TaskGroup ... is finished. A group is a scope: tasks are created inside the async with block and nothing survives it. Background work that must outlive a request needs a supervisor task that owns its own group.

Is gather deprecated in favour of TaskGroup?

No. gather remains appropriate when every task should run to completion regardless of the others, and asyncio.wait is still the tool for racing. TaskGroup is the better default for fail-fast concurrency, where continuing after one child fails wastes work.