Skip to content

Cancelling a Task and Waiting for It to Finish

A background consumer is stopped with task.cancel(), the next line closes the database pool, and the logs fill with InterfaceError: pool is closed from the consumer's cleanup code. Or a reload handler cancels the old worker and starts a new one, and for a few hundred milliseconds both are writing to the same file. The root cause is the same: task.cancel() does not stop a task. It schedules a CancelledError to be thrown into the coroutine at its next suspension point and returns immediately. The task then runs its except and finally blocks — which may await, flush buffers and release connections — and only afterwards is it done. Code that treats cancel() as synchronous races that cleanup. This guide builds a correct cancel-and-wait, bounds it for tasks that never finish, handles groups, and explains the cancelling() counter that makes timeouts and task groups behave.

Prerequisites

cancel() returns before the task is done 3 lanes over time. cancel() returns before the task is done racing caller cancel() close pool task running cleanup awaits pool done correct caller cancel() await task close pool time → The window between cancel() and done is where cleanup runs — never close what it needs.

1. Request cancellation, then await the task

The minimal correct sequence has two parts: request, then wait. Awaiting a cancelled task re-raises its CancelledError in the awaiting coroutine, so the wait must catch it — but only the one that came from the task, not a cancellation aimed at the caller.

import asyncio


async def consumer() -> None:
    try:
        await asyncio.sleep(3600)                 # stand-in for "await queue.get()"
    except asyncio.CancelledError:
        await asyncio.sleep(0.2)                  # flush buffers, ack in-flight work
        raise                                     # always re-raise


async def main() -> None:
    task = asyncio.create_task(consumer())
    await asyncio.sleep(0)                        # let it start

    task.cancel()
    print("done right after cancel():", task.done())      # False

    try:
        await task
    except asyncio.CancelledError:
        if asyncio.current_task().cancelling():   # *we* were cancelled too: propagate
            raise
    print("done after await:", task.done(), "cancelled:", task.cancelled())  # True True


asyncio.run(main())

The cancelling() check distinguishes the two sources of CancelledError that can surface at await task: the task's own cancellation, which you requested and want to absorb, and a cancellation of the current coroutine that happened to arrive while it waited, which must propagate or shutdown hangs.

Verify: the first print shows False and the second shows True True, with about 200 ms between them — that interval is the cleanup that code after a bare cancel() would have raced.

2. Bound the wait for tasks that do not stop

Awaiting a cancelled task can hang forever. A coroutine that catches CancelledError and continues its loop, a cleanup block that awaits an unresponsive network call, or a synchronous call that never yields all keep the task alive. Use asyncio.wait() with a timeout: unlike asyncio.wait_for(), it does not cancel the task again on timeout and does not raise, so it reports the situation instead of compounding it.

import asyncio
import logging

log = logging.getLogger("shutdown")


async def cancel_and_wait(task: asyncio.Task, *, grace: float = 5.0,
                          msg: str | None = None) -> bool:
    """Cancel `task` and wait up to `grace` seconds. True if it finished."""
    if task.done():
        return True
    task.cancel(msg)
    done, _ = await asyncio.wait({task}, timeout=grace)
    if not done:
        log.error("task %s ignored cancellation for %.1fs", task.get_name(), grace)
        return False
    if not task.cancelled() and task.exception() is not None:
        log.warning("task %s raised while cancelling: %r", task.get_name(), task.exception())
    return True

asyncio.wait() never raises the task's exception, so there is nothing to catch, and a cancellation of the caller still propagates out of the await asyncio.wait(...) normally. Retrieving task.exception() also marks it as observed, which prevents the "exception was never retrieved" warning at garbage collection.

Verify: against the well-behaved consumer, cancel_and_wait returns True after the cleanup delay. Against a coroutine that swallows CancelledError, it returns False after exactly grace seconds and logs the task name.

After cancel(), what state is the task in? A decision on What did asyncio.wait report with 3 outcomes. After cancel(), what state is the task in? What did asyncio.wait report? in done, task.cancelled() stopped cleanly release its resources in done, has exception cleanup failed log and release anyway in pending after grace ignored the cancel log name and stack Only the third outcome is a bug in the task; the wait must never hide it.

3. Cancel a group and collect every outcome

Shutdown usually stops many tasks at once. Cancel them all first, then wait for all of them together — cancelling and awaiting one by one multiplies the grace period by the number of tasks.

import asyncio


async def cancel_all(tasks: set[asyncio.Task], grace: float = 5.0) -> list[asyncio.Task]:
    """Cancel every task, wait once for the whole set, return the stragglers."""
    pending = {t for t in tasks if not t.done()}
    for t in pending:
        t.cancel("shutdown")
    if not pending:
        return []
    done, still_running = await asyncio.wait(pending, timeout=grace)
    for t in done:
        if not t.cancelled() and t.exception() is not None:
            print(f"{t.get_name()} failed during shutdown: {t.exception()!r}")
    return sorted(still_running, key=lambda t: t.get_name())

If the tasks belong to an asyncio.TaskGroup, you do not need this at all: cancelling the task that owns the group, or raising inside it, cancels every child and the async with block waits for all of them before exiting. That built-in wait is one of the main reasons to prefer structured concurrency with TaskGroup over loose tasks.

Verify: cancelling three tasks with 200 ms cleanups takes about 200 ms in total, not 600 ms, and the return value lists only tasks that genuinely ignored cancellation.

4. Understand cancelling() and uncancel()

Since Python 3.11 every task keeps a count of pending cancellation requests. cancel() increments it; uncancel() decrements it. asyncio.timeout() and TaskGroup rely on this counter to tell their own cancellation apart from an external one: when a timeout fires, it cancels the task, catches the resulting CancelledError, and calls uncancel() before converting it to TimeoutError. The check in step 1 relies on the same counter. A coroutine that catches CancelledError and carries on without calling uncancel() leaves the count stuck above zero, and every later decision based on it is wrong.

import asyncio


async def child() -> None:
    await asyncio.sleep(10)


async def swallow_once(fix: bool) -> None:
    try:
        await asyncio.sleep(10)
    except asyncio.CancelledError:
        if fix:
            asyncio.current_task().uncancel()         # suppression acknowledged
    print("cancelling() after the handler:", asyncio.current_task().cancelling())

    helper = asyncio.create_task(child())
    await asyncio.sleep(0)
    helper.cancel()
    try:
        await helper
    except asyncio.CancelledError:
        if asyncio.current_task().cancelling():      # the step 1 check
            print("  misread: treats the helper's cancel as its own, stops")
            raise
    print("  helper stopped, carrying on")


async def main() -> None:
    for fix in (False, True):
        task = asyncio.create_task(swallow_once(fix))
        await asyncio.sleep(0)
        task.cancel()
        await asyncio.wait({task}, timeout=1)


asyncio.run(main())

In the first run the swallowed cancellation leaves cancelling() at 1, so when the task later cancels a helper of its own, the step 1 check concludes that the task itself is being cancelled and stops it. Current releases of asyncio.timeout() record the count when the block is entered, so a stale count no longer breaks timeouts themselves — but any code that asks cancelling() a yes-or-no question, including the step 1 pattern and your own shutdown logic, still gets the wrong answer. If a coroutine deliberately suppresses a cancellation — for example a worker that treats "cancel current job" as a request to skip to the next one — it must call asyncio.current_task().uncancel() in the handler.

Verify: the first run prints cancelling() after the handler: 1 followed by the misread; the second run, with uncancel(), prints 0 and carries on.

5. Name tasks and pass a cancel message

When cancel_and_wait reports a straggler at 3 a.m., "Task-1873 ignored cancellation" is not actionable. Name every long-lived task at creation, and pass a message to cancel() so the task's own handler — and anything that logs the CancelledError — can say why it was stopped.

import asyncio
import logging

log = logging.getLogger("worker")


async def worker(queue: asyncio.Queue) -> None:
    try:
        while True:
            item = await queue.get()
            try:
                await asyncio.sleep(0.01)             # process(item)
            finally:
                queue.task_done()
    except asyncio.CancelledError as exc:
        reason = exc.args[0] if exc.args else "unspecified"
        log.info("%s stopping: %s", asyncio.current_task().get_name(), reason)
        raise


async def main() -> None:
    queue: asyncio.Queue[int] = asyncio.Queue()
    workers = {asyncio.create_task(worker(queue), name=f"worker-{i}") for i in range(4)}
    await asyncio.sleep(0.1)
    for t in workers:
        t.cancel("config reload")
    await asyncio.wait(workers, timeout=5)


logging.basicConfig(level=logging.INFO)
asyncio.run(main())

Verify: each worker logs worker-N stopping: config reload. In a hung shutdown, inspecting pending tasks with asyncio.all_tasks now lists meaningful names and the frame each straggler is stuck in.

Stopping workers in the right order 5 ordered steps. Stopping workers in the right order stop accepting new work close the listener or queue cancel with a reason task.cancel("shutdown") wait once, bounded asyncio.wait(tasks, timeout=grace) report stragglers names and frames, not Task-1873 close shared resources pools and clients last Resources close last because cleanup code is still using them.

Verification

Cancellation is handled correctly when:

  • Nothing runs after its dependencies are gone: resources a task uses are closed only after cancel_and_wait or cancel_all returns, and no "pool closed" or "transport closed" errors appear during shutdown.
  • Waits are bounded: every cancel-and-wait has a grace period, and stragglers are logged by name instead of hanging the process.
  • Group shutdown is parallel: stopping N tasks takes roughly one cleanup interval, not N of them.
  • Caller cancellation propagates: cancelling the coroutine that is doing the waiting still stops it promptly.
  • The cancellation counter stays truthful: code that intentionally swallows a cancellation calls uncancel(), so cancelling() reads zero afterwards and later timeouts and checks behave.

Pitfalls & edge cases

  • asyncio.run() hangs on stubborn tasks. At exit, asyncio.run() cancels remaining tasks and waits for them without a timeout. A task that swallows CancelledError in a loop keeps the process alive forever; stop such tasks yourself with a bounded wait before main() returns.
  • Using wait_for as the grace timer. asyncio.wait_for(task, 5) cancels the task again on timeout and then waits for that cancellation to complete, so it can still block. asyncio.wait() reports and returns.
  • Cancelling a task blocked in synchronous code. Cancellation is delivered at the next await. A task stuck in time.sleep() or a blocking socket read will not see it until the call returns; move such calls into a thread with asyncio.to_thread, accepting that the thread itself cannot be cancelled.
  • Shielded work outlives the cancel. asyncio.shield() lets the inner operation continue after the outer task is cancelled, so waiting for the outer task does not wait for the shielded work. Track the inner task separately if shutdown must include it.
  • Double cancellation during cleanup. A second cancel() while the handler is awaiting cleanup interrupts the cleanup itself. Signal handlers that cancel on every SIGTERM should cancel once and ignore repeats.

Frequently Asked Questions

Does task.cancel() stop an asyncio task immediately?

No. task.cancel() only requests cancellation and returns immediately. The CancelledError is thrown into the coroutine at its next await, after which its except and finally blocks run, possibly awaiting cleanup. The task is finished only when that completes, so await the task or wait on it before releasing resources it uses.

How do I wait for a cancelled task with a timeout?

Cancel it, then call asyncio.wait({task}, timeout=grace). asyncio.wait returns the done and pending sets without raising and without cancelling the task again, so you can log tasks that ignored the cancellation. Avoid asyncio.wait_for for this, because on timeout it cancels the task again and waits for that to finish.

What do Task.cancelling() and Task.uncancel() do?

Since Python 3.11 each task counts pending cancellation requests. cancel() increments the count and uncancel() decrements it. asyncio.timeout and TaskGroup use the count to tell their own cancellations from external ones. A coroutine that deliberately suppresses a CancelledError should call uncancel(), otherwise the count stays above zero and later code misreads the task as being cancelled.

Why does my program hang at exit after cancelling tasks?

A task is catching CancelledError and continuing, or its cleanup awaits something that never completes. asyncio.run cancels leftover tasks at exit and waits for them without a timeout, so such a task blocks shutdown indefinitely. Find it with a bounded asyncio.wait, log its name and stack, and fix the handler to re-raise.