Future Objects & Callbacks in Python Asyncio

A Future is the single-result awaitable that sits underneath everything asyncio does, and the callback APIs (call_soon, call_later, call_at, add_done_callback, call_soon_threadsafe) are the loop's only public way to put work onto the ready queue. This reference is narrow on purpose: not coroutines, not high-level orchestration, but the low-level mechanics of manually creating a future with loop.create_future(), resolving it with set_result/set_exception, registering completion callbacks, scheduling plain callables, and crossing the thread boundary safely. These are the primitives you reach for when you are bridging a non-async world — a C extension, an SDK that calls you back, a thread pool returning results — into async/await without corrupting loop state.

Everything below flows from one fact: the event loop is a single thread that drains a queue of callbacks. A Future is a small object holding a state, a result-or-exception slot, and a list of done-callbacks. "Resolving" a future means stamping its result and scheduling its done-callbacks onto that queue; "awaiting" a future means parking the current coroutine as one of those callbacks. Getting the queue discipline right — and never touching a future from the wrong thread — is the whole game.

Architectural principles

• A Future is just state plus a callback list. It is not driven by a coroutine. Some external party — the loop, a timer, a thread, your own code — must call set_result() or set_exception(), or the future stays PENDING forever and every awaiter hangs.
• Resolution schedules, it does not run. set_result() does not invoke done-callbacks inline; it marks the future done and uses loop.call_soon() to enqueue each callback for the next loop iteration. There is always at least one tick between resolution and an awaiter resuming.
• Callbacks run synchronously in the loop thread. Once a done-callback or a call_soon callable is popped from the ready queue it runs to completion with no yielding. Any blocking or CPU-heavy work in it stalls the entire loop until it returns.
• The loop is not thread-safe except for one method. Every future and loop method assumes it is called from the loop thread. The single sanctioned cross-thread entry point is loop.call_soon_threadsafe(); everything else from another thread is undefined behaviour.
• A Future is a bridge, a Task is a driver. Task is a Future subclass that drives a coroutine. Use a raw Future to represent a result produced outside the coroutine machinery; use a Task to run a coroutine concurrently.

How callbacks land on the ready queue

To use these primitives correctly you have to picture the loop's core iteration. Each turn of the loop does three things in a fixed order: it runs every callback currently in the ready deque (loop._ready), it polls the selector for sockets that became readable/writable and appends their callbacks, then it checks the timer heap (loop._scheduled) and moves any due timers into the ready deque. The loop then blocks in the selector until the next I/O event or the next timer deadline, and repeats. For the full anatomy of how the selector, timers, and executors compose into one iteration, start from Asyncio Fundamentals & Event Loop Architecture, the overview this section sits under.

Each callback API feeds that machinery at a different point. loop.call_soon(cb) appends directly to _ready, so cb runs on the very next iteration. loop.call_later(delay, cb) and loop.call_at(when, cb) push onto the timer heap; the loop only moves them to _ready once their deadline passes, so they are subject to scheduling latency, never sub-tick precision. future.set_result() walks the future's done-callback list and call_soons each one. And loop.call_soon_threadsafe(cb) does what call_soon does but additionally writes a byte to the loop's self-pipe, forcing the selector to wake immediately instead of sleeping until the next deadline — which is precisely why it is the only safe cross-thread call. Awaiting a future is the mirror image: await fut registers the running task as a done-callback on fut and suspends; when something resolves fut, that done-callback re-enqueues the task's next step. This is the same producer/consumer cycle that drives task scheduling and lifecycle — a task is simply a future that keeps re-scheduling itself.

Pattern catalogue

The five patterns below are the load-bearing uses of futures and callbacks. Each is a recipe with a clear trigger; most production code needs only the first two, but the thread-safe and wrapping variants are essential when you integrate non-async libraries.

Create a future and resolve it from a callback

The canonical use: you need an awaitable whose result arrives from something the loop will call back — a timer, an I/O event, or your own scheduled callable. Create it with loop.create_future() (preferred over asyncio.Future() because it picks up the loop's configured future class), hand it to the producer, and await it.

import asyncio


async def resolve_from_timer() -> str:
    loop = asyncio.get_running_loop()
    fut = loop.create_future()

    # call_later schedules a plain callable onto the timer heap; when the
    # deadline passes the loop moves it into _ready and runs it, which
    # resolves the future and wakes our await.
    loop.call_later(0.1, fut.set_result, "deadline reached")

    return await fut


print(asyncio.run(resolve_from_timer()))

Prefer loop.create_future() over constructing asyncio.Future() directly: the former binds the future to the running loop and respects custom loop implementations such as uvloop. Pass fut.set_result as the callback itself rather than wrapping it in a lambda when you can — fewer closures, fewer reference cycles.

Diagnostic Hook: A future that is created but whose producer never fires leaves the awaiter stuck. Wrap every await fut in asyncio.timeout() so a missing resolution surfaces as a TimeoutError instead of a silent hang, and log the future's repr() (it shows state and registered callbacks) when the timeout trips.

Register a completion callback with add_done_callback

When you want to react to a future finishing without awaiting it — fan-out monitoring, metrics, cleanup — register a done-callback. It fires once, in the loop thread, on the tick after the future reaches DONE or CANCELLED, and receives the future as its only argument.

import asyncio


def on_done(fut: asyncio.Future) -> None:
    # Runs in the loop thread on the tick after resolution. Must not block.
    if fut.cancelled():
        print("cancelled")
    elif (exc := fut.exception()) is not None:
        print(f"failed: {exc!r}")
    else:
        print(f"ok: {fut.result()}")


async def main() -> None:
    loop = asyncio.get_running_loop()
    fut = loop.create_future()
    fut.add_done_callback(on_done)
    fut.set_result(42)
    await asyncio.sleep(0)  # let the scheduled callback run


asyncio.run(main())

Callbacks fire in registration (FIFO) order. Always branch on cancelled() before calling result() or exception(), because both raise CancelledError on a cancelled future. Use remove_done_callback() to drop a handler that is no longer relevant; otherwise a callback that captures large objects keeps them alive for the future's lifetime.

Diagnostic Hook: Exceptions raised inside a done-callback do not propagate — they are routed to loop.call_exception_handler. Install a custom exception handler in production so these are logged with context instead of vanishing, and assert len(fut._callbacks) stays bounded under load to catch handlers you forgot to remove.

Schedule a plain callable with call_soon / call_later / call_at

When you need to defer work to a later tick without an awaitable in play — yielding control to break up a long section, debouncing, or firing a timer — schedule a callable directly. All three return a Handle (or TimerHandle) whose .cancel() deschedules the pending call.

import asyncio


async def main() -> None:
    loop = asyncio.get_running_loop()

    loop.call_soon(lambda: print("runs next tick"))
    h = loop.call_later(0.05, lambda: print("runs after 50ms"))
    loop.call_at(loop.time() + 0.10, lambda: print("runs at absolute deadline"))

    # Descheduling a not-yet-fired timer is cheap and safe.
    if False:
        h.cancel()

    await asyncio.sleep(0.2)


asyncio.run(main())

Use call_at with loop.time() (a monotonic clock) for absolute deadlines that must not drift if the wall clock changes; use call_later for relative delays. Neither is precise: the callback fires on the first loop iteration after the deadline, so a busy loop adds latency. For anything you need to wait on, prefer resolving a future (the first pattern) over polling a flag set by call_later.

Diagnostic Hook: Timer drift is your signal of an overloaded loop. Record loop.time() inside a call_later callback and compare against the intended deadline; a growing gap means callbacks ahead of it in _ready are running too long. Cross-reference with loop.slow_callback_duration warnings.

Resolve a future safely from another thread

The most common real-world need: a background thread (or thread-pool worker) produces a result and must hand it to an awaiting coroutine. You cannot call fut.set_result() from that thread — the future and loop are not thread-safe. Route the resolution through loop.call_soon_threadsafe(), which enqueues the call on the loop thread and wakes the selector.

import asyncio
import threading


def worker(loop: asyncio.AbstractEventLoop, fut: asyncio.Future, payload: int) -> None:
    try:
        result = payload * 2  # stand-in for blocking work
        # Capture the future's loop in this thread; never touch fut directly.
        loop.call_soon_threadsafe(fut.set_result, result)
    except Exception as exc:  # noqa: BLE001
        loop.call_soon_threadsafe(fut.set_exception, exc)


async def main() -> int:
    loop = asyncio.get_running_loop()
    fut = loop.create_future()
    threading.Thread(target=worker, args=(loop, fut, 21), daemon=True).start()
    return await fut


print(asyncio.run(main()))

The loop reference must be the one running the await — capture it with get_running_loop() on the async side and pass it in. call_soon_threadsafe is the only loop method safe to call off-thread; for the broader discipline of sharing state across the async/thread boundary see hybrid concurrency models, and for offloading the blocking work itself see CPU-bound task offloading.

Diagnostic Hook: If results never arrive, confirm in the worker that loop.is_running() is true and that you passed the running loop, not one fetched with get_event_loop() in the wrong context. A RuntimeError: ... attached to a different loop is the tell that the future and the resolving call belong to different loops.

Wrap a callback API as an awaitable

The payoff pattern: package the future plumbing into a reusable coroutine so callers see clean async/await. This is exactly how asyncio itself adapts protocols and transports. Create the future, register it with the callback API, and await it — exposing nothing of the future to the caller.

import asyncio
from typing import Any, Callable


def register(on_done: Callable[[Any], None]) -> None:
    """Stand-in for a third-party API that calls back with a result."""
    asyncio.get_event_loop().call_later(0.1, on_done, {"status": "ok"})


async def call_async() -> Any:
    loop = asyncio.get_running_loop()
    fut = loop.create_future()

    def _callback(result: Any) -> None:
        if not fut.done():  # guard against double-resolution
            fut.set_result(result)

    register(_callback)
    return await fut


print(asyncio.run(call_async()))

The if not fut.done() guard is essential whenever the underlying API might call back more than once: a second set_result() raises InvalidStateError. The next page builds this into a production-grade helper with timeouts and thread safety — see bridging callback APIs to async with Futures.

Diagnostic Hook: Wrappers leak if the underlying API can finish without calling back (error path, timeout). Always pair the await with asyncio.timeout() and register a cancellation/cleanup path that deregisters the callback, then count outstanding wrapper futures as a gauge metric so a callback that never fires shows up as a slow leak rather than a single hang.

Resource boundaries

Futures and callbacks have no built-in backpressure — the loop will accept unlimited pending timers, scheduled callables, and unresolved futures. The boundaries you must impose:

• Cap concurrent wrapper futures. Each in-flight create_future() bridge holds the awaiting task, its callback closure, and any captured payload. Under a burst of inbound callbacks this is unbounded memory. Gate creation behind a Semaphore or a bounded queue so you fail fast instead of OOM.
• Bound the timer heap. Thousands of call_later timers (a per-connection idle timeout, say) bloat loop._scheduled and slow every iteration's heap maintenance. Coalesce timers, or cancel the previous TimerHandle before scheduling a new one for the same key.
• Keep callbacks O(microseconds). A done-callback or call_soon callable holds the loop thread until it returns. Anything CPU-heavy belongs in run_in_executor, which itself hands back a Future you can await — the natural seam between callback land and async land.
• Drain on shutdown. Pending futures that are never resolved must be cancelled during teardown, and call_later timers cancelled, or loop.close() warns about pending callbacks and in-flight resolutions are lost.

Integrated production example

A bounded adapter that turns a callback-and-thread SDK into a clean async API: it caps concurrency with a semaphore, resolves futures across the thread boundary with call_soon_threadsafe, enforces a per-call deadline, and guarantees the future is always resolved exactly once.

import asyncio
import logging
import random
import threading
import time
from typing import Any, Callable

logging.basicConfig(level=logging.INFO)
log = logging.getLogger("bridge")


class CallbackSDK:
    """A fictional thread-based SDK: you submit work and it calls back later."""

    def submit(self, payload: int, on_done: Callable[[int | None, Exception | None], None]) -> None:
        def _run() -> None:
            time.sleep(random.uniform(0.05, 0.3))  # blocking work
            if payload < 0:
                on_done(None, ValueError(f"bad payload {payload}"))
            else:
                on_done(payload * 2, None)

        threading.Thread(target=_run, daemon=True).start()


class AsyncBridge:
    def __init__(self, sdk: CallbackSDK, max_inflight: int = 32) -> None:
        self._sdk = sdk
        self._gate = asyncio.Semaphore(max_inflight)  # bound concurrency
        self.inflight = 0

    async def call(self, payload: int, *, timeout: float = 1.0) -> int:
        loop = asyncio.get_running_loop()
        async with self._gate:
            fut: asyncio.Future[int] = loop.create_future()
            self.inflight += 1

            def _on_done(result: int | None, exc: Exception | None) -> None:
                # Runs in the SDK's thread -> hop back to the loop thread.
                def _resolve() -> None:
                    if fut.done():  # already timed out / cancelled
                        return
                    if exc is not None:
                        fut.set_exception(exc)
                    else:
                        fut.set_result(result)  # type: ignore[arg-type]

                loop.call_soon_threadsafe(_resolve)

            self._sdk.submit(payload, _on_done)
            try:
                async with asyncio.timeout(timeout):
                    return await fut
            finally:
                self.inflight -= 1


async def main() -> None:
    bridge = AsyncBridge(CallbackSDK(), max_inflight=8)
    payloads = [random.choice([5, 7, -1, 9]) for _ in range(12)]

    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(_guarded(bridge, p)) for p in payloads]
    log.info("results: %s", [t.result() for t in tasks])


async def _guarded(bridge: AsyncBridge, payload: int) -> str:
    try:
        return f"{payload}->{await bridge.call(payload)}"
    except (ValueError, TimeoutError) as exc:
        return f"{payload}->ERR({type(exc).__name__})"


asyncio.run(main())

Diagnostic Hook: Export bridge.inflight as a gauge. If it climbs toward max_inflight and stays there, the SDK is calling back slower than you submit — either the thread pool behind it is saturated or some calls never call back. Pair it with a counter of TimeoutErrors: a rising timeout rate with steady inflight means callbacks are being dropped, not merely delayed.

Diagnostic Hook: the must-watch signals for futures

Three signals catch the overwhelming majority of future/callback bugs in production. (1) Unresolved-future gauge — count create_future() calls minus resolutions; a monotonic rise is a producer that stopped firing. (2) loop.slow_callback_duration (default 0.1s) — lower it in staging to flush out done-callbacks doing real work in the loop thread. (3) Cross-loop errors — alert on any RuntimeError mentioning "different loop" or InvalidStateError; both mean a future is being resolved from the wrong thread or twice. Run with PYTHONASYNCIODEBUG=1 while reproducing to get tracebacks for callbacks that exceed the threshold.

Failure modes

Failure mode	Root cause	Detection	Fix
Awaiter hangs forever	Future created but producer never calls set_result/set_exception	Unresolved-future gauge rises; await never returns	Wrap awaits in asyncio.timeout(); ensure every code path resolves the future
InvalidStateError on resolve	set_result/set_exception called twice on the same future	Exception logged from a callback or resolver	Guard with if not fut.done(); ensure the callback API fires once
RuntimeError: different loop	Future resolved from a thread without call_soon_threadsafe, or loop mismatch	RuntimeError in worker; result never arrives	Capture the running loop with get_running_loop(); resolve via call_soon_threadsafe
Loop stalls / latency spikes	CPU-bound or blocking work inside a done-callback or call_soon callable	slow_callback_duration warnings; timer drift	Move work to run_in_executor/asyncio.to_thread; keep callbacks trivial
Silent swallowed errors	Exception raised inside add_done_callback handler	Nothing logged; behaviour just "stops"	Install a loop.call_exception_handler; branch on cancelled() before result()
Memory creep over days	Done-callbacks capturing large objects, never removed	tracemalloc shows closure growth; len(fut._callbacks) high	remove_done_callback() stale handlers; use weakref for self-referential targets

Frequently Asked Questions

When should I use asyncio.Future directly instead of asyncio.Task?

Use a raw Future to represent a single result produced outside the coroutine machinery: bridging a callback-based SDK, returning a value from a background thread, or building a custom awaitable. Use a Task to run a coroutine concurrently, because Task is a Future subclass that drives the coroutine and adds automatic exception propagation, cancellation, and stack-trace retention.

Why does set_result not run my done-callbacks immediately?

set_result marks the future done and schedules each registered done-callback with loop.call_soon, which appends it to the ready queue for the next loop iteration. The callbacks therefore run on a later tick, not inline. To observe them in a small script, yield control once with await asyncio.sleep(0).

How do I resolve a Future safely from a background thread?

Never call set_result or set_exception directly from another thread, because futures and the loop are not thread-safe. Capture the running loop on the async side with asyncio.get_running_loop(), pass it to the thread, and resolve via loop.call_soon_threadsafe(fut.set_result, value). That method is the only loop call safe to invoke off-thread; it enqueues the resolution and wakes the selector.

What causes InvalidStateError when resolving a Future?

InvalidStateError is raised when set_result or set_exception is called on a future that is already done, typically because the underlying callback fired twice or both a timeout and the callback resolved it. Guard every resolution with if not fut.done() before setting the result, and ensure each code path resolves the future exactly once.

Why are exceptions raised inside add_done_callback silently lost?

Done-callbacks run in the loop thread, and any exception they raise is routed to loop.call_exception_handler rather than propagated to an awaiter. If you have not installed a custom handler the error is only logged by the default handler and can be easy to miss. Install loop.set_exception_handler in production and keep callback bodies wrapped in try/except.

Related

• Asyncio Fundamentals & Event Loop Architecture — up to the overview for how the selector, timers, and executors compose the loop iteration that runs every callback.
• Bridging callback APIs to async with Futures — the step-by-step guide to turning a callback or thread-based library into an awaitable.
• Task Scheduling & Lifecycle — how a Task (a Future subclass) re-schedules itself through the same ready queue.
• Hybrid concurrency models — patterns for sharing state and results between async tasks and threads.
• CPU-bound task offloading — using run_in_executor, which returns a Future you await, to keep heavy work off the loop thread.