Skip to content

Avoiding InvalidStateError When Setting Future Results

asyncio.InvalidStateError: invalid state is one of those errors that shows up rarely, under load, in logs from a callback nobody remembers writing. The future-based bridge worked in every test; in production, a reply arrived a few milliseconds after its request timed out, or two replicas answered the same request, or a retry path resolved a future the original path had already resolved. A future can be settled exactly once — with a result, an exception, or cancellation — and every later attempt to settle it raises. Reading it has matching rules: asking a pending future for its result raises too. None of this is complicated, but the races that trigger it are timing-dependent, so the fix has to be structural rather than a try/except sprinkled where the last traceback appeared. This guide maps the states and transitions, reproduces each error, and builds the settle-once and first-wins patterns that remove them.

Prerequisites

A future settles exactly once 4 stages from pending to settle again. A future settles exactly once pending await suspends settle once result, error, cancel terminal readable forever settle again InvalidStateError cancel() on a terminal future returns False instead of raising.

1. Reproduce every InvalidStateError

A future has three terminal states — finished with a result, finished with an exception, cancelled — and one non-terminal state, pending. Transitions are allowed only out of pending. The quickest way to internalise the rules is to trip each one.

import asyncio


async def main() -> None:
    loop = asyncio.get_running_loop()

    pending = loop.create_future()
    for label, read in [("result() while pending", pending.result),
                        ("exception() while pending", pending.exception)]:
        try:
            read()
        except asyncio.InvalidStateError as exc:
            print(f"{label}: {exc}")                    # Result is not set. / Exception is not set.

    done = loop.create_future()
    done.set_result(1)
    for label, settle in [("set_result twice", lambda: done.set_result(2)),
                          ("set_exception after result", lambda: done.set_exception(ValueError()))]:
        try:
            settle()
        except asyncio.InvalidStateError as exc:
            print(f"{label}: {exc}")                    # invalid state

    cancelled = loop.create_future()
    cancelled.cancel()
    try:
        cancelled.set_result(1)
    except asyncio.InvalidStateError as exc:
        print(f"set_result after cancel: {exc}")        # invalid state
    try:
        cancelled.result()
    except asyncio.CancelledError:
        print("result() of a cancelled future raises CancelledError")

    print("cancel() on a finished future returns:", done.cancel())   # False, no error


asyncio.run(main())

Note the asymmetry: settling a finished future raises, while cancel() on a finished future just returns False. And reading a cancelled future raises CancelledError, not InvalidStateError. Tasks add one more rule: task.set_result() raises RuntimeError: Task does not support set_result operation, because only the task's coroutine may finish it.

Verify: each line prints its specific message, and the final line shows cancel() returning False.

2. Find the race that settles a future twice

In real code the second settle comes from a different path than the first. The classic pair is a timeout and a late reply: the waiter's timeout cancels the future, then the reply handler calls set_result() on it.

import asyncio


class ReplyRouter:
    def __init__(self) -> None:
        self.pending: dict[int, asyncio.Future] = {}

    async def request(self, rid: int, timeout: float) -> str:
        future = asyncio.get_running_loop().create_future()
        self.pending[rid] = future
        try:
            async with asyncio.timeout(timeout):
                return await future                     # timeout cancels `future`
        finally:
            pass                                        # bug 1: entry is never removed

    def on_reply(self, rid: int, payload: str) -> None:
        self.pending[rid].set_result(payload)           # bug 2: no state check


async def main() -> None:
    router = ReplyRouter()
    loop = asyncio.get_running_loop()
    loop.call_later(0.05, router.on_reply, 7, "late reply")     # arrives after the timeout
    try:
        await router.request(7, timeout=0.01)
    except TimeoutError:
        print("request timed out")
    await asyncio.sleep(0.1)                            # the late reply raises in the callback


asyncio.run(main())

Running it prints request timed out and then the loop's exception handler reports InvalidStateError from on_reply. Both bugs contribute: the stale entry lets the late reply find the future at all, and the missing check lets it try to settle a cancelled one. Each bug alone would still cause trouble — the stale entry also leaks memory for every timed-out request.

Verify: the exception handler output names on_reply and invalid state, about 50 ms after the timeout message.

3. Settle once with a guarded helper

Fix the structure, not the symptom. Every code path that resolves a future goes through one helper that checks done() and reports whether it actually settled the future, and the owner removes the future from any lookup table when it completes, whatever the outcome.

import asyncio


def settle(future: asyncio.Future, *, result=None, exception: BaseException | None = None) -> bool:
    """Resolve `future` if it is still pending. Returns True if this call settled it."""
    if future.done():
        return False
    if exception is not None:
        future.set_exception(exception)
    else:
        future.set_result(result)
    return True


class SafeReplyRouter:
    def __init__(self) -> None:
        self.pending: dict[int, asyncio.Future] = {}
        self.late_replies = 0

    async def request(self, rid: int, timeout: float) -> str:
        future = asyncio.get_running_loop().create_future()
        self.pending[rid] = future
        future.add_done_callback(lambda _f: self.pending.pop(rid, None))
        async with asyncio.timeout(timeout):
            return await future

    def on_reply(self, rid: int, payload: str) -> None:
        future = self.pending.get(rid)
        if future is None or not settle(future, result=payload):
            self.late_replies += 1                       # a metric, not an exception


async def main() -> None:
    router = SafeReplyRouter()
    loop = asyncio.get_running_loop()
    loop.call_later(0.05, router.on_reply, 7, "late reply")
    try:
        await router.request(7, timeout=0.01)
    except TimeoutError:
        pass
    await asyncio.sleep(0.1)
    print("late replies:", router.late_replies, "| pending entries:", len(router.pending))   # 1 | 0


asyncio.run(main())

Counting late replies instead of discarding them silently keeps an operational signal: a rising late-reply rate means timeouts are set below the dependency's real latency. On a single event loop the done() check and the set_result() cannot be interleaved by other coroutines because there is no await between them; from other threads, the helper must be scheduled onto the loop.

Verify: no exception is reported, late replies: 1, and the pending table is empty.

A late reply racing a timeout 4 lanes over time. A late reply racing a timeout waiter await future timeout future pending cancelled reply, unguarded set_result reply, guarded done(): count late time → The guard and the table cleanup turn an exception into a metric.

4. Let the first of several producers win

Hedged requests, replica reads and "whichever source answers first" logic deliberately have several producers for one future. Expect all but one of them to lose, make losing cheap, and cancel the losers' work.

import asyncio
import random


async def replica(name: str, winner: asyncio.Future, delay: float) -> None:
    await asyncio.sleep(delay)
    if settle(winner, result=f"answer from {name}"):
        print(f"{name} won")
    # losers simply return: no exception, no log noise


async def first_answer(replicas: dict[str, float]) -> str:
    loop = asyncio.get_running_loop()
    winner = loop.create_future()
    tasks = [asyncio.create_task(replica(name, winner, delay)) for name, delay in replicas.items()]
    try:
        return await winner
    finally:
        for task in tasks:
            task.cancel()                                # stop the losers' remaining work
        await asyncio.gather(*tasks, return_exceptions=True)


async def main() -> None:
    delays = {"eu-1": 0.03, "eu-2": 0.01, "us-1": 0.05}
    print(await first_answer(delays))                   # eu-2 won / answer from eu-2


asyncio.run(main())

For a small fixed set of coroutines, asyncio.wait(..., return_when=FIRST_COMPLETED) achieves the same without a manual future. The explicit future is useful when producers are callbacks or when results arrive through different mechanisms. Adding a delay before launching the extra producers turns this into request hedging, covered in hedging requests to cut tail latency.

Verify: exactly one won line prints, the result matches the fastest replica, and no losing replica raises.

5. Read futures only when they are done

The reading side has its own rules. result() and exception() are for futures that are already done — typically inside a done-callback, or after asyncio.wait() reported them done. Everywhere else, await the future, which suspends until it is settled and raises the stored exception or CancelledError for you.

import asyncio


def describe(future: asyncio.Future) -> str:
    """Safe inspection of any future, in any state."""
    if not future.done():
        return "pending"
    if future.cancelled():
        return "cancelled"
    exc = future.exception()                             # also marks it as retrieved
    return f"failed: {exc!r}" if exc else f"result: {future.result()!r}"


async def main() -> None:
    loop = asyncio.get_running_loop()
    futures = [loop.create_future() for _ in range(4)]
    futures[1].set_result(42)
    futures[2].set_exception(LookupError("missing"))
    futures[3].cancel()
    print([describe(f) for f in futures])

    futures[0].add_done_callback(lambda f: print("callback sees:", describe(f)))
    futures[0].set_result("late")
    await asyncio.sleep(0)


asyncio.run(main())

Calling exception() on a failed future marks the exception as retrieved, which suppresses the "Future exception was never retrieved" warning — useful in monitoring code that inspects futures it does not own. Do not store StopIteration in a future: current Python versions convert it to a RuntimeError because it interacts badly with generator-based iteration.

Verify: the list prints pending, result: 42, failed: LookupError('missing') and cancelled, and the callback sees result: 'late'.

Reading a future in each state A grid of 4 rows by 3 columns. Reading a future in each state state result() exception() await pending InvalidStateError InvalidStateError suspends result the value None the value exception raises it the exception raises it cancelled CancelledError CancelledError CancelledError Only await is safe in every state; the methods are for done futures.

Verification

Future handling is robust when:

  • Every resolution goes through one guarded path that checks done() and reports whether it settled the future.
  • Lookup tables clean themselves via a done-callback, so late events find nothing to settle.
  • Late or losing producers are counted, not raised, giving a signal about timeouts and hedging.
  • Readers await or inspect only done futures, never calling result() on pending ones.
  • Tasks are never settled externally; they are cancelled or awaited instead.

Pitfalls & edge cases

  • Wrapping set_result in try/except InvalidStateError. It hides the race but keeps the stale table entry and loses the late-reply signal. Guard with done() and clean up instead.
  • Checking done() in one callback and settling in another. Any await or scheduled hop between the check and the set reopens the race. Keep them in the same synchronous step.
  • Cross-thread resolution. The guard is only race-free on the loop thread; from other threads, schedule it with call_soon_threadsafe.
  • Futures shared across loops. A future belongs to the loop that created it; settling it from another loop's callbacks has the same problems as another thread.
  • Forgetting cancelled waiters. A future cancelled by its only waiter still occupies memory in any table that references it until removed.

Frequently Asked Questions

What causes asyncio InvalidStateError: invalid state?

Trying to settle a future that is no longer pending: calling set_result or set_exception on a future that already has a result or exception, or that was cancelled. The usual trigger is a race such as a reply arriving after a timeout cancelled the future, or two code paths both resolving the same future.

Why do I get InvalidStateError: Result is not set?

You called result or exception on a future that is still pending. Those methods only work on done futures. Await the future instead, which suspends until it is settled, or call result only inside a done-callback or after asyncio.wait has reported the future as done.

How do I safely set a future's result when it might be cancelled?

Check future.done() immediately before calling set_result or set_exception, in the same synchronous step on the event loop thread, and skip it if the future is already done. Remove the future from any lookup table with a done-callback so late events cannot find it.

Can I call set_result on an asyncio Task?

No. A Task's outcome is determined by its coroutine, so set_result and set_exception raise RuntimeError. To stop a task early, call cancel on it; to provide a value from outside, use a separate Future that the task awaits.