Skip to content

How to Safely Share State Between Async Tasks and Threads

You have an asyncio service that also runs OS threads — a ThreadPoolExecutor for blocking I/O, a vendor SDK with its own callback thread, a to_thread worker — and they share state: a cache, a counter, a config dict. The symptoms when this goes wrong are nasty precisely because they are intermittent: a dictionary that is occasionally missing a key, a counter that is off by a few under load, an event loop that hangs for no obvious reason. The root cause is almost always the same: asyncio primitives are cooperative and loop-affine, OS threads are preemptive, and the two synchronization worlds do not mix.

This guide is a diagnostic-to-implementation path: prove where the boundary is, build a thread-safe bridge so the runtimes never touch the same object directly, pair locks correctly so neither side deadlocks the other, and validate the result under deliberately hostile concurrency.

Prerequisites

Decoupling thread producers from async consumers A thread-safe queue.Queue holds items from worker threads; a loop-side drain task moves them into an asyncio.Queue for coroutines, so the two runtimes never share an object. Bridge, never share Worker threads preemptive put() queue.Queue thread-safe + drain task run_in_executor(get) Async tasks cooperative await asyncio.Queue .get() The two runtimes never call methods on the same object.

1. Diagnose where the boundary actually is

Cross-runtime races show up as sporadic deadlocks, silently corrupted dictionaries, or an unresponsive loop that is blocked waiting on a thread-held resource. Before adding synchronization, find the exact point where the loop yields to a thread and where shared state is mutated. Amplify the race window by shrinking the thread switch interval, and capture C-level stacks with faulthandler.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import asyncio
import logging
import sys
import threading
from functools import wraps
from typing import Any, Callable, Dict

logger = logging.getLogger("state_diagnostics")


def diagnostic_state_wrapper(func: Callable[..., Any]) -> Callable[..., Any]:
    @wraps(func)
    def wrapper(*args: Any, **kwargs: Any) -> Any:
        ctx = threading.current_thread().name
        try:
            is_async = asyncio.get_running_loop().is_running()
        except RuntimeError:
            is_async = False
        original = sys.getswitchinterval()
        sys.setswitchinterval(1e-6)            # widen the preemption window
        try:
            logger.debug("mutate enter | thread=%s | on_loop=%s | %s",
                         ctx, is_async, func.__name__)
            return func(*args, **kwargs)
        finally:
            sys.setswitchinterval(original)


    return wrapper


@diagnostic_state_wrapper
def update_shared_cache(cache: Dict[str, Any], key: str, value: Any) -> None:
    cache[key] = value

Verify: run with python -X faulthandler your_app.py. With setswitchinterval(1e-6), any unsynchronized compound mutation that was "usually fine" now corrupts or raises reliably — that reproducibility is the signal you have located the real boundary.

Diagnostic Hook: Log threading.current_thread().name and the on-loop flag at every mutation site. A mutation that logs on_loop=False for an object you assumed was loop-only is your bug.

2. Architect a thread-safe queue bridge

The robust fix is to stop sharing the object at all. Decouple the runtimes with two queues: a queue.Queue (thread-safe, blocking) on the thread side and an asyncio.Queue (loop-aware) on the async side, joined by a loop-side drain task. The thread never touches an asyncio object; the loop never blocks on a queue.Queue.get() directly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import asyncio
import queue
from typing import Any, Optional


class StateBridge:
    """Decouples thread producers from async consumers via dual queues."""

    def __init__(self, maxsize: int = 1000):
        self._thread_q: queue.Queue[Optional[Any]] = queue.Queue(maxsize=maxsize)
        self._async_q: asyncio.Queue[Any] = asyncio.Queue(maxsize=maxsize)
        self._drainer: Optional[asyncio.Task] = None

    def start(self) -> None:
        self._drainer = asyncio.create_task(self._drain())

    def submit_from_thread(self, payload: Any) -> None:
        """Thread-safe. Raises queue.Full as backpressure when saturated."""
        self._thread_q.put_nowait(payload)

    async def consume(self) -> Any:
        return await self._async_q.get()

    async def _drain(self) -> None:
        loop = asyncio.get_running_loop()
        while True:
            # Blocking get() runs in a thread so the loop stays responsive.
            payload = await loop.run_in_executor(None, self._thread_q.get)
            if payload is None:                 # poison pill -> shutdown
                break
            await self._async_q.put(payload)    # backpressure to the producer

    async def shutdown(self) -> None:
        self._thread_q.put(None)
        if self._drainer:
            await self._drainer

Verify: push items from several threads with submit_from_thread while a coroutine loops on consume(). Item count in equals item count out, with zero direct cross-runtime access. Watch self._async_q.qsize() climb toward maxsize under burst load — that bounded growth is correct backpressure, not a leak.

Diagnostic Hook: Track both queue.Queue.qsize() and asyncio.Queue.qsize(). A full thread queue with an empty async queue means the drain task is starved or stalled — check it is actually scheduled.

3. Pair locks correctly and prevent deadlock

When copying through a queue is too expensive and you must share mutable state in place, use a threading.Lock for cross-thread access and an asyncio.Lock only within the loop — and never confuse them. threading.Lock blocks the whole thread; held across an await, it freezes the loop. asyncio.Lock raises or misbehaves if acquired off its owning loop. Bound async acquisition with a timeout so loop starvation surfaces as an error instead of a silent hang.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import asyncio
import threading
from contextlib import asynccontextmanager
from typing import AsyncIterator


class CrossRuntimeStateManager:
    def __init__(self) -> None:
        self._thread_lock = threading.Lock()
        self._async_lock = asyncio.Lock()
        self._state: dict[str, str] = {}

    def update_from_thread(self, key: str, value: str,
                           loop: asyncio.AbstractEventLoop) -> None:
        with self._thread_lock:                 # synchronous, no await inside
            self._state[key] = value
        loop.call_soon_threadsafe(self._notify, key)   # back onto the loop

    @asynccontextmanager
    async def read(self, timeout: float = 2.0) -> AsyncIterator[dict[str, str]]:
        try:
            async with asyncio.timeout(timeout):
                await self._async_lock.acquire()
            yield dict(self._state)             # snapshot copy under the lock
        except TimeoutError as exc:
            raise RuntimeError("state read timed out: loop starvation?") from exc
        finally:
            if self._async_lock.locked():
                self._async_lock.release()

    def _notify(self, key: str) -> None:
        ...                                     # observer hook, runs on loop thread

Verify: the locked region in update_from_thread contains no await (it cannot — it is a sync method), and the async read always releases. Under contention the read either returns a consistent snapshot or raises the timeout RuntimeError; it never hangs the loop indefinitely.

Diagnostic Hook: Wrap asyncio.Lock.acquire() in asyncio.timeout(). A fired timeout is a precise, actionable signal of loop starvation — far better than an unbounded hang you have to attach a debugger to.

4. Validate under hostile concurrency

A bug that appears once per million operations will not show up in a casual test. Force interleaving: many threads writing while many coroutines read, with the switch interval cranked down, and assert a structural invariant (final count, checksum, version monotonicity) at the end.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import asyncio
import pytest
from concurrent.futures import ThreadPoolExecutor


@pytest.mark.asyncio
async def test_state_integrity_under_load():
    mgr = CrossRuntimeStateManager()
    iterations, threads = 5000, 8
    loop = asyncio.get_running_loop()

    def writer(start: int, end: int) -> None:
        for i in range(start, end):
            mgr.update_from_thread(f"key_{i}", f"val_{i}", loop)

    async def reader() -> int:
        async with mgr.read(timeout=1.0) as snap:
            return len(snap)

    with ThreadPoolExecutor(max_workers=threads) as ex:
        chunk = iterations // threads
        futures = [ex.submit(writer, i * chunk, (i + 1) * chunk)
                   for i in range(threads)]
        await asyncio.gather(*(reader() for _ in range(100)))  # concurrent reads
        for f in futures:
            f.result()                                          # surface thread errors

    async with mgr.read() as final:
        assert len(final) == iterations, f"corruption: {len(final)} != {iterations}"

Verify: run pytest -p no:cacheprovider. The final assertion must hold every run; flakiness here is not a flaky test, it is a real race you have not yet closed. Re-run with sys.setswitchinterval(1e-6) set in a fixture to multiply the pressure.

Diagnostic Hook: Run the suite repeatedly (pytest --count=50 with pytest-repeat) and treat any single failure as a hard fail. Cross-runtime races are probabilistic; one failure in fifty means the bug is present, not rare.

Verification

A correctly bridged service shows: the event loop never logs slow-callback warnings attributable to lock waits; the integrity test passes deterministically across dozens of runs; queue sizes rise and fall with load but do not grow unbounded; and no RuntimeError about a wrong event loop ever appears. If all four hold under your switch-interval-amplified load test, the boundary is race-free.

Pitfalls & Edge Cases

  • Holding threading.Lock across an await. This stalls the entire loop while the lock is held — every coroutine starves. Keep locked sections synchronous, or copy data out under the lock and process after release.
  • Using asyncio.Lock (or Queue/Event) from a worker thread. These are loop-affine and not thread-safe; off-loop use races or raises RuntimeError. Use threading.Lock/queue.Queue on the thread side and bridge with call_soon_threadsafe.
  • Assuming the GIL makes compound operations atomic. The GIL protects a single bytecode op, not a read-modify-write sequence (d[k] = d[k] + 1, lst.append then lst.pop). Multi-step mutations need an explicit lock.
  • Swallowing queue.Empty / QueueEmpty. Ignoring these leads to silent item loss or infinite blocking. Handle them explicitly or use the blocking get() off the loop via run_in_executor.
  • Forgetting cancellation cleanup. If the drain task is cancelled, in-flight items in the queue.Queue are stranded. Drain or re-enqueue on shutdown, and always send the poison pill.

Frequently Asked Questions

Can I use a single lock for both asyncio tasks and threads?

No. asyncio.Lock is not thread-safe and is bound to the loop that created it; using it from another thread races or raises RuntimeError. Use threading.Lock for cross-thread access and asyncio.Lock for intra-loop coordination, or decouple the runtimes entirely with a thread-safe queue bridge.

How do I prevent deadlocks when sharing state between async and threaded workers?

Never hold a threading.Lock across an await boundary, since it freezes the whole event loop. Keep locked regions synchronous and short, schedule state updates back onto the loop with loop.call_soon_threadsafe, and bound asyncio.Lock acquisition with asyncio.timeout() so starvation raises an error instead of hanging.

Is queue.Queue safe to use directly with asyncio?

It is thread-safe but its get() blocks, which would stall the event loop if called on the loop thread. Drain it from the loop via loop.run_in_executor(None, q.get) into an asyncio.Queue, or use a dedicated bridge task. Async consumers then await the asyncio.Queue normally.