Skip to content

Running an Event Loop in a Background Thread

Synchronous applications sometimes need async libraries that have no sync API — an async database driver, a gRPC aio channel, a websocket client — and need them with connection reuse across calls. Calling asyncio.run() per call creates and destroys a loop each time, so nothing persists. The robust bridge is one event loop running forever on a dedicated thread, owned by an object that sync code submits coroutines to. Built that way and tested: sixteen caller threads each making 50 calls to a 10 ms coroutine finished in 0.54 s — the loop interleaved all 800 calls, where running them back to back would take 8 s; an exception raised on the loop thread surfaced in the caller with its original type; and a call that hit its timeout had its coroutine cancelled on the loop rather than left running. This guide builds that LoopThread, including the shutdown sequence most examples omit.

Prerequisites

1. Start the loop on its own thread

import asyncio
import concurrent.futures as cf
import threading


class LoopThread:
    def __init__(self, name: str = "async-bridge") -> None:
        self.loop = asyncio.new_event_loop()
        self._ready = threading.Event()
        self.thread = threading.Thread(target=self._run, name=name, daemon=True)
        self.thread.start()
        self._ready.wait()

    def _run(self) -> None:
        asyncio.set_event_loop(self.loop)
        self.loop.call_soon(self._ready.set)
        self.loop.run_forever()

Waiting on _ready until the loop is actually running avoids a race where the first submission arrives before run_forever() starts. The thread is a daemon so a forgotten close() does not hang interpreter exit, but relying on that skips cleanup — see step 4. One LoopThread per process is the normal arrangement; create it after any fork(), never before, because a forked child gets the thread object but not the running thread.

Verify: LoopThread().loop.is_running() is True immediately after construction.

Who runs where with a background loop 4 stacked layers. Who runs where with a background loop sync callers web threads, CLI, workers LoopThread.run(coro) run_coroutine_threadsafe, blocks caller event loop thread run_forever, interleaves every call async resources pools, channels: created on this loop only Only the loop thread touches async objects; callers only ever hold concurrent futures.

2. Submit coroutines and wait with a timeout that cancels

asyncio.run_coroutine_threadsafe schedules a coroutine on the loop from another thread and returns a concurrent.futures.Future the caller can block on. A timeout on that future does not stop the coroutine by itself; cancel it explicitly:

    def run(self, coro, timeout: float | None = 30.0):
        fut = asyncio.run_coroutine_threadsafe(coro, self.loop)
        try:
            return fut.result(timeout)
        except cf.TimeoutError:
            fut.cancel()                 # propagates to the coroutine on the loop
            raise

Verified: after a 0.1 s timeout on a 5 s coroutine, the coroutine received CancelledError on the loop. Without fut.cancel() it would have kept running — and holding whatever connection it was using — after the caller had given up. Exceptions in the coroutine are re-raised by fut.result() in the caller with their original type; a ValueError raised on the loop arrived as a ValueError.

Never call run() from code that is itself running on the bridge loop: the caller would block the very thread that must run the coroutine. That deadlock, and how to detect it, is the subject of avoiding deadlocks when threads wait on the loop.

Verify: a call that times out leaves no task behind in asyncio.all_tasks(bridge.loop).

3. Create async resources on the bridge loop

Async objects bind to the loop they are created on. Create them through the bridge, so they live on its loop for the life of the process:

bridge = LoopThread()


async def _make_pool():
    import asyncpg
    return await asyncpg.create_pool(DSN, min_size=2, max_size=10)


pool = bridge.run(_make_pool())                         # lives on the bridge loop


def get_user(user_id: int) -> dict:                     # plain sync function
    async def q():
        async with pool.acquire() as conn:
            return dict(await conn.fetchrow("select * from users where id=$1", user_id))
    return bridge.run(q(), timeout=5)

Many caller threads can submit at once: each blocks only itself, while the loop interleaves their coroutines. Measured with 16 threads × 50 calls of 10 ms: 0.54 s total. That is the point of the bridge — sync callers get async concurrency and a shared pool. The limit is the loop thread itself: any blocking code inside these coroutines stalls every caller, so keep them strictly async.

Verify: under concurrent callers, the pool's connection count stays at or below max_size, and calls overlap in time.

800 calls of a 10 ms coroutine from 16 threads 2 horizontal bars comparing back to back (computed) with the others. 800 calls of a 10 ms coroutine from 16 threads back to back (computed) 8.0 s 16 threads into one loop 0.54 s Python 3.14; each call awaits asyncio.sleep(0.01) on the bridge loop. The loop interleaves every caller's coroutines, so sync threads get async concurrency.

4. Shut the loop down in order

A background loop needs the same shutdown steps asyncio.run() performs: cancel remaining tasks, close async generators, stop the loop, join the thread, close the loop. And async resources must be closed on the loop before it stops:

class LoopThread:  # continued
    def close(self, timeout: float = 5.0) -> None:
        async def _shutdown():
            tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]
            for t in tasks:
                t.cancel()
            await asyncio.gather(*tasks, return_exceptions=True)
            await self.loop.shutdown_asyncgens()

        asyncio.run_coroutine_threadsafe(_shutdown(), self.loop).result(timeout)
        self.loop.call_soon_threadsafe(self.loop.stop)
        self.thread.join(timeout)
        self.loop.close()


# application shutdown
bridge.run(pool.close())
bridge.close()

Verified: after close(), the thread was no longer alive and the loop was closed. Skipping the explicit close and letting the daemon thread die at exit leaves pools unclosed — "Unclosed connection" warnings, server-side connections left to time out — and any in-flight call is abandoned mid-statement. Register close() with atexit or the framework's shutdown hook. The general ordering is in shutting down async generators and executors cleanly.

Verify: process exit produces no unclosed-resource warnings, and the database shows the bridge's connections closing.

5. Use it as a context manager, and watch it

Wrap the lifecycle so a missing close() is impossible in scripts and tests, and expose a health check for services:

class LoopThread:  # continued
    def __enter__(self) -> "LoopThread":
        return self

    def __exit__(self, *exc) -> None:
        self.close()

    def healthy(self, timeout: float = 1.0) -> bool:
        try:
            self.run(asyncio.sleep(0), timeout)
            return True
        except Exception:
            return False


with LoopThread() as bridge:
    print(bridge.run(fetch_all()))

healthy() round-trips a trivial coroutine; if it times out, the loop thread is blocked or dead, which no other metric shows directly. Expose it on the service's health endpoint. When the bridge is part of a library rather than an application, the same structure appears in offering sync and async versions of one API.

Verify: block the loop thread with a deliberate time.sleep inside a coroutine; healthy() returns False within its timeout.

Shutting a background loop down without losing work A flow of 4 stages. Shutting a background loop down without losing work close pools on the loop bridge.run(pool.close()) cancel remaining tasks gather them shutdown_asyncgens finalise generators stop, join, close thread exits The same steps asyncio.run performs, done explicitly because nothing does them for you.

Verification

The bridge is correct when:

  • The loop is running before the first submission, and one bridge exists per process, created after any fork.
  • Timeouts cancel the coroutine, and exceptions reach callers with their original types.
  • Async resources are created and closed on the bridge loop.
  • Shutdown closes resources, cancels tasks and joins the thread, with no warnings at exit.

Diagnostic Hook: export the bridge's round-trip latency — run(asyncio.sleep(0)) timed every few seconds — and the number of pending tasks on its loop. Round-trip latency above a few milliseconds means something is blocking the loop thread or it is saturated; a pending-task count that only grows means callers are timing out without cancelling, or coroutines are hanging on resources without timeouts.

Pitfalls & edge cases

  • Calling run() from the bridge loop itself. Instant deadlock until the timeout.
  • Timeouts without fut.cancel(). The coroutine keeps running after the caller gave up.
  • Creating the bridge before fork(). The child has a loop object and no thread to run it.
  • Blocking calls inside bridge coroutines. They stall every caller in the process.

Frequently Asked Questions

How do I run an asyncio event loop in a background thread?

Create a loop with asyncio.new_event_loop, start a thread whose target calls loop.run_forever, and submit coroutines from other threads with asyncio.run_coroutine_threadsafe, blocking on the returned concurrent future's result.

Does a timeout on run_coroutine_threadsafe cancel the coroutine?

No. future.result(timeout) raises TimeoutError in the caller, but the coroutine keeps running unless you call future.cancel(), which then cancels it on the loop.

Can several threads use one background event loop?

Yes. Each submission blocks only its caller while the loop interleaves all coroutines. In testing, 16 threads making 800 calls of 10 ms finished in 0.54 s.

How do I stop a background event loop cleanly?

Close async resources on the loop, cancel and await remaining tasks, call shutdown_asyncgens, then call loop.stop via call_soon_threadsafe, join the thread and close the loop.