Skip to content

Running Async Tests in Parallel

Async tests spend most of their time waiting — on a database, a broker, a fake network — so they parallelize well across processes, and pytest-xdist runs them on several workers with one flag. What breaks is everything the tests share outside the process. Measured on Python 3.14 with pytest 9, pytest-asyncio 1.4 and pytest-xdist 3.8, on 200 tests that each reset a Redis counter, incremented it five times with 10 ms pauses and checked it: one process took 11.12 s; 2, 4 and 8 workers took 6.16 s, 3.61 s and 2.46 s — when each worker used its own key prefix. With all workers sharing one key, all 200 failed at every worker count, as concurrent increments from other workers broke each test's assertion. A session fixture that created a database schema produced 12 errors out of 40 tests on 4 workers when the schema name was shared, and 0 with a schema per worker. This guide parallelizes async tests and isolates what they share.

Prerequisites

1. Measure the serial baseline and the speed-up

Each test here does about 50 ms of awaited I/O against a real Redis server:

@pytest.mark.asyncio
@pytest.mark.parametrize("i", range(200))
async def test_counter(i, redis_client, ns):
    key = f"test:{ns}:counter"
    await redis_client.delete(key)
    for _ in range(5):
        await redis_client.incr(key)
        await asyncio.sleep(0.01)
    assert int(await redis_client.get(key)) == 5

pytest -q                 # 200 passed in 11.12 s
pytest -q -n 8            # 200 passed in 2.46 s (each worker with its own key prefix)

Measured: 11.12 s serially; 6.16 s with 2 workers, 3.61 s with 4 and 2.46 s with 8 — 4.5 times faster, with about 0.4 s of worker start-up included in the wall time. Each xdist worker is a separate process with its own event loop, so async fixtures and loop scopes behave exactly as in a serial run, per worker.

Verify: the parallel run passes the same tests as the serial one, and its wall time is recorded.

200 async tests against Redis, pytest-xdist A grid of 5 rows by 3 columns. 200 async tests against Redis, pytest-xdist workers shared Redis key key prefix per worker 1 (serial) 200 passed, 11.12 s 200 passed, 11.12 s 2 200 failed 200 passed, 6.16 s 4 200 failed 200 passed, 3.61 s 8 200 failed 200 passed, 2.46 s 4, session fixture creating a schema shared name: 12 errors of 40 schema per worker: 0 errors pytest 9.1, pytest-asyncio 1.4, pytest-xdist 3.8, Python 3.14.

2. Namespace shared external state per worker

Tests written for a serial run assume they own whatever they touch. With several workers, two tests can reset and modify the same key at the same time. Measured with one shared key: all 200 tests failed at 2, 4 and 8 workers, because increments from other workers' tests landed between a test's reset and its assertion. xdist sets PYTEST_XDIST_WORKER — gw0, gw1 and so on — in each worker; use it to namespace:

WORKER = os.environ.get("PYTEST_XDIST_WORKER", "gw0")

@pytest.fixture
def ns() -> str:
    return WORKER

# Redis keys:      f"test:{ns}:counter"
# queues/topics:   f"orders-{ns}"
# temp dirs:       tmp_path (already unique per test)

With the prefix, every run passed. Within a worker, tests still run one at a time, so a per-worker namespace is enough unless tests themselves run concurrently.

Verify: a search of the test suite finds no hard-coded key, queue, topic or file name outside a worker namespace.

3. Create session resources per worker

Session-scoped fixtures run once per worker, not once per run. A session fixture that creates a database schema under a fixed name runs on every worker at the same time:

SCHEMA = f"test_{os.environ.get('PYTEST_XDIST_WORKER', 'gw0')}"

@pytest_asyncio.fixture(loop_scope="session", scope="session")
async def db():
    conn = await asyncpg.connect(DSN)
    await conn.execute(f"DROP SCHEMA IF EXISTS {SCHEMA} CASCADE; CREATE SCHEMA {SCHEMA}; "
                       f"CREATE TABLE {SCHEMA}.items (id int PRIMARY KEY)")
    yield conn
    await conn.execute(f"DROP SCHEMA IF EXISTS {SCHEMA} CASCADE")
    await conn.close()

Measured with 40 tests on 4 workers: with one shared schema name, 28 passed and 12 errored, as workers dropped and recreated the schema under each other; with a schema per worker, all 40 passed in 0.94 s. The same applies to databases, Kafka topics, Redis databases and object-storage buckets. For expensive one-time setup, create a template once — xdist's documentation shows a file-lock pattern for "once per run" fixtures — and give each worker a copy.

Verify: each session fixture's external resources include the worker ID in their names.

Wall time for 200 async tests 4 horizontal bars comparing serial with the others. Wall time for 200 async tests serial 11.12 s -n 2 6.16 s -n 4 3.61 s -n 8 2.46 s With a key prefix per worker; shared keys failed every test.

4. Give servers and ports per-worker values

Tests that start a server — a live ASGI app, a fake upstream — on a fixed port collide in the same way: the second worker's server fails to bind. Let the OS choose:

@pytest.fixture(scope="session")
def live_server():
    sock = socket.socket()
    sock.bind(("127.0.0.1", 0))                    # ephemeral port, unique per worker
    port = sock.getsockname()[1]
    sock.close()
    proc = subprocess.Popen([sys.executable, "-m", "uvicorn", "app:app", "--port", str(port)])
    wait_until_listening("127.0.0.1", port)
    yield f"http://127.0.0.1:{port}"
    proc.terminate()
    proc.wait(timeout=10)

Binding to port 0 and releasing it leaves a small window in which another process could take the port; servers that accept a pre-bound socket close it entirely. In-process ASGI tests need no port at all, as in testing ASGI apps with httpx ASGITransport, which is another reason to keep most API tests in-process.

Verify: no test fixture uses a fixed port number.

5. Choose the worker count and keep order-independence

The speed-up flattens as workers contend for the same external services and CPU: 2.46 s with 8 workers against 3.61 s with 4 here. Use -n auto locally and a fixed count in CI matched to the runner, and keep an eye on the slowest tests, since a worker that draws them sets the wall time; --dist loadgroup with @pytest.mark.xdist_group keeps related tests on one worker when they must share state:

pytest -n 8 --dist loadgroup          # tests marked with the same xdist_group share a worker
pytest -p no:xdist                    # serial run for debugging

Parallel runs reorder tests, which exposes hidden dependencies between them; fix those rather than serializing. A test that passes only after another one has run will fail randomly under xdist. Leaks that only appear when many tests share a process are caught by detecting leaked tasks in tests.

Verify: the suite passes with -n 8 and with -p no:xdist, and with the test order randomized.

Making an async suite parallel-safe A flow of 5 stages. Making an async suite parallel-safe Baseline serial: 11.12 s xdist -n 8: 2.46 s Namespace keys, topics by worker id Session fixtures schema per worker Ports ephemeral, never fixed Shared names failed every test; per-worker names failed none.

Verification

Async tests run safely in parallel when:

  • External names include the worker ID — keys, queues, schemas, topics.
  • Session fixtures create per-worker resources and clean them up.
  • Servers use ephemeral ports.
  • The suite passes in parallel, serially and in random order.

Diagnostic Hook: when a suite passes serially but fails under pytest -n, look for external names without a worker ID. A shared Redis key failed all 200 tests at every worker count here, and a shared schema name errored 12 of 40.

Pitfalls & edge cases

  • Shared keys across workers. Measured: 200 of 200 tests failed.
  • Session fixtures creating fixed-name resources. Measured: 12 errors of 40.
  • Fixed ports for test servers. The second worker cannot bind.
  • Assuming session scope means once per run. It is once per worker.

Frequently Asked Questions

Can pytest-asyncio tests run in parallel?

Yes, across processes with pytest-xdist: 200 I/O-bound tests went from 11.12 s to 2.46 s with -n 8. Each worker has its own event loop.

Why do my async tests fail with pytest -n?

Usually shared external state. Tests sharing one Redis key failed 200 of 200 under xdist; giving each worker its own prefix fixed it.

How do I get the xdist worker ID in a fixture?

Read os.environ["PYTEST_XDIST_WORKER"] (gw0, gw1, ...) and include it in names of keys, schemas, topics and other external resources.

Do session-scoped fixtures run once with pytest-xdist?

Once per worker. A session fixture creating a shared schema caused 12 errors in 40 tests on 4 workers; a schema per worker caused none.