Skip to content

Choosing a Concurrency Model for a WebSocket Gateway

A market-data gateway must hold 50,000 WebSocket connections, push five price updates per second to each subscribed client, and authenticate new connections at a peak of a few hundred per second. The team's options are the usual three — a thread per connection, asyncio, or several processes — and the discussion usually runs on intuition: "threads don't scale", "Python is too slow for this", "just use Go". A gateway is a good case for replacing intuition with arithmetic, because its load decomposes cleanly into idle connections (memory), message fan-out (CPU per message × subscribers) and connection setup (bursty CPU). This guide measures each on real hardware, turns the measurements into a sizing model, and derives the architecture: asyncio for the connections, encode-once fan-out with per-client backpressure, and processes — not threads — when one loop's CPU runs out.

Prerequisites

Concurrency overhead for 50,000 idle connections 2 bars comparing coroutines + tasks with the others. Concurrency overhead for 50,000 idle connections coroutines + tasks 55 MiB threads (resident only) 1048 MiB Measured per unit with the step 1 script, multiplied by 50,000; excludes socket buffers. Threads also reserve 8 MiB of virtual stack each, which hits limits first.

1. Measure the memory of an idle connection

Most gateway connections are idle most of the time, so memory per idle connection bounds how many fit on a host. Measure the resident memory of many waiting coroutines and of many blocked threads in the same way: read VmRSS before and after creating them.

import asyncio
import sys
import threading
import time


def rss_kib() -> int:
    with open("/proc/self/status") as status:           # Linux
        for line in status:
            if line.startswith("VmRSS"):
                return int(line.split()[1])
    raise RuntimeError("VmRSS not found")


def coroutines(n: int) -> float:
    async def connection(closed: asyncio.Event) -> None:
        await closed.wait()                               # an idle client

    async def run() -> float:
        base = rss_kib()
        closed = asyncio.Event()
        tasks = [asyncio.create_task(connection(closed)) for _ in range(n)]
        await asyncio.sleep(0.2)
        used = rss_kib() - base
        closed.set()
        await asyncio.gather(*tasks)
        return used / n

    return asyncio.run(run())


def threads(n: int) -> float:
    base = rss_kib()
    closed = threading.Event()
    workers = [threading.Thread(target=closed.wait, daemon=True) for _ in range(n)]
    for w in workers:
        w.start()
    time.sleep(0.5)
    used = rss_kib() - base
    closed.set()
    for w in workers:
        w.join()
    return used / n


if __name__ == "__main__":
    print(f"coroutine + task: {coroutines(50_000):.2f} KiB each")
    print(f"thread:           {threads(5_000):.2f} KiB each")

On Linux with Python 3.14, an idle coroutine with its task used 1.13 KiB of resident memory, and an idle thread 21.5 KiB — about nineteen times more — before counting the default 8 MiB of virtual stack reserved per thread, which is what exhausts address space and thread limits well before resident memory does. A real WebSocket connection adds its socket buffers and protocol state to either model, typically tens of kilobytes, but that cost is the same for both; the difference is the per-connection overhead of the concurrency unit itself.

Verify: the thread figure is an order of magnitude above the coroutine figure on your host; at 50,000 connections the difference is roughly 55 MiB versus 1 GiB of overhead.

2. Measure CPU per message and derive fan-out cost

The dominant CPU cost in a push gateway is preparing and writing each outbound message. Encoding a message once and sending the same bytes to every subscriber turns a per-client cost into a per-message cost.

import json
import time

PRICE_UPDATE = {"type": "price", "symbol": "ACME", "bid": 101.25, "ask": 101.27,
                "ts": 1789661814013, "book": [[101.2, 300], [101.1, 500], [101.0, 900]]}


def encode_cost(n: int = 200_000) -> float:
    started = time.perf_counter()
    for _ in range(n):
        json.dumps(PRICE_UPDATE)
    return (time.perf_counter() - started) / n


def fanout_cpu_seconds_per_second(clients: int, msgs_per_client_s: float, symbols: int,
                                  encode_s: float, write_s: float, encode_once: bool) -> float:
    messages = clients * msgs_per_client_s
    encodes = messages if not encode_once else symbols * msgs_per_client_s
    return encodes * encode_s + messages * write_s


if __name__ == "__main__":
    encode = encode_cost()
    print(f"json.dumps: {encode * 1e6:.2f} µs per message")
    for once in (False, True):
        cores = fanout_cpu_seconds_per_second(clients=50_000, msgs_per_client_s=5, symbols=500,
                                              encode_s=encode, write_s=8e-6, encode_once=once)
        print(f"encode_once={once}: {cores:.2f} cores for 250,000 sends/s")

json.dumps measured about 2 µs per update. With 50,000 clients receiving five updates per second, per-client encoding alone costs 0.5 cores; encoding once per symbol reduces encoding to almost nothing. What remains is the per-send cost — framing and a transport write, assumed here at 8 µs, which you should measure for your WebSocket library — giving roughly two cores of work for 250,000 sends per second. One event loop provides one core of Python execution, so this workload needs about three event loops, which is the number that decides the architecture.

Verify: the per-client encoding variant costs visibly more than the encode-once variant, and the remaining cost is dominated by the per-send term.

Where the gateway spends CPU 4 stacked layers from Encode to Result. Where the gateway spends CPU Encode once per update ~2 µs per message negligible if shared Send per client per message framing + write dominant: ~2 cores Connect token verification bursty after deploys offload + bound Result ~2 cores of sends 60% per loop 4 processes Connections cost memory; messages cost CPU; the CPU decides the process count.

3. Hold connections on asyncio, with backpressure per client

The numbers from steps 1 and 2 rule out a thread per connection on memory, and they show that throughput per process is CPU-bound, not connection-bound. Within one process, asyncio holds the connections; the design detail that keeps it stable is a bounded outbound queue per client, so one slow consumer cannot stall the fan-out.

import asyncio
from dataclasses import dataclass, field


@dataclass(eq=False)                                       # identity hashing: usable in sets
class Client:
    name: str
    send_delay: float                                      # simulates network speed
    outbox: asyncio.Queue = field(default_factory=lambda: asyncio.Queue(maxsize=64))
    sent: int = 0
    dropped: int = 0

    async def writer(self) -> None:
        while True:
            frame = await self.outbox.get()
            await asyncio.sleep(self.send_delay)          # await ws.send(frame)
            self.sent += 1


class Gateway:
    def __init__(self) -> None:
        self.subscribers: dict[str, set[Client]] = {}

    def publish(self, symbol: str, payload: bytes) -> None:
        for client in self.subscribers.get(symbol, ()):   # encoded once, shared bytes
            try:
                client.outbox.put_nowait(payload)
            except asyncio.QueueFull:
                client.dropped += 1                        # slow client: drop, never block publish


async def main() -> None:
    gateway = Gateway()
    fast = [Client(f"fast-{i}", 0.0) for i in range(1000)]
    slow = Client("slow", 0.05)
    gateway.subscribers["ACME"] = set(fast) | {slow}
    writers = [asyncio.create_task(c.writer()) for c in (*fast, slow)]
    for tick in range(200):
        gateway.publish("ACME", b'{"symbol":"ACME","bid":101.25}')
        await asyncio.sleep(0.001)
    await asyncio.sleep(0.05)
    for w in writers:
        w.cancel()
    print("fast client sent:", fast[0].sent, "| slow client sent:", slow.sent, "dropped:", slow.dropped)


asyncio.run(main())

The publisher never awaits a client: it puts shared bytes into each outbox without blocking and counts drops for clients whose queue is full. All fast clients received every update while the slow client received a few and dropped the rest instead of holding up everyone. Whether to drop, conflate to the latest price, or disconnect a slow client is a product decision; the mechanics are covered in handling WebSocket backpressure with slow consumers.

Verify: fast clients report 200 messages sent, and the slow client reports a large drop count rather than slowing the publisher.

4. Scale CPU with processes, not threads

When one loop's core is not enough, add loops in separate processes: threads within one process would share one GIL on a regular CPython build and add no Python throughput. Each process runs its own event loop, accepts connections from a shared port, and receives updates from a shared bus.

import math
from dataclasses import dataclass


@dataclass(frozen=True)
class GatewaySizing:
    clients: int
    msgs_per_client_s: float
    send_cpu_s: float                  # measured per send, framing + write
    loop_headroom: float = 0.6         # keep each loop at 60% so latency stays flat
    overhead_kib_per_client: float = 40.0    # socket buffers + protocol state, measured

    @property
    def cores_needed(self) -> float:
        return self.clients * self.msgs_per_client_s * self.send_cpu_s

    @property
    def processes(self) -> int:
        return max(1, math.ceil(self.cores_needed / self.loop_headroom))

    @property
    def memory_gib_per_process(self) -> float:
        per_process = self.clients / self.processes
        return per_process * (self.overhead_kib_per_client + 1.13) / 1024 / 1024


sizing = GatewaySizing(clients=50_000, msgs_per_client_s=5, send_cpu_s=8e-6)
print(f"{sizing.cores_needed:.1f} cores -> {sizing.processes} processes, "
      f"{sizing.memory_gib_per_process:.2f} GiB each")

Two cores of send work at 60% headroom per loop means four processes of about half a gigabyte each. The processes share a listening port through SO_REUSEPORT or a load balancer, and every process must receive every relevant update, which means publishing updates through a broker — Redis pub/sub, NATS or Kafka — rather than from one process's memory, as discussed in message brokers and event streams. On a free-threaded build, several loops in one process become an option, with the shared-state caveats that come with it.

Verify: the calculator reports four processes; halving send_cpu_s after optimising the send path halves the process count.

5. Offload connection-time CPU, keep the loop for I/O

Authentication at connect time — verifying a JWT signature, fetching entitlements — is CPU-heavy and bursty: a reconnect storm after a deploy produces thousands per second. Running it on the loop that also pushes updates stalls every client's stream. Bound concurrent handshakes and move signature verification off the loop.

import asyncio
import hashlib
import hmac
import time
from concurrent.futures import ThreadPoolExecutor

SECRET = b"demo-secret"
HANDSHAKES = asyncio.Semaphore(200)                 # shed reconnect storms instead of queuing forever
AUTH_POOL = ThreadPoolExecutor(max_workers=4, thread_name_prefix="auth")


def verify_token(token: bytes) -> bool:
    body, _, signature = token.rpartition(b".")
    digest = hmac.new(SECRET, body, hashlib.sha256).hexdigest().encode()
    for _ in range(2000):                            # stands in for asymmetric signature cost
        hashlib.sha256(digest).digest()
    return hmac.compare_digest(digest, signature)


async def accept(token: bytes) -> bool:
    try:
        async with asyncio.timeout(2.0):
            async with HANDSHAKES:
                loop = asyncio.get_running_loop()
                return await loop.run_in_executor(AUTH_POOL, verify_token, token)
    except TimeoutError:
        return False                                 # client retries with backoff


async def main() -> None:
    body = b'{"sub":"u-1"}'
    token = body + b"." + hmac.new(SECRET, body, hashlib.sha256).hexdigest().encode()
    started = time.perf_counter()
    results = await asyncio.gather(*(accept(token) for _ in range(500)))
    print(f"{sum(results)} handshakes verified in {time.perf_counter() - started:.2f}s")


asyncio.run(main())

A thread pool helps only where the heavy work releases the GIL — hashlib does for large inputs, and many cryptography libraries do for signature checks — otherwise use a process pool, as described in CPU-bound task offloading. The semaphore and timeout turn a reconnect storm into bounded work plus fast rejections, instead of an ever-growing handshake queue that starves the streams.

Verify: all 500 handshakes verify, and a loop-lag probe running alongside stays low while they are processed.

What to add when the gateway runs out? A decision on Which resource is exhausted with 3 outcomes. What to add when the gateway runs out? Which resource is exhausted? memory per host more hosts or leaner per-connection state one loop above 60% CPU more processes shared port + broker connect-time CPU bound + offload auth reject fast when full Threads are not on this list: on a GIL build they add memory, not throughput.

Verification

The gateway architecture is justified when:

  • Memory per connection is measured, and the chosen model fits the target connection count with headroom.
  • CPU per send is measured, and the process count derives from it with a per-loop headroom target.
  • Fan-out never blocks on a client: payloads are encoded once, and slow clients drop, conflate or disconnect.
  • Each process receives all updates through a broker, and connections spread across processes.
  • Connection-time CPU is bounded and offloaded, so reconnect storms do not stall streams.

Pitfalls & edge cases

  • Encoding per client for personalisation. Per-client fields defeat encode-once. Split messages into a shared payload and a small per-client envelope, or group clients by variant.
  • Sticky state in one process. Subscriptions held only in memory disappear when a process restarts; clients must resubscribe on reconnect, and reconnects must be jittered.
  • Unbounded outboxes. An unbounded per-client queue turns one slow client into a memory leak. Bound every outbox.
  • Measuring on loopback only. Real networks make sends slower and more variable; measure send cost under realistic latency before finalising the process count.
  • Ignoring kernel limits. File descriptor limits, ephemeral ports on the proxy, and socket buffer memory cap connections independently of Python; raise and monitor them.

Frequently Asked Questions

Should a Python WebSocket gateway use threads or asyncio?

asyncio for holding connections. In our measurement an idle coroutine with its task used about 1.1 KiB of resident memory versus about 21 KiB for an idle thread, plus the thread's reserved stack. Threads also add no Python throughput on a regular CPython build, so they cost memory without buying CPU.

How many WebSocket connections can one asyncio process handle?

Connection count is usually limited by memory and kernel limits, while throughput is limited by CPU: one event loop executes Python on one core. Measure CPU per send and messages per second; when send work approaches about 60 percent of one core, add processes rather than connections per process.

How do I broadcast to many WebSocket clients efficiently in Python?

Encode each message once and send the same bytes to every subscriber, give each client a bounded outbound queue with its own writer task, and never await individual clients inside the publish loop. Decide explicitly whether slow clients drop messages, receive only the latest value, or are disconnected.

How do multiple gateway processes all receive the same updates?

Publish updates through a shared broker such as Redis pub/sub, NATS or Kafka, and have every gateway process subscribe to the topics its clients need. Connections are spread across processes with SO_REUSEPORT or a load balancer, and clients resubscribe after reconnecting to a different process.