Skip to content

Tuning TCP Keepalive for Long-Lived Async Connections

A worker holds a database connection, a message-broker session and a pool of upstream HTTP connections, all mostly idle between bursts. After a network hiccup — a NAT table entry expiring, a load balancer recycling, a firewall dropping idle flows — the sockets are dead, but nobody has told either side. The next query hangs until some other timeout fires, and if none does, forever. TCP has a mechanism for exactly this, disabled by default in most stacks: keepalive probes. When it is enabled, Linux's defaults wait two hours before the first probe, which is far longer than any NAT idle timeout and useless for detecting a dead peer in a service. This guide enables and tunes keepalive on asyncio connections, computes how long detection actually takes, adds TCP_USER_TIMEOUT for connections with unacknowledged data, and explains when only an application-level heartbeat will do.

Prerequisites

An idle connection that quietly died 4 lanes over time. An idle connection that quietly died traffic requests idle NAT entry ageing out gone: connection dead default keepalive first probe in 2 hours tuned 30/10/3 probes declared dead time → The probe also refreshes the NAT entry, so tuning prevents as well as detects.

1. See the defaults you are inheriting

Keepalive is off per socket by default, and the system defaults behind it are measured in hours.

import asyncio
import socket


async def main() -> None:
    async def handle(reader, writer):
        await reader.read(1)
        writer.close()

    server = await asyncio.start_server(handle, "127.0.0.1", 0)
    port = server.sockets[0].getsockname()[1]
    reader, writer = await asyncio.open_connection("127.0.0.1", port)
    sock = writer.get_extra_info("socket")                     # the real socket object

    print("SO_KEEPALIVE:", sock.getsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE))
    print("TCP_KEEPIDLE:", sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE), "s")
    with open("/proc/sys/net/ipv4/tcp_keepalive_time") as f:
        print("system tcp_keepalive_time:", f.read().strip(), "s")

    writer.close()
    server.close()
    await server.wait_closed()


asyncio.run(main())

On this machine the socket reported SO_KEEPALIVE: 0 — probes disabled — with a system default idle of 7,200 seconds. Even after enabling keepalive, the stock settings would send the first probe two hours into an idle period, by which time a NAT device with a five-minute idle timeout has long since forgotten the flow.

Verify: your socket reports keepalive disabled, and the system idle time is in the thousands of seconds.

2. Enable and tune keepalive on asyncio sockets

writer.get_extra_info("socket") gives the underlying socket for a stream connection; servers can tune each accepted socket the same way. Set the four options together — enabling keepalive without tuning the timers inherits the two-hour default.

import socket


def tune_keepalive(sock: socket.socket, *, idle: int = 30, interval: int = 10,
                   count: int = 3, user_timeout_ms: int | None = 60_000) -> None:
    """Detect a dead peer in roughly idle + interval * count seconds."""
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
    for name, value in (("TCP_KEEPIDLE", idle),        # seconds idle before the first probe
                        ("TCP_KEEPINTVL", interval),   # seconds between probes
                        ("TCP_KEEPCNT", count)):       # probes before giving up
        option = getattr(socket, name, None)
        if option is not None:                         # macOS/Windows: see the pitfalls
            sock.setsockopt(socket.IPPROTO_TCP, option, value)
    if user_timeout_ms is not None and hasattr(socket, "TCP_USER_TIMEOUT"):
        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT, user_timeout_ms)


async def connect_tuned(host: str, port: int, **kwargs):
    reader, writer = await asyncio.open_connection(host, port, **kwargs)
    tune_keepalive(writer.get_extra_info("socket"))
    return reader, writer

Reading the options back after setting them showed 1, 30, 10, 3 and a user timeout of 60,000 ms, confirming they took effect. For servers, apply the same function inside the connection callback; for client libraries that own their sockets, look for a socket-options hook — most async drivers expose one, and several accept a list of (level, option, value) tuples directly.

Verify: getsockopt returns the values you set, and the connection still carries traffic normally.

3. Compute the detection time you are buying

Keepalive detection is arithmetic: the first probe goes out after idle seconds of silence, then up to count probes spaced interval apart. Choose values from how quickly the service must notice, and from the shortest idle timeout on the path.

def detection_seconds(idle: int, interval: int, count: int) -> int:
    return idle + interval * count


for name, (idle, interval, count) in {
    "Linux defaults": (7200, 75, 9),
    "service default": (30, 10, 3),
    "aggressive": (10, 5, 3),
}.items():
    print(f"{name:>16}: idle={idle}s intvl={interval}s cnt={count} -> "
          f"dead peer detected after ~{detection_seconds(idle, interval, count)}s")

The defaults detect a dead peer after about 7,875 seconds — more than two hours. A 30/10/3 configuration detects it in 60 seconds, and 10/5/3 in 25. Two constraints bound the choice from below: probes cost a packet per connection per idle period, which matters with tens of thousands of connections, and an idle time shorter than the network's round trip plus jitter produces false positives on slow links. Keep idle comfortably below the shortest idle timeout on the path — typically the NAT or load-balancer timeout — so the probe refreshes the flow before it is dropped.

Verify: the computed detection time is shorter than the idle timeout of every intermediary the connection crosses.

How long until a dead peer is detected 3 bars comparing Linux defaults 7200/75/9 with the others. How long until a dead peer is detected Linux defaults 7200/75/9 ~7,875 s service default 30/10/3 60 s aggressive 10/5/3 25 s Computed as idle + interval x count; the defaults were read from this machine. Pick a detection time below the shortest idle timeout on the network path.

4. Add TCP_USER_TIMEOUT for connections with data in flight

Keepalive only helps an idle connection. When data has been sent and not acknowledged, the kernel retransmits, and the default retransmission schedule can take a quarter of an hour before reporting the failure. TCP_USER_TIMEOUT bounds that: it is the maximum time transmitted data may remain unacknowledged before the connection is declared dead.

import asyncio
import socket


async def demonstrate_options() -> None:
    server = await asyncio.start_server(lambda r, w: None, "127.0.0.1", 0)
    port = server.sockets[0].getsockname()[1]
    reader, writer = await asyncio.open_connection("127.0.0.1", port)
    sock = writer.get_extra_info("socket")

    tune_keepalive(sock, idle=30, interval=10, count=3, user_timeout_ms=60_000)
    print({
        "SO_KEEPALIVE": sock.getsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE),
        "TCP_KEEPIDLE": sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE),
        "TCP_KEEPINTVL": sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL),
        "TCP_KEEPCNT": sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT),
        "TCP_USER_TIMEOUT_ms": sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT),
    })
    writer.close()
    server.close()
    await server.wait_closed()


asyncio.run(demonstrate_options())

Set TCP_USER_TIMEOUT to roughly the keepalive detection time so both paths — idle and in-flight — fail in the same order of magnitude. Note the interaction: when a user timeout is set, it also overrides how long keepalive probing may continue, so configuring it far below idle + interval × count makes the probe count irrelevant.

Verify: all five options read back as configured on a live connection.

5. Know when only an application heartbeat works

TCP keepalive proves that a socket is alive between two kernels. It cannot prove that the peer application is healthy, and it never crosses a proxy that terminates the connection.

import asyncio
import time


async def application_heartbeat(send, receive_pong, interval: float = 15.0,
                                timeout: float = 5.0) -> None:
    """Prove the peer application, not just its kernel, is still answering."""
    while True:
        await asyncio.sleep(interval)
        started = time.perf_counter()
        await send({"type": "ping", "ts": started})
        try:
            async with asyncio.timeout(timeout):
                await receive_pong()
        except TimeoutError:
            raise ConnectionError(f"no pong within {timeout}s; peer application is unresponsive")

Use keepalive for the transport and a heartbeat for the application, with the heartbeat interval shorter than the shortest proxy idle timeout so it doubles as traffic that keeps intermediaries from dropping the flow. A hung peer — one whose process is alive but whose event loop is blocked — answers TCP probes from its kernel and never answers a ping, which is exactly the failure the application heartbeat exists to catch, as measured in measuring event loop lag in production.

Verify: blocking the peer's event loop for longer than the timeout triggers the heartbeat error while TCP keepalive stays satisfied.

What each liveness mechanism proves A grid of 4 rows by 2 columns. What each liveness mechanism proves mechanism detects blind to TCP keepalive dead peer or path while idle a hung application TCP_USER_TIMEOUT unacknowledged data in flight an idle dead path application heartbeat peer application not answering nothing at its layer pool recycling connections closed cleanly elsewhere mid-use failures Long-lived connections usually need the first three together.

Verification

Keepalive is configured correctly when:

  • It is enabled explicitly: every long-lived socket reports SO_KEEPALIVE: 1, verified by reading the option back.
  • Timers are tuned, not inherited: idle, interval and count give a detection time that matches the service's requirements.
  • Idle time fits the path: it is shorter than the shortest NAT, firewall or load-balancer idle timeout in front of the connection.
  • In-flight failures are bounded: TCP_USER_TIMEOUT is set in the same range as the keepalive detection time.
  • Application health is checked separately: a heartbeat with its own timeout detects a peer whose kernel answers but whose application does not.

Pitfalls & edge cases

  • Platform differences. macOS uses TCP_KEEPALIVE for the idle time and has no TCP_USER_TIMEOUT; Windows uses SIO_KEEPALIVE_VALS. Guard each option with hasattr, as in the helper above.
  • Tuning the wrong socket. get_extra_info("socket") on a TLS connection returns the underlying socket, which is what you want; make sure you are not tuning a listening socket and expecting accepted ones to inherit every option.
  • Very aggressive settings at scale. Probes are per connection: 50,000 connections with a 10-second idle send a lot of packets. Keep probing rare relative to the connection count.
  • False positives on mobile or satellite links. High latency and jitter can exceed a short interval; use a longer interval with a higher count instead of a very short idle.
  • Expecting keepalive to replace pool recycling. Pools should still bound connection lifetime and validate connections, because a proxy can close a connection cleanly without either side noticing until the next use.

Frequently Asked Questions

How do I enable TCP keepalive on an asyncio connection?

Get the socket from the stream writer with writer.get_extra_info("socket"), then call setsockopt to set SO_KEEPALIVE, and on Linux also TCP_KEEPIDLE, TCP_KEEPINTVL and TCP_KEEPCNT. Servers can do the same for each accepted connection inside the connection handler.

What are good TCP keepalive settings for a service?

Choose them from how fast you must detect a dead peer and from the shortest idle timeout on the path. An idle of 30 seconds, an interval of 10 and a count of 3 detects a dead peer in about 60 seconds and keeps NAT entries alive, compared with roughly two hours using Linux defaults.

What is TCP_USER_TIMEOUT and how is it different from keepalive?

Keepalive probes an idle connection, while TCP_USER_TIMEOUT bounds how long already-transmitted data may stay unacknowledged before the connection is aborted. Without it, retransmissions can continue for many minutes. Set it in the same range as the keepalive detection time so both failure paths are bounded.

Do I still need application-level heartbeats if TCP keepalive is enabled?

Yes, when the peer application's health matters. Keepalive is answered by the peer's kernel, so a process whose event loop is blocked still looks alive. An application ping with its own timeout detects that, and also works across proxies that terminate TCP connections.