Skip to content

Detecting Leaked Sockets and File Descriptors

A descriptor leak is the quietest failure an async service has. Memory leaks show up in RSS graphs; descriptor leaks show up as OSError: [Errno 24] Too many open files at whatever moment the count crosses the limit, which is usually during peak traffic and never in staging. They are also easy to create by accident: in the measurement below, 50 HTTP clients created and never closed held 100 descriptors — two each — while 50 requests through a single shared client added two in total. One gauge exported from the start turns this from an incident into a chart.

Prerequisites

What a per-request client costs in descriptors 3 bars comparing baseline with the others. What a per-request client costs in descriptors baseline 9 fds 50 clients, never closed 109 fds (+100) 50 requests, one client 11 fds (+2) Each unclosed client held two descriptors; closing them returned the count to 9. Two descriptors per leaked client is enough to exhaust a container limit in minutes.

1. Export the count before you need it

Counting open descriptors is one os.listdir:

def open_fds() -> int:
    return len(os.listdir("/proc/self/fd"))            # Linux; psutil.Process().num_fds() elsewhere


async def sample_resources(interval: float = 30.0) -> None:
    while True:
        OPEN_FDS.set(open_fds())
        await asyncio.sleep(interval)

Export the limit alongside it, because the count means nothing without it:

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
FD_LIMIT.set(soft)

The measured baseline for a small service was 9 descriptors; the limit on that machine was 524,288, while a container is frequently 1,024. Alert at 70% of the soft limit, and treat any sustained upward trend as a leak regardless of the absolute number — a healthy service's descriptor count is flat, not merely below the limit.

Verify: the gauge is stable over hours of steady traffic.

2. Read the table when it grows

/proc/self/fd contains a symlink per descriptor, and the target says what it is:

def fd_breakdown() -> dict[str, int]:
    counts: dict[str, int] = {}
    for fd in os.listdir("/proc/self/fd"):
        try:
            target = os.readlink(f"/proc/self/fd/{fd}")
        except OSError:
            continue                                   # the fd closed under us
        kind = target.split(":")[0] if ":" in target else "file"
        counts[kind] = counts.get(kind, 0) + 1
    return counts

A healthy process in testing produced {'socket': 4, 'file': 3, 'anon_inode': 1}. The categories map directly onto causes:

  • socket:[...] — HTTP clients, database pools, Redis, brokers, listening sockets.
  • file — files opened without a context manager, or log files rotated but not closed.
  • pipe:[...] — subprocess pipes whose parent end was never closed, as in piping data between subprocesses.
  • anon_inode:[eventpoll] — event loops; more than a couple means loops are being created and not closed.

lsof -p <pid> gives the same information with remote addresses attached, which is what identifies which upstream the leaked sockets point at.

Verify: the growing category names the subsystem, before you read any code.

Where descriptors go in an async service A grid of 5 rows by 2 columns. Where descriptors go in an async service consumer leaks when seen as HTTP and DB clients created per request, never closed socket:[...] in /proc/self/fd files opened without a context manager a path in /proc/self/fd subprocess pipes the parent keeps its end pipe:[...] loops and watchers a loop created and not closed anon_inode:[eventpoll] listening sockets servers not closed on reload socket:[...], LISTEN Verified breakdown of a healthy process: 4 sockets, 3 files, 1 anon_inode.

3. Let Python tell you what was not closed

ResourceWarning is emitted when an unclosed socket or file is garbage collected, and it is silent by default:

python -W always::ResourceWarning -X tracemalloc=5 app.py

Which produced, for a deliberately leaked socket:

ResourceWarning: unclosed <socket.socket fd=8, family=2, type=1, proto=0,
                 laddr=('127.0.0.1', 50898), raddr=('127.0.0.1', 8098)>

The raddr names the destination, so you know which client leaked it. Adding -X tracemalloc=5 makes Python include the allocation traceback in the warning — the line that created the socket — which is usually enough to close the investigation immediately.

Route warnings into your logger so they are visible in production rather than only on a terminal:

logging.captureWarnings(True)
warnings.simplefilter("always", ResourceWarning)

In tests, make them fatal: filterwarnings = error::ResourceWarning in your pytest configuration turns a leak into a failing test at the moment it is introduced.

Verify: a deliberately leaked client produces a ResourceWarning in your logs.

Diagnosing a descriptor leak 5 ordered steps. Diagnosing a descriptor leak export the fd count one gauge, cheap correlate with traffic growth per request is the tell read /proc/self/fd group by kind run with -W always::ResourceWarning names the unclosed object fix the ownership one client, closed at shutdown The gauge is what turns a 3 a.m. crash into a graph that was rising for a week.

4. Fix the ownership, not the symptom

Almost every descriptor leak in an async service is the same shape: a client created per request or per call, when it was designed to be created once.

# the leak
async def handler(request):
    client = httpx.AsyncClient()                       # 2 descriptors, never returned
    return await client.get(UPSTREAM)


# the fix
async def handler(request):
    return await request.state.http.get(UPSTREAM)      # one client, from the lifespan

Measured: fifty unclosed clients took the count from 9 to 109, and aclose() on all of them returned it to 9 — the descriptors are released on close, not on the object becoming unreachable, which is why a leak persists as long as anything holds a reference.

The same applies to every pooled resource: database pools, Redis clients, broker connections, gRPC channels. Create in the lifespan, store in state, close in the teardown. Where a client genuinely must be short-lived, async with guarantees the close even on an exception:

async with httpx.AsyncClient() as client:              # closed on every path
    return await client.get(url)

Raising RLIMIT_NOFILE is not a fix. It buys time proportional to the increase, which for a leak growing with traffic is usually hours.

Verify: the descriptor count returns to baseline after a burst of requests.

Client per request, or one client 2 columns contrasting a client per request, one client per process. Client per request, or one client a client per request leaks and slows 2 descriptors each, measured a new TLS handshake per request no connection reuse exhausts the limit under load one client per process the intended design created in the lifespan keep-alive connections reused +2 descriptors for 50 requests closed once, at shutdown The same rule applies to database pools, Redis clients and broker connections.

5. Watch the boundaries where descriptors escape

Three places leak descriptors even when clients are managed correctly.

Subprocess pipes. The parent holds its end of every pipe it creates; a pipe passed to a child must be closed in the parent, or the child never sees EOF and the descriptor stays open.

Cancelled operations. A task cancelled while connecting may leave a socket in a half-open state until it is garbage collected. Cleanup belongs in finally, as in cancellation patterns.

Event loops. asyncio.run() closes its loop; asyncio.new_event_loop() does not unless you call close(). A process that creates a loop per unit of work — a common shape when bridging async code into a synchronous worker — leaks an eventpoll descriptor each time.

loop = asyncio.new_event_loop()
try:
    loop.run_until_complete(main())
finally:
    loop.close()                                       # or the eventpoll fd stays

Verify: anon_inode:[eventpoll] stays at one or two for the process's lifetime.

Verification

Descriptor use is under control when:

  • The count is exported alongside the limit, and alerts at 70%.
  • The count is flat under steady traffic, not merely below the limit.
  • Clients are created once in the lifespan and closed at shutdown.
  • ResourceWarning is visible in logs and fatal in tests.
  • Subprocess pipes are closed in the parent after spawning.
  • Event loops are closed when created manually.

Pitfalls & edge cases

  • Raising the limit as a fix. It converts a crash today into a crash next week.
  • Counting only sockets. Files, pipes and eventpoll descriptors share the same limit.
  • Relying on garbage collection. Descriptors are released on close(); a referenced object never gets collected.
  • ResourceWarning disabled. It is off by default, so the warning that would have caught the leak never appears.
  • Per-request event loops. Each one is an eventpoll descriptor plus its own pools.
  • Container limits. A 1,024 soft limit is common, so the margin is far smaller than on a developer machine.

Frequently Asked Questions

How do I count open file descriptors in a Python process?

On Linux, len(os.listdir("/proc/self/fd")); portably, psutil.Process().num_fds(). Export it as a gauge along with the soft RLIMIT_NOFILE, and alert when it reaches about 70% of the limit or whenever it trends upward under steady traffic.

Why does creating an HTTP client per request leak descriptors?

Because the client owns a connection pool that is released on close, not on the object becoming unreachable. Measured, 50 unclosed httpx clients held 100 descriptors — two each — while 50 requests through a single shared client added two in total.

How do I find which code leaked a socket?

Run with -W always::ResourceWarning and -X tracemalloc=5. Python emits a warning naming the socket, including its remote address, and the tracemalloc option adds the traceback of where it was created. Route warnings into your logger so this works in production too.

What are anon_inode:[eventpoll] descriptors?

Event loop selectors. One or two per process is normal; a growing count means event loops are being created without being closed — typically asyncio.new_event_loop() in a bridge or a test helper, without a matching loop.close().

Should I raise the file descriptor limit?

Raise it if your legitimate concurrency needs more — a server with thousands of connections does. Do not raise it in response to a leak: the count grows with traffic, so a higher limit only moves the crash. Fix the ownership instead.