Skip to content

Reading Line-Delimited Protocols with Asyncio Streams

Line-delimited protocols are everywhere in infrastructure: SMTP, Redis' inline commands, IRC, statsd over TCP, printer and PLC control channels, and a wide assortment of in-house services that predate JSON over HTTP. They look trivial to parse — read until a newline, handle the command — and that impression survives exactly until the first production incident, which is usually one of three: a client that sends 200 MB without a newline and takes the process down, a handler that hangs forever on a peer that stopped talking, or a parser that quietly loses the last command because the connection ended without a trailing delimiter.

StreamReader gives you the right tools for all three, provided you use the bounded variants and handle the exceptions they raise. This guide covers reading lines safely, choosing the buffer limit, dealing with LimitOverrunError and IncompleteReadError, enforcing idle deadlines, and handling the CRLF and partial-line cases that protocol specs are vague about.

Prerequisites

Bytes arriving without a delimiter Three lanes showing segments accumulating in the reader buffer until the limit is exceeded. Bytes arriving without a delimiter segments part 1 part 2 part 3 no newline yet reader buffer grows toward limit outcome LimitOverrunError bytes received, left to right The limit is what turns an unbounded peer into a bounded, catchable error.

1. Read one line at a time, bounded

readline() returns everything up to and including the next \n, and it is already bounded by the reader's limit.

import asyncio


async def handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
    while True:
        line = await reader.readline()
        if not line:                       # b"" means EOF, not an empty line
            break                          # (an empty line arrives as b"\n")
        command = line.rstrip(b"\r\n")     # tolerate both LF and CRLF
        writer.write(execute(command) + b"\r\n")
        await writer.drain()

The if not line test is the EOF check and it is easy to get wrong: a blank line sent by the peer is b"\n", which is truthy, while a closed connection yields b"". Stripping \r\n rather than \n means the same handler serves peers that follow the RFC and peers that do not. Verify: connect with nc, send a command followed by Enter, then press Ctrl-D; the handler should process the command and then exit the loop cleanly.

2. Size the buffer limit to your longest legitimate line

The reader buffers up to limit bytes while hunting for the delimiter. Both defaults and extremes cause incidents.

import asyncio

MAX_LINE = 16 * 1024          # the longest command this protocol defines


async def main() -> None:
    server = await asyncio.start_server(
        handle, "0.0.0.0", 9000,
        limit=MAX_LINE,       # StreamReader buffer, per connection
    )
    async with server:
        await server.serve_forever()

The default is 64 KiB, which is generous for a control protocol and far too small for one that ships base64 payloads on a single line. Size it just above the largest line the protocol can legitimately produce: too small and you reject valid traffic, too large and every connected peer can pin that much memory — at 10,000 connections, a 1 MiB limit is 10 GB of exposure. Verify: send a line one byte under the limit (accepted) and one byte over (rejected), and confirm the process memory scales with connections × limit as expected.

3. Handle the overrun instead of crashing

When the delimiter never arrives within limit bytes, readline() raises. Catch it, tell the peer, and close — the alternative is a handler task dying silently and the connection leaking.

import asyncio


async def read_line_safely(reader: asyncio.StreamReader) -> bytes | None:
    try:
        line = await reader.readline()
    except ValueError:
        # LimitOverrunError (a ValueError subclass): no delimiter within `limit`.
        # The bytes are still in the buffer; the only safe move is to drop the peer.
        return None
    except asyncio.IncompleteReadError as exc:
        return exc.partial or None          # stream ended mid-line
    return line or None

The subtlety worth knowing: readline() wraps readuntil() and converts LimitOverrunError into a ValueError while leaving the oversized data in the buffer, so you cannot "skip past" the bad line and continue — attempting to resynchronise just hits the same error. Closing the connection is the correct response. If you must tolerate long lines, use readuntil() directly and drain in chunks. Verify: send limit + 1 bytes with no newline and confirm the server logs a protocol error and closes the connection instead of raising an unhandled exception.

What each read outcome means Five-row matrix of read outcomes and their handling. What each read outcome means condition what to do b'' returned peer closed cleanly exit the read loop b'\n' returned an empty command dispatch it normally line without delimiter EOF mid-line apply the partial-line policy ValueError raised no delimiter within limit protocol error: close IncompleteReadError stream ended mid-read treat as disconnect Five outcomes, five different responses — collapsing them hides real faults.

4. Put a deadline on every read

A peer that connects and says nothing holds a task, a socket, and its buffer indefinitely. Idle timeouts are how you get them back.

import asyncio

IDLE = 60.0            # no command for a minute -> close
BUSY = 10.0            # a line started but never finished


async def serve(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
    try:
        while True:
            try:
                async with asyncio.timeout(IDLE):
                    line = await reader.readline()
            except TimeoutError:
                writer.write(b"421 idle timeout\r\n")
                await writer.drain()
                break
            if not line:
                break
            async with asyncio.timeout(BUSY):
                await dispatch(line, writer)
    finally:
        writer.close()
        await writer.wait_closed()

Two different budgets matter: how long a peer may be silent between commands (generous — minutes for an interactive session) and how long a single command may take to process (tight — seconds). Collapsing them into one number forces you to choose between disconnecting idle-but-healthy sessions and letting a stuck command hold a connection. Verify: open a connection, send nothing, and confirm it is closed after IDLE seconds with the protocol's own error line delivered first.

5. Use readuntil for non-newline delimiters

Many protocols terminate on something other than \n — a null byte, \r\n\r\n, or a sentinel word.

import asyncio


async def read_message(reader: asyncio.StreamReader) -> bytes:
    """Read up to a blank-line terminator, as in header blocks."""
    try:
        block = await reader.readuntil(b"\r\n\r\n")
    except asyncio.LimitOverrunError as exc:
        raise ValueError(f"header block exceeds buffer ({exc.consumed} bytes read)") from exc
    except asyncio.IncompleteReadError as exc:
        raise ConnectionResetError("peer closed mid-header") from exc
    return block[:-4]

Since Python 3.13 readuntil() also accepts a tuple of separators, which is how you support both \n and \r\n terminators without reading byte by byte. Note that readuntil() raises LimitOverrunError directly (unlike readline(), which converts it), so catch the specific exception when you call it yourself. Verify: send a header block split across three TCP segments and confirm it is reassembled into one message.

6. Handle the trailing partial line at EOF

Many clients close without a final delimiter. Whether that last fragment is a valid command is a protocol decision — but silently dropping it is never the right one.

import asyncio


async def drain_stream(reader: asyncio.StreamReader) -> list[bytes]:
    lines: list[bytes] = []
    while True:
        line = await reader.readline()
        if not line:
            break
        if line.endswith(b"\n"):
            lines.append(line.rstrip(b"\r\n"))
        else:
            # EOF reached with data still buffered: an unterminated final line.
            log.warning("unterminated final line (%d bytes) — %s",
                        len(line), "accepting" if ACCEPT_PARTIAL else "discarding")
            if ACCEPT_PARTIAL:
                lines.append(line)
            break
    return lines

readline() returns whatever is buffered when EOF arrives, so a line without a terminator comes back without one — that check is the only way to distinguish "complete line" from "connection ended mid-line". Choose a policy explicitly: strict protocols should reject, log-shipping style protocols usually accept. Verify: send foo\nbar and close without a newline; the parser must report two entries or one plus a warning, according to your policy — never one silently.

The test that proves the parser A pipeline of four boundary test cases for line-delimited parsing. The test that proves the parser one byte per segment worst-case split ten commands in one packet worst-case merge limit + 1 with no newline overrun path close mid-line partial policy A parser that survives all four is boundary-independent.

Verification

  • Boundaries are irrelevant. Sending a command one byte at a time and sending ten commands in a single packet both produce identical parsed output.
  • Overruns are contained. A peer sending limit + 1 bytes without a delimiter gets a protocol error and a closed connection, and process memory returns to baseline.
  • Idle peers are reclaimed. Connection count and open file descriptors return to baseline within IDLE seconds of the last command.

Pitfalls and edge cases

  • Treating b"" and b"\n" alike. Empty bytes mean EOF; a newline means an empty command. Conflating them either drops blank lines or spins forever at EOF.
  • Assuming \n only. Most internet protocols specify CRLF, and many clients send bare LF. Strip both on read and emit CRLF on write.
  • Catching ValueError too broadly. readline() surfaces overruns as ValueError; a blanket handler around your dispatch logic can swallow it and loop forever on the same unreadable buffer.
  • No limit on a public listener. The 64 KiB default is a bound, but on a public port it is a per-connection memory grant; size it deliberately.
  • Parsing in the read loop. Heavy per-line work runs on the loop thread and stalls every other connection; hand the line to a worker for anything expensive.
  • Missing drain() on the response path. A client that never reads your responses fills your send buffer indefinitely — see drain and write backpressure.
  • Unicode decoding per line. Decoding with str.decode() on a boundary-split multi-byte sequence raises; decode after framing, never before.

Frequently Asked Questions

What does LimitOverrunError mean in asyncio streams?

It means the separator was not found within the StreamReader's buffer limit — the peer sent more bytes than limit without terminating the line. readline() re-raises it as a ValueError and leaves the data in the buffer, so you cannot skip the bad line and continue; the correct response is to report a protocol error and close the connection, or to raise the limit if the traffic was legitimate.

How do I set the buffer size for asyncio.StreamReader?

Pass limit= to asyncio.open_connection() or asyncio.start_server(); it defaults to 64 KiB. Size it just above the longest line your protocol can legitimately produce. Remember it is per connection: multiply by your expected connection count to see the memory you are committing.

How do I detect the end of a stream when reading lines?

readline() returns b"" at EOF and b"\n" for an empty line, so test for empty bytes rather than falsiness of the stripped value. If the final line arrives without a terminator, readline() returns it without the trailing newline — check whether the result ends with the delimiter to distinguish a complete line from a truncated one.

Can readuntil match more than one delimiter?

From Python 3.13, readuntil() accepts a tuple of separators and returns as soon as any of them matches, which covers protocols that permit both LF and CRLF. On earlier versions, read with a single delimiter and normalise afterwards, or implement the scan yourself over a bytearray buffer at the protocol layer.