Skip to content

Publishing with Confirms in aio-pika

RabbitMQ publisher confirms are the broker's acknowledgement that it has taken responsibility for a message. aio-pika enables them by default, and each publish waits for its confirm, which makes a simple publishing loop slow enough that turning confirms off looks attractive. Measured on Python 3.14 with aio-pika 10.1.0 against RabbitMQ 4 in Docker, publishing persistent 256-byte messages to a durable queue: with confirms off, a sequential loop ran at 20,815 messages per second; with confirms on and each publish awaited, 1,571; with confirms on and 100 publishes awaited together, 7,988. Then the broker was killed with SIGKILL mid-stream. Without confirms, the publisher had seen 25,302 publishes return successfully, and 20,480 messages were in the queue after restart — 4,822 lost with no error at the point of publishing. With confirms, 13,772 were confirmed and 13,772 were in the queue: none lost. Separately, a message to a routing key with no queue was not an error at all: publish returned a Basic.Return instead of an Ack. This guide keeps confirms and makes them fast.

Prerequisites

1. Measure the cost of awaiting each confirm

With confirms on, await exchange.publish(...) returns when the broker confirms the message — for a persistent message on a durable queue, after it has been written. A loop that awaits each one pays a round trip and a write per message:

channel = await connection.channel(publisher_confirms=True)     # aio-pika's default
for event in events:
    await channel.default_exchange.publish(
        aio_pika.Message(event, delivery_mode=aio_pika.DeliveryMode.PERSISTENT),
        routing_key="events",
    )

Measured with 20,000 messages: 1,571 per second, against 20,815 with publisher_confirms=False. The thirteen-fold difference is why confirms get turned off. The next steps show what that costs, and how to recover most of the speed without it.

Verify: publishing throughput is measured with confirms on, and the number is compared with the rate the service actually needs.

20,000 persistent messages to a durable queue A grid of 4 rows by 3 columns. 20,000 persistent messages to a durable queue mode messages/s after SIGKILL of the broker confirms off, sequential 20,815 25,302 returned ok, 20,480 survived: 4,822 lost confirms on, await each 1,571 not measured confirms on, 100 in flight 7,988 13,772 confirmed, 13,772 survived: 0 lost confirms on, 1,000 in flight 7,486 not measured aio-pika 10.1.0, RabbitMQ 4, 256-byte persistent messages.

2. See what is lost without confirms

Without confirms, publish returns as soon as the frame is written to the client's socket. Nothing tells the publisher whether the broker received it, let alone stored it. The crash test published batches of 100 persistent messages in a loop, and the broker was killed with docker kill -s KILL 1.5 seconds in:

ok = 0
for start in range(0, 200_000, 100):
    results = await asyncio.gather(*(publish_one() for _ in range(100)), return_exceptions=True)
    for r in results:
        if isinstance(r, BaseException):
            raise r
        ok += 1                                  # what the publisher believes was sent

Without confirms, 25,302 publishes had returned without error when the connection failed, and after restart the durable queue held 20,480 — 4,822 messages, 19% of what the publisher believed it had sent, were gone, and nothing identified which ones. With confirms, the publisher counted 13,772 confirmed messages and the queue held exactly 13,772; every unconfirmed publish raised, so the publisher knew precisely which messages to resend. Persistent messages and durable queues are necessary for surviving a restart, but without confirms there is no way to know what was persisted.

Verify: a crash test — kill the broker during publishing, restart it, compare counts — shows the publisher's confirmed count equals the stored count.

3. Keep many confirms in flight

Confirms are asynchronous on the wire: the broker can confirm many messages while the publisher keeps sending. Awaiting a group of publishes together keeps the channel busy:

async def publish_batch(exchange, bodies: list[bytes], routing_key: str):
    results = await asyncio.gather(*(
        exchange.publish(aio_pika.Message(b, delivery_mode=aio_pika.DeliveryMode.PERSISTENT),
                         routing_key=routing_key)
        for b in bodies
    ), return_exceptions=True)
    failed = [b for b, r in zip(bodies, results) if isinstance(r, BaseException)]
    return failed                                  # resend these; everything else is confirmed

Measured: 100 publishes in flight reached 7,988 messages per second — five times the one-at-a-time rate and 38% of the unconfirmed rate — and 1,000 in flight reached 7,486, no better. A few hundred in flight is enough; more only holds more memory in the client. Every message is still individually confirmed, so the failed list is exact.

Verify: the publisher keeps a bounded number of publishes in flight, and its throughput with confirms meets the required rate.

Publishing throughput with and without confirms 4 horizontal bars comparing confirms off (19% lost on crash) with the others. Publishing throughput with and without confirms confirms off (19% lost on crash) 20,815/s confirms on, 100 in flight 7,988/s confirms on, 1,000 in flight 7,486/s confirms on, await each 1,571/s Batching recovered most of the speed and kept every guarantee.

4. Check for returned messages

A confirm says the broker handled the message; it does not say a queue received it. When no queue matches the routing key, the broker drops the message — or, if it was published with mandatory=True, returns it. aio-pika defaults to mandatory=True, and measured: publishing to a routing key with no queue raised nothing. publish returned an aio_pika.message.DeliveredMessage wrapping a Basic.Return, where a routed message returned a pamqp.commands.Basic.Ack:

from pamqp.commands import Basic

result = await exchange.publish(message, routing_key=key)
if not isinstance(result, Basic.Ack):
    raise UnroutableMessage(f"{key!r}: broker returned the message")

Code that only catches exceptions treats a returned message as delivered. Typical causes are a typo in a routing key, a queue that has not been declared yet at start-up, or a binding removed by a deployment. Declaring queues and bindings from the publisher at start-up, or alerting on returns, closes the gap.

Verify: a publish to an unbound routing key is detected as a failure by the publishing code, in a test.

5. Resend unconfirmed messages safely

When the connection fails, publishes that were not confirmed raise. They may or may not have reached the queue — the broker might have stored a message and died before sending the confirm — so resending can create duplicates. Make that safe with a message identifier that consumers deduplicate on:

message = aio_pika.Message(
    body,
    message_id=event_id,                             # stable across retries
    delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
)

Publishers resend what was not confirmed; consumers skip message_ids they have already processed. For events that originate in a database transaction, writing them to an outbox table in the same transaction and publishing from there gives the same guarantee across process crashes; see implementing the transactional outbox pattern in asyncio. connect_robust reconnects channels automatically, but it does not resend messages whose publish raised — that remains the publisher's job.

Verify: after a broker restart during publishing, every event appears in the queue at least once, and consumers process each message_id once.

A confirmed publish and an unroutable one A sequence of 5 messages between 3 participants. A confirmed publish and an unroutable one publisher exchange durable queue publish, persistent, message_id route + write to disk Basic.Ack -> publish returns Ack publish to an unbound key Basic.Return -> DeliveredMessage, no exception Check the result type, not just for exceptions.

Verification

Publishing is reliable when:

  • Publisher confirms are on, and throughput comes from keeping a bounded number of publishes in flight.
  • A crash test shows confirmed counts matching stored counts.
  • Returned messages are detected by checking for Basic.Ack, not only for exceptions.
  • Unconfirmed messages are resent with stable message_ids that consumers deduplicate.

Diagnostic Hook: when downstream counts fall short of a publisher's log of sent messages after a broker restart, check publisher_confirms on the publishing channel. Without confirms, 4,822 of 25,302 apparently successful publishes were missing after a hard kill in this test.

Pitfalls & edge cases

  • Turning confirms off for speed. Measured: 19% of messages lost in a crash, silently.
  • Awaiting each confirm. Measured: 1,571 msg/s against 7,988 with 100 in flight.
  • Treating a return value as success. An unroutable message returned Basic.Return.
  • Resending without message ids. Unconfirmed is not the same as undelivered.

Frequently Asked Questions

Are publisher confirms enabled by default in aio-pika?

Yes: connection.channel() uses publisher_confirms=True, and publish waits for the broker's Ack. Awaiting each publish in turn gave 1,571 messages per second.

How do I publish quickly with confirms in aio-pika?

Keep many publishes in flight with asyncio.gather and collect the failures. 100 in flight reached 7,988 msg/s with every message still confirmed.

Do I lose messages without publisher confirms?

You can, without knowing it. After a hard broker kill, 4,822 of 25,302 publishes that returned successfully were missing; with confirms, none were.

Why does publishing to a missing queue not raise in aio-pika?

With the default mandatory=True, the broker returns the message and publish returns a DeliveredMessage wrapping Basic.Return. Check that the result is a Basic.Ack.