Exactly-Once Delivery Does Not Exist

Queue and stream products advertise exactly-once semantics, and the phrase means something narrower than it sounds. Understanding the distinction determines whether your consumers are correct or merely lucky.

For a separate people-operations application of the same measurement discipline, see how to identify mouse jigglers among a remote team.

Why delivery cannot be exactly once

A broker sends a message to a consumer. The consumer processes it and sends an acknowledgement. The acknowledgement is lost.

The broker cannot distinguish a lost acknowledgement from a consumer that died before processing. It has two options: redeliver, risking duplicate processing; or not redeliver, risking loss. There is no third option, and this is not an engineering limitation to be solved — it is the same impossibility as the client retry problem. See every request will be retried.

So every message system chooses one:

At-most-once. Acknowledge before processing. Messages can be lost. Fine for metrics samples, wrong for orders.

At-least-once. Acknowledge after processing. Messages can be duplicated. The default nearly everywhere, and the right default.

"Exactly-once." Not a delivery guarantee. What is actually provided is described below.

What the products actually provide

Broker-side deduplication. The producer attaches a sequence number or identifier; the broker discards duplicates it recognises within a retention window. This makes publishing effectively idempotent — a producer retry does not create a second message. It says nothing about consumption.

Transactional read-process-write. Where the consumer's output goes back into the same system, the offset commit and the output write can be made atomic. Kafka's transactions do this: consume from topic A, produce to topic B, commit the offset — all or nothing.

This is exactly-once processing within one system's boundary. It is genuinely useful and it is precisely limited: the moment your side effect leaves that system — a database write, an HTTP call, an email — the atomicity is gone, because the two systems cannot participate in one commit.

The honest framing: exactly-once processing is achievable within a transactional boundary. Exactly-once delivery across a network is not.

The consumer-side patterns that work

Since duplicates will arrive, the consumer must make them harmless. Three approaches, in order of preference.

Idempotent operations

The best answer, because it requires no extra state.

UPDATE order SET status = 'shipped' WHERE id = $1 applied twice has the same effect as once. UPDATE counter SET n = n + 1 does not.

Where the operation is naturally idempotent, redelivery is a non-event and you can stop here.

Deduplication by message identifier

Record processed message IDs; skip ones already seen.

The atomicity requirement is the whole thing. Recording the ID and performing the work must commit together:

BEGIN;
  INSERT INTO processed_messages (id) VALUES ($1);  -- unique constraint
  -- ... the work ...
COMMIT;

A unique violation on the insert means it was already processed; roll back and acknowledge.

Doing this in two steps reintroduces the bug you were fixing. If the work commits and the ID record does not, a redelivery does it again.

Retention matters. The ID table cannot grow forever, and the retention window must exceed the broker's maximum redelivery delay. A message redelivered after your cleanup is processed twice.

Transactional outbox

For the common case where processing a message produces an outbound side effect.

Write the business change and a row describing the intended side effect in the same transaction. A separate process reads the outbox and performs the side effect, retrying until it succeeds.

This does not eliminate duplicates — the outbox process can perform the call and crash before marking it done. It converts the problem into one the receiver can solve with idempotency, which is why external APIs ask for an idempotency key.

What it does guarantee: the intent is durable, and it will happen at least once. That is the achievable version.

Ordering, which is a separate problem

Frequently conflated with delivery guarantees and independent of them.

Ordering is usually per-partition, not global. Kafka guarantees order within a partition. Across partitions there is no order, and the parallelism you want is exactly what removes it.

Consequences:

Related messages must share a partition key. All events for one order go to one partition, or they can be processed out of order.

Parallel consumers within a partition break order. Processing messages from one partition concurrently means the guarantee is gone. If you need both throughput and order, partition more finely rather than parallelising within a partition.

Redelivery breaks order too. A failed message that is retried later arrives after messages that came behind it. If order matters, a failure must block the partition — which is a real availability cost and needs deciding deliberately.

Design for out-of-order where you can. A consumer that ignores an update with an older version number than it already holds is robust to reordering without needing the broker to guarantee anything.

Poison messages

A message that always fails will be redelivered forever, blocking the partition behind it if order is preserved.

Limit attempts and route failures to a dead letter queue.

Then actually monitor the dead letter queue. An unmonitored one is a place where data goes to be forgotten, and it fills up silently for months.

Distinguish permanent from transient failures. A malformed message will never succeed and should go to the dead letter queue on the first attempt. A downstream timeout should be retried. Treating them identically means either wasting attempts on hopeless messages or discarding recoverable ones.

Checking your consumers

Deliver every message twice in tests, and assert the second is a no-op. This is the equivalent of the double-request test for APIs and it finds most of these bugs.

Deliver out of order and see what happens.

Kill the consumer between processing and acknowledgement and confirm the redelivery is harmless. Awkward to arrange and the case most likely to be broken.

Confirm the deduplication window exceeds the maximum redelivery delay, including after a long outage when a large backlog is redelivered.

The summary

At-least-once is what you get, whatever the marketing says.

"Exactly-once" means exactly-once processing inside one transactional boundary — real, useful, and not a delivery guarantee.

Consumers must be idempotent, either naturally or through deduplication committed atomically with the work.

Ordering is per-partition and easy to lose through parallelism or retries.

Design so a duplicate is boring, and the guarantee you did not get stops mattering.