Part 6 · Async & Event-Driven Systems

At-Most-Once, At-Least-Once, Exactly-Once

Acknowledge too early and a crash loses the payment; acknowledge too late and it charges twice. At-most-once, at-least-once and why true exactly-once is impossible — plus idempotent consumers, Kafka's exactly-once and the transactional outbox.

Lesson 378 min readintermediate#messaging#kafka

The Problem

A queue is a line of messages waiting to be handled. A worker is a program that takes messages from the line and does the work. Here a worker reads the message "charge customer 42 ₹2,499". It calls the payment provider, and the charge works. Then, before the worker tells the queue "done", the server crashes.

The queue never got an acknowledgement (ACK, a short "I finished this one" reply from the worker). So it redelivers (sends again) the message to another worker. The customer is charged twice.

Now do it the other way. The worker tells the queue "done" first, then crashes before it charges. The message is gone, so the customer is never charged, and you give the product away for free.

Every messaging system makes you choose how it may fail: it can lose messages or it can duplicate them. Delivery semantics means the rules for how many times a message is handled. Knowing them tells you which failure you have, and how to still get a correct result.

Words used in this lesson. A producer is a program that sends messages. A broker is the server that stores and passes on messages (Kafka, SQS and RabbitMQ are brokers). A consumer is a program that reads messages. A side effect is a change in the outside world, such as a card charge or an email. An offset is a bookmark: the position of the last message a consumer has finished in a Kafka log.

The Core Idea

Imagine sending an important letter.

At-most-once means a message is handled zero or one time. It is like dropping a postcard in a post box. It goes once. If it is lost, it is lost. You never send a second copy.
At-least-once means a message is handled one or more times. It is like sending a registered letter and sending it again until you get a signed receipt. It will surely arrive. But if the receipt gets lost, your friend may get two copies.
Exactly-once means a message is handled one time, no more and no less. This is what you want. Over an unreliable postal service, you cannot truly promise it. What you can do is send the letter at-least-once and have your friend throw away any duplicates (each letter has a unique reference number). The effect is then "exactly once".

How It Works

Where messages can be lost or duplicated

A hop is one step a message takes from one program to the next. Every hop has a "did it arrive?" problem:

Producer
Broker
Consumer
Side effect
DB write, payment, email

Every hop has its own “did it arrive?” problem. · flow

Producer → broker: the send times out. Did the broker save the message? If the producer tries again, there may be a duplicate.
Broker → consumer: the consumer gets the message but crashes. Should the broker send it again?
Consumer → side effect: the effect happened, but the acknowledgement did not.

The key is the order of two steps: "process" and "acknowledge / commit offset". To commit an offset means to save the bookmark, which tells the broker "I am done up to here".

At-most-once

Acknowledge first, then process.

Receive
the message arrives
ACK / commit offset
the broker now thinks it's done
Crash here
the message is LOST — never processed
Process
only if we got this far

At-most-once — acknowledge first, then process. · timeline

✅ Never processed twice. Simple and fast.
❌ Messages can be lost.
Use for: data where a rare loss is fine and duplicates would hurt or be pointless. Examples are some metrics samples, "user is typing" signals, or live location pings (a newer one comes soon anyway).

At-least-once

Process first, then acknowledge.

Receive
the message arrives
Process
charge, write, send
Crash here
no ACK → redelivered → processed AGAIN
ACK / commit offset
only after the work is done

At-least-once — process first, then acknowledge. The practical default. · timeline

✅ Nothing is lost. Every message is processed at least once.
❌ Duplicates are possible.
This is the practical default in SQS, RabbitMQ (with manual acks), Kafka (committing after processing), Google Pub/Sub, and most systems.

Also, producers retry. So duplicates can enter at the first hop too, unless the broker removes them. Kafka's idempotent producer does this. (Idempotent means doing the same thing many times gives the same result as doing it once. Think of pressing a lift button twice: the lift still comes once.)

Exactly-once: why true end-to-end is impossible in general

This is the classic Two Generals problem. Two sides talk over a channel that can lose messages. Neither side can ever be fully sure that the other got the last message, because the confirmation can be lost too. So a system cannot promise, in general, that an external side effect (charging a card, sending an email) happens exactly once by messaging alone.

What you can do:

Exactly-once effect = at-least-once delivery + idempotent (or de-duplicated) processing.

People often call this "effectively once". Idempotent means that doing an action many times has the same result as doing it once. De-duplicated means that repeated copies of a message are found and dropped.

Making consumers idempotent

1. De-duplicate using message IDs. Every message carries a unique ID. The consumer saves each processed ID in the same database transaction as the work. A transaction is a group of database steps that either all succeed or all fail. A unique constraint is a database rule that refuses a second row with the same value.

SQL

BEGIN;  INSERT INTO processed_messages (message_id) VALUES ('msg-8f2a…');  -- UNIQUE constraint  -- if this fails with "duplicate key" → already processed → ROLLBACK and just ACK  UPDATE accounts SET balance = balance - 2499 WHERE user_id = 42;COMMIT;-- then ACK the message

The dedup record and the work are saved together. So after a crash you keep both or neither. A redelivered message hits the unique constraint and is skipped safely.

2. Make the operation idempotent by its own nature.

Use "set" instead of "add": status = 'shipped' instead of count = count + 1.
Use upserts. An upsert means "insert, or update if it already exists": INSERT ... ON CONFLICT (order_id) DO NOTHING.
Use conditional updates, which change a row only if it is in the expected state: UPDATE orders SET status='paid' WHERE id=5521 AND status='pending'.

3. Pass idempotency keys to external systems. An idempotency key is a unique label you send with a request, so the receiver can spot a repeat. When you charge a card or call another API, send a key made from the message ID (post 34). If the message is processed twice, the payment provider sees the same key and does not charge twice.

4. Use dedup windows in the broker. Some brokers remove duplicates for you within a time window. For example, SQS FIFO queues drop a message with the same deduplication ID if it arrives within 5 minutes of the first. That helps with producer retries. It does not replace idempotent consumers.

Kafka's exactly-once semantics (EOS)

Kafka is a broker that keeps messages in an ordered log. A topic is a named log, and a partition is one slice of a topic. Kafka offers exactly-once processing for pipelines that read from Kafka, process, and write back to Kafka:

The idempotent producer means retries do not create duplicates in a partition. Each producer has an ID, and each message has a sequence number. The broker drops a repeat.
Transactions let a program that is both consumer and producer do these two steps atomically (all or nothing):
1. write output messages to one or more topics, and
2. commit its input offsets,
as one unit. Either both happen or neither does.
Consumers that read with isolation.level=read_committed only see messages from committed transactions. They do not see messages from open or cancelled ones.

Diagram

[input topic] ──► app reads, transforms ──► [output topic]
                     └── commit offsets + outputs in ONE Kafka transaction

Kafka Streams and Flink (tools that process streams of events) use this to give exactly-once results inside their pipelines. But when you write to an external system (a database, an email service, a payment API), you need idempotency on that side again.

The dual-write problem and the transactional outbox

A very common bug: your service must update its database AND publish an event. These are two writes to two systems, so this is called a dual write.

step 1 / 4

Order service

Database

Kafka

UPDATE orders SET status = 'paid'
✅ committed
publish order_paid
💥 crash, or Kafka unreachable
Order is paid in the database, but billing, shipping and email never hear about it

The dual-write problem. The database and the event stream can end up disagreeing. · sequence · step through

Or do it in the other order: the event is sent, but the database update fails. Again the database and the event stream disagree. You cannot put a database and Kafka inside one normal transaction.

The fix is the transactional outbox pattern. An outbox is a table in your own database that holds events waiting to be sent, like the "out" tray on a desk:

1
BEGIN
one local transaction in the service's own database
2
UPDATE orders SET status = 'paid'
the state change
3
INSERT INTO outbox (id, topic, payload)
the event, as a row
4
COMMIT
both saved atomically — or neither
5
Relay / CDC (Debezium)
reads new outbox rows and publishes them to Kafka
6
Consumers
may see an event twice — deduplicate by event ID

The transactional outbox. State change and “an event must be sent” commit together. · flow

The state change and "an event must be sent" are saved atomically.
The relay publishes at-least-once (it might publish twice after a crash). So consumers must still be idempotent, using the event ID.
CDC (change data capture) means reading a database's change log to find new or changed rows. Debezium is a tool that does this.

This pattern is one of the most important tools in event-driven systems (post 39).

Ordering and retries

Retries can also change the order of messages. Say message 1 fails and is retried after message 2 succeeds. Then consumers see 2 before 1. Protect against this with:

per-key ordering (messages with the same key go to the same partition or message group, post 35),
version numbers or timestamps in events, so consumers ignore older updates ("I already have version 7, ignore version 6").

Summary

Semantics	How	Risk	Typical use
At-most-once	Ack before processing	Loss	Metrics samples, presence, telemetry
At-least-once	Ack after processing	Duplicates	Default for most work
Effectively-once	At-least-once + idempotent/dedup processing	Needs careful design	Payments, orders, inventory, billing
Kafka EOS	Idempotent producer + transactions + read_committed	Only within Kafka pipelines	Stream processing

Trade-offs

At-most-once: simple and fast, but loses data.
At-least-once: no loss, but you must handle duplicates.
Exactly-once machinery (dedup tables, transactions, outbox): correct results, but more storage, more latency, more code and more things to monitor.
Kafka transactions: strong guarantees inside Kafka, but extra overhead, and they don't cover external side effects.

Common mistake: trusting a vendor's "exactly-once" label and skipping idempotency. Always ask: "what happens if this message is processed twice?"

In the Real World

Payment systems. Payment providers and fintech companies use at-least-once messaging together with idempotency keys and unique transaction references. So a retried message can never cause a second debit. This is the same idea as in post 34, used on queues.

Kafka's exactly-once work. Confluent, the company started by Kafka's creators, published a well-known article. It explains how idempotent producers and transactions give Kafka exactly-once semantics for stream processing. It is clear about the limit: exactly-once inside Kafka, and idempotency needed outside it.

Transactional outbox in microservices. Many companies use the outbox pattern with CDC tools like Debezium. The site microservices.io recommends it, because it solves the dual-write problem without distributed transactions (one transaction spread over many systems).

SQS and duplicate emails. Teams that use standard SQS queues soon learn that a message can sometimes arrive more than once. Duplicate welcome emails are a classic sign. The fix is to save "email sent for user X" before or together with the send, and to skip the send if it is already saved.

Interview Questions

Q1Explain at-most-once, at-least-once and exactly-once delivery.

At-most-once acknowledges before processing, so a crash loses the message. At-least-once processes before acknowledging, so a crash causes redelivery and duplicates. Exactly-once end to end isn't achievable in general over unreliable networks; you get an exactly-once effect by combining at-least-once delivery with idempotent or deduplicated processing.

Q2How do you make a message consumer idempotent?

Record the message ID in a processed-messages table with a unique constraint in the same transaction as the work, so a redelivery fails the insert and is skipped. Or make operations naturally idempotent (set instead of increment, upserts, conditional updates) and pass idempotency keys to external APIs.

Q3What does Kafka's exactly-once semantics actually cover?

Read-process-write pipelines that stay inside Kafka: the idempotent producer removes retry duplicates, and transactions commit output messages and input offsets atomically, with consumers reading read_committed. Side effects in external systems still need their own idempotency.

Q4What is the dual-write problem, and how does the outbox pattern solve it?

Updating a database and publishing an event are two separate systems, so a crash between them leaves them inconsistent. The outbox writes the event as a row in the same local transaction as the state change; a relay or CDC then publishes outbox rows at-least-once.

Q5How do you handle out-of-order redeliveries?

Keep per-key ordering through partitions or message groups, and put a version or sequence number in events so consumers ignore anything older than what they've already applied.

Key Takeaways

Messages can be lost or duplicated at every hop. The order of process vs acknowledge decides which.
At-most-once (ack first) risks loss. At-least-once (process first) risks duplicates, and it's the practical default.
True end-to-end exactly-once is impossible in general. Aim for effectively-once: at-least-once plus idempotent or de-duplicated consumers.
Use dedup tables in the same transaction, natural idempotency (upserts, conditional updates) and idempotency keys for external calls.
Solve the dual-write problem with the transactional outbox. Kafka's EOS works within Kafka pipelines only.

Keep learning

Related series