Queues vs Pub/Sub

Point-to-point queues

                          ┌──► Worker 1  (takes msg A)
Producer ──► [ Queue ] ───┼──► Worker 2  (takes msg B)
                          └──► Worker 3  (takes msg C)

• Each message is processed by exactly one of the consumers. This is the competing consumers pattern (lesson 18). A consumer is a program that reads messages. Here the consumers compete for the next message.
• Adding workers increases throughput (the amount of work done per second).
• When a consumer acknowledges a message (tells the queue "I finished it"), the message is removed from the queue.

Use it when a piece of work should be done once: resize this image, send this email, charge this invoice, generate this report.

Publish/subscribe

                               ┌──► Email service
Publisher ──► [ Topic: orders ]┼──► Warehouse service
                               └──► Analytics service

• The publisher (the sender) sends a message to a topic (a named channel).
• Every subscriber (a receiver that signed up to the topic) gets its own copy.
• Subscribers are independent. A slow analytics service does not slow down emails.
• The publisher is decoupled, which means it does not depend on its subscribers. New subscribers can be added without changing it.

Use it when one event matters to several different services: "order placed", "user signed up", "payment failed".

The real-world answer: combine both

Usually you want both behaviours at once:

• every type of service gets a copy (pub/sub between services), and
• within each service, only one instance handles each copy (a queue within a service).

Pattern 1: Topic fan-out to queues (AWS SNS + SQS style)

Fan-out means one message is copied to many places. SNS is Amazon's pub/sub topic service. SQS is Amazon's queue service.

Each service gets its own queue subscribed to the topic. Within a queue, workers compete. Result: one email, one packing job and one analytics record per order, no matter how many worker instances there are.

Pattern 2: Kafka consumer groups

Kafka (lesson 36) has this built in. Kafka is a system that stores messages in a log (an ordered list that you only add to). Consumers join a consumer group, which is a named team of consumers that share the work:

• Different groups each get every message (pub/sub).
• Within a group, each message goes to one member (queue).

Pattern 3: RabbitMQ exchanges

RabbitMQ is a popular message broker. A broker is a server that receives, stores and delivers messages. In RabbitMQ, producers (senders) send messages to an exchange, which routes them to queues by rules:

• Direct exchange: route by an exact routing key, a label on the message (order.created → the order queue).
• Fanout exchange: copy to all bound queues (pure pub/sub).
• Topic exchange: route by patterns, where * stands for exactly one word (order.*.india → the India team's queue).
• Headers exchange: route by message headers.

Push vs pull

• Push: the broker sends messages to consumers as they arrive (RabbitMQ by default, SNS to HTTP endpoints, Google Pub/Sub push). It has low latency, but consumers need flow control (for example, a "prefetch" limit, which caps how many unfinished messages a consumer may hold) so they are not overwhelmed.
• Pull: consumers ask for messages when ready (SQS, Kafka, Google Pub/Sub pull). Consumers control their own pace. This gives natural back pressure, which means a busy consumer simply asks for fewer messages, so the system slows down safely.

Ordering

"Messages arrive in the order they were sent" is harder than it sounds when you have many consumers:

• With competing consumers, message 1 might go to a slow worker and message 2 to a fast one, so message 2 finishes first.
• Global ordering (everything in one strict order) limits you to one consumer at a time, which greatly lowers throughput.
• The practical solution is per-key ordering: all messages for the same key (like the same order ID or user ID) stay in order, while different keys are processed in parallel.
  • Kafka does this with partitions (a topic is split into partitions, and the same key always goes to the same partition).
  • SQS FIFO queues (first in, first out) do it with message group IDs.

Order 5521: created → paid → shipped    (must stay in order)Order 7730: created → paid              (independent, can run in parallel)

Retention: delete or keep?

• Classic queues (RabbitMQ, SQS) delete messages once acknowledged. The queue is a to-do list.
• Log-based systems (Kafka, Pulsar, Kinesis, Redis Streams) keep messages for a set time (days, or even forever in Kafka and Pulsar), even after they are read. Consumers track their position in the log, called an offset. The system is a history book, and a new consumer can replay the past.

Replay means reading old messages again. It is very useful. You can add a new service and let it process last week's events. Or you can fix a bug and run the old events again.

Popular systems