Message Queues: A Gentle Introduction

The basic flow

                           ┌─────────────────────┐
[API: place order] ──msg──►│ Queue: order-events │──► [Worker 1] send email
   (producer)              │ ■ ■ ■ ■ ■           │──► [Worker 2] send email
                           └─────────────────────┘──► [Worker 3] send email
                                                        (consumers)

For the order example:

The user gets a fast response, and the slow or unreliable tasks happen in the background. "Publish" means "put a message on the queue". "Ack" is short for acknowledgement, a note from the worker that says "I finished this message".

Why queues help

1. Faster responses. The user only waits for the essential work.

2. Decoupling. Decoupling means that parts of the system do not depend on each other directly. The API doesn't need to know about the email service, the SMS service or the warehouse system. It just publishes "order placed". You can add a new consumer (say, a fraud-check service) without changing the API.

3. Resilience. If the email service is down for 10 minutes, messages wait in the queue. When it recovers, workers process the backlog, and nothing is lost. Without a queue, those 10 minutes of emails would simply fail.

4. Smoothing traffic spikes (load levelling). A spike is a sudden burst of traffic. During a flash sale, orders arrive at 20,000 per minute, but the warehouse system can only handle 2,000 per minute. The queue absorbs the spike, and workers process at a steady, safe rate. The queue grows during the rush and drains afterwards.

Incoming:   ▁▁▁█████████▁▁▁▁▁▁   (flash-sale spike)
Queue:      ▁▁▁▂▄▆███▇▅▃▂▁▁▁▁   (grows, then drains)
Processing: ▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃   (steady, at the rate downstream can bear)

5. Easy scaling. Queue getting long? Add more workers. Many workers taking from the same queue is called the competing consumers pattern (the workers compete for messages). Each message goes to just one of them.

Acknowledgements: not losing messages

What if a worker takes a message and then crashes halfway through sending the email? Queues handle this with acknowledgements (acks):

In Amazon SQS (Simple Queue Service, a queue run by AWS), this timeout is called the visibility timeout. In RabbitMQ (a popular open-source message broker, which is a server that routes messages), if a consumer disconnects without acking, the message is redelivered.

There's an important consequence: a message might be processed more than once. For example, the worker sent the email, then crashed before acking. So consumers should be safe to run twice (idempotent). For example, record "email sent for order 5521" and skip it if it's already done. Idempotent means that doing the same thing twice gives the same result as doing it once. Part 6 goes deep on this.

When things keep failing

Some messages will fail over and over: bad data, a bug, a missing record. If they're retried forever, they waste resources and can block other work. The usual solution:

• Retry a few times, with increasing delays.
• Then move the message to a dead-letter queue (DLQ), a separate queue for "problem" messages, which engineers can inspect and fix. "Dead letter" is an old post-office term for a letter that cannot be delivered.

(More in Part 6.)

Back pressure

If producers keep adding messages faster than consumers can ever process them, the queue grows without limit. Messages get old and useless (an OTP, or one-time password, that arrives 20 minutes late), and memory or disk fills up. You need to:

• Monitor queue length and message age.
• Auto-scale workers based on queue length (start more workers when the queue grows, and stop them when it shrinks).
• If needed, push back on producers (slow them down, or reject non-essential work). This is called back pressure.

What should go on a queue?

Good fits:

• Sending emails, SMS and push notifications.
• Image and video processing (resizing, transcoding: converting video to other formats and sizes).
• Generating reports, invoices and PDFs.
• Syncing data to search indexes, analytics and data warehouses (large databases built for reports).
• Calling slow or unreliable third-party APIs.
• Anything the user doesn't need to see immediately.

Poor fits:

• Anything the user must see the result of right now: login checks, "is this username available?", getting a price.
• Simple, fast work where a queue only adds complexity.

Popular message queues

How the user sees async results

If work happens later, how does the user find out?

• Just tell them: "We'll email your invoice shortly."
• Polling: the app asks again and again. It checks GET /jobs/123 every few seconds ("Processing… Done!").
• Push: the server sends the news to the client. It uses WebSockets (a connection that stays open so the server can send data at any time) or push notifications when the job finishes (Part 5).
• Webhooks: for other systems, "we'll call your URL when it's done". A webhook is a web address that you give us, and we send a request to it when something happens.