Background Jobs, Retries & Dead-Letter Queues

Task queues vs message brokers

• A message broker (RabbitMQ, SQS, Redis, Kafka) is a server that stores messages and passes them from senders to receivers.
• A task queue framework is a library that runs jobs on top of a broker. It adds useful features: retries, scheduling, priorities, job status and dashboards.

// BullMQ example: enqueue in the request, process in a workerawait emailQueue.add("welcome", { userId: 42 }, { attempts: 5, backoff: { type: "exponential", delay: 2000 } });new Worker("email", async (job) => {  await sendWelcomeEmail(job.data.userId);   // should be idempotent (post 37)!});

Designing good jobs

1. Keep job payloads small: pass IDs, not whole objects. The payload is the data you put inside the job message.

2. Make every job idempotent. Idempotent means running it many times gives the same result as running it once. Jobs will run more than once, because of retries, crashes and redelivery (post 37). "Send welcome email" should first check "already sent?". "Charge invoice" should use an idempotency key (a unique label that lets the receiver spot a repeat).

3. Make big jobs resumable, or split them. A job that handles 1 million rows should work in chunks (for example 1,000 rows each) and save its progress. Or it can fan out into many small jobs. Then a crash means redoing 1,000 rows, not 1 million.

4. Set timeouts. A timeout is a maximum time you allow. A job that is stuck forever holds a worker forever. Give each job type a sensible maximum run time.

5. Separate queues by priority and duration.

This way, a flood of slow report jobs never delays OTP emails (OTP means one-time password, a short code sent to a user). Give each queue its own workers, or give them weighted priority (a bigger share of worker time for important queues).

Workers in practice

• Concurrency is how many jobs one worker runs at the same time (with threads, async code or processes, post 4). Tune it to your work. CPU-bound jobs (they spend time calculating) get a low number. I/O-bound jobs (they spend time waiting for networks or disks) get a higher number.
• Prefetch is how many jobs a worker takes from the queue in advance. If it is too high, one worker keeps many jobs while other workers sit idle.
• Visibility timeout and heartbeat. When a worker takes a job, the queue hides it for a set time. This is the visibility timeout, or lease. A heartbeat is a regular "I am still alive" signal. Long jobs must extend their lease (post 18). Otherwise the queue thinks the worker died and gives the job to someone else, while the first worker is still running it.
• Graceful shutdown means stopping in a clean way. On deploy, workers should stop taking new jobs, finish (or safely give back) their current jobs, then exit. Set a grace period (the time allowed to finish) that is long enough for typical jobs.
• Auto-scaling means adding or removing workers by itself. Base it on queue depth (how many jobs wait) and job age, not only on CPU.

Scheduled and recurring jobs

• Delayed jobs run later: "send a reminder in 24 hours", "retry in 5 minutes". Most frameworks can schedule a job for later.
• Recurring jobs (cron) run again and again on a time rule. Cron is the classic Unix scheduler. Example: "every night at 2 AM, clean up expired sessions".

The distributed cron problem: if you run the cron schedule on every server, the job runs N times (once per server). Fixes:

• a single scheduler process (with a standby one ready) that adds the job to the queue once,
• leader election (the servers vote so that only one is in charge) or a distributed lock (a lock that all servers share, so only one can hold it). Examples are a Redis lock with a TTL (time to live: the lock frees itself after a set time) or a database advisory lock,
• platform features like Kubernetes CronJobs or cloud schedulers,
• and, as always, making the job idempotent, so an accidental double run does no harm.

Workflow engines for multi-step jobs

Some "jobs" are really long processes with many steps: "charge card → reserve stock → book courier → send confirmation". If booking fails, you must refund the card. Building this as a chain of queued jobs, with your own state tracking, gets messy.

A workflow engine is a tool that runs multi-step processes for you. It keeps the state, and handles retries, timeouts and compensation (undo steps, like the refund):

• Temporal (created by engineers who originally built Uber's Cadence),
• AWS Step Functions,
• Apache Airflow (for data pipelines),
• Azure Durable Functions.

These tools are closely linked to sagas (a saga is a long process made of steps, where each step has an undo step, post 39).

Retries

Transient vs permanent failures

A retry means running a failed job again. Before you retry, ask: will trying again help? A transient failure is short and goes away by itself. A permanent failure stays until someone fixes it.

Retrying permanent failures wastes resources and delays everything else.

Retry with exponential backoff and jitter

Backoff means waiting before a retry. Exponential backoff makes the wait longer each time, for example 1s, 2s, 4s, 8s. Do not retry at once, and do not retry forever:

Add random jitter. Jitter is a small random change to the wait time. For example, wait between 0 and 8 seconds instead of exactly 8. Then thousands of failed jobs do not all retry at the same instant and overload the service that is trying to recover. (Post 41 covers this in depth.)

Retry storms

A retry storm is when many retries together overload a service that is already weak. It happens when a downstream service (a service you call, like a partner API) goes down:

• every job fails and retries,
• the retries add load exactly when the service is weakest,
• its recovery is delayed, which causes more failures and more retries.

How to protect yourself:

• cap attempts per job (set a maximum number),
• use backoff with jitter,
• keep retry budgets (a limit on total retries, for example "retries may add at most 10% extra load"),
• use circuit breakers. A circuit breaker is like an electric fuse: after many failures it stops calls to that service for a while (post 42),
• rate-limit workers that call weak third-party services (a rate limit caps how many calls per second are allowed).

Retries and ordering

Queues with strict order (Kafka partitions, FIFO queues) have a special problem. Say message #5 fails and you retry it in place. Then every message behind it must wait. This is called head-of-line blocking: one stuck item at the front of the line blocks all the others. The common fix is retry topics (a topic is a named message stream in Kafka):

The main topic keeps flowing while failed messages wait in separate topics with longer and longer delays. This breaks strict order for the failed messages. If order really matters, hold back the later messages that have the same key.

Dead-letter queues (DLQs)

A dead-letter queue (DLQ) is a separate queue that holds messages after they have failed too many times (or are clearly invalid). It is the "problem shelf".

Why DLQs matter:

• Poison messages (messages that always fail) stop blocking workers or wasting their time.
• Nothing is lost without a sign. Failed work is kept so you can look at it.
• After you fix the bug or the data, you can replay (send again) the messages.

Built-in support:

• Amazon SQS: a "redrive policy" moves a message to a DLQ after maxReceiveCount failed receives. There is also a "redrive" feature to move messages back.
• RabbitMQ: dead-letter exchanges. Rejected or expired messages are sent to another exchange.
• Kafka: there is no built-in DLQ for normal consumers. You build one with retry and DLQ topics (Kafka Connect sink connectors and many frameworks give you this).
• Task frameworks: Sidekiq's "Dead" set, Celery and BullMQ failed-job lists.

DLQ best practices:

1. Alert on DLQ size and growth. A DLQ that nobody watches is a black hole.
2. Store context with each dead message: the error, the stack trace (the list of function calls at the error), the attempt count and the times.
3. Build tools to look at, fix and safely replay messages, one by one or in bulk.
4. Replay with care. Make sure the bug is fixed and consumers are idempotent. Replaying 50,000 messages at once can overload systems, so rate-limit it.
5. Set retention (how long messages are kept). DLQ messages expire too. SQS keeps them for up to 14 days, so act within that time.

What to monitor

• Queue depth (jobs waiting) and oldest job age. Age is often the better signal: "OTP jobs are 3 minutes old" is an incident.
• Throughput (jobs processed per second) and failure rate per job type.
• Retry counts and DLQ size.
• Job duration, as p50 and p99. p50 is the typical time (half the jobs are faster). p99 is the slow end (99 of 100 jobs are faster). Watch it to catch slow jobs before they time out.