Queues, Background Jobs and Webhooks

At 4+ years you have used a queue, a cron job or a webhook. Interviewers want to know whether you know what happens when they go wrong. A worker crashes in the middle of a job. A provider sends the same event twice. A cron job runs on three servers at once. This lesson explains the ideas and runs real code against BullMQ 6.3 (on Redis 7) and a webhook endpoint.

For the design side (it has a section on job queues with retries) read Design Problems, Lesson 07. For safe retries on HTTP APIs read REST API Design. The Stripe and BullMQ basics are in Libraries, Lesson 13. This lesson goes deeper.

Words used in this lesson

• A queue is a waiting line of jobs. A producer adds to the back and a consumer takes from the front. Like numbered tickets at a bakery.
• A job is one unit of background work with its data, such as "send the welcome email to user 7".
• A producer is the code that adds jobs. A worker (or consumer) is a program that takes jobs and runs them.
• A broker is the server that stores queues and hands out messages. Redis, RabbitMQ and Amazon SQS are brokers.
• Backoff means waiting longer between each retry. Jitter is a small random amount added to the wait, so many failed jobs do not all retry at the same moment.
• A dead-letter queue (DLQ) is a separate place for jobs that failed too many times. Like the "return to sender" box at the post office: a person looks at it later.
• A poison message is a message that fails every time (for example, bad data), so retrying can never help.
• Idempotent means doing something twice has the same result as doing it once. Setting a status to paid is idempotent. Adding 10 to a balance is not.
• A webhook is an HTTP request that another service sends to your server when something happens, such as "payment succeeded".
• A cron expression is a short schedule text, such as 0 3 * * * for "every day at 03:00".
• HMAC (hash-based message authentication code) is a short value computed from a message and a shared secret. If the message changes, the value no longer matches. It proves the sender knew the secret.

Why use a queue

Without a queue the request does everything while the user waits: save the order, send the email, call the invoice service. A queue helps in four ways:

The costs are real: one more system to run, "pending" states in the UI, harder debugging, and duplicates to handle.

BullMQ in practice

BullMQ is a Node.js library that stores jobs in Redis (an in-memory database) and runs them with workers. It is the most used job queue in the Node ecosystem.

import { Queue, Worker } from 'bullmq'const connection = { host: '127.0.0.1', port: 6379, maxRetriesPerRequest: null }export const emails = new Queue('emails', { connection })// web app (producer)await emails.add('welcome', { userId: 7 })// separate worker process (consumer)new Worker('emails', async job => {  await sendWelcomeEmail(job.data.userId)}, { connection, concurrency: 5 })

Defaults I checked in the BullMQ docs and source:

• attempts is 1, so there are no retries unless you set it.
• A worker runs one job at a time unless you set concurrency.
• Finished jobs are kept forever. Set removeOnComplete and removeOnFail or Redis fills up.
• Redis needs maxmemory-policy noeviction and persistence (AOF). If Redis evicts keys, the queue breaks silently.
• Job data is plain text in Redis. Put an ID in the job, not secrets or card numbers.

Retries, backoff and dead letters

Most failures are temporary (a timeout, a 503). Retrying with a growing wait fixes them. BullMQ has fixed and exponential backoff and an optional jitter from 0 to 1. With exponential the wait is 2^(attempt-1) * delay.

Some failures are permanent (an invalid email address). Throw UnrecoverableError and BullMQ fails the job at once, even if attempts remain. BullMQ's docs do not describe a built-in dead-letter queue. After the last attempt the job stays in the failed set, where you can inspect it and call job.retry(). For a real DLQ, copy it yourself. This is what I ran:

import { Queue, Worker, UnrecoverableError, Job } from 'bullmq'const dead = new Queue('emails-dead', { connection })const worker = new Worker('emails', async (job: Job) => {  if (job.name === 'bad-address') throw new UnrecoverableError('invalid email')  if (job.name === 'always-down') throw new Error('provider down')}, { connection })// 'failed' fires after EVERY failed attempt, so check if this was the last oneworker.on('failed', async (job, err) => {  if (!job) return  const last = job.attemptsMade >= (job.opts.attempts ?? 1) || err instanceof UnrecoverableError  if (last) await dead.add('dead', { name: job.name, data: job.data, reason: err.message })})await emails.add('always-down', { to: 'c@d.co' },  { attempts: 3, backoff: { type: 'exponential', delay: 500 } })

Result: always-down ran at about 0 s, 0.7 s and 1.8 s (waits of 500 ms and 1,000 ms), then landed in the dead queue. bad-address ran once and went straight there. A flaky job that failed twice succeeded on attempt 3.

A DLQ is not a trash can. Someone must watch it (alert when it is not empty), read why jobs failed, fix the bug and replay them.

Idempotency: the most important idea

Queues can deliver a job twice (next section). The only reliable defense is to make the work idempotent. Three ways, from weakest to strongest:

1. Naturally safe operations: UPDATE orders SET status = 'paid', upserts, "create if not exists". Avoid balance = balance + 10.
2. A unique key in your database. Before the side effect, insert a row with a unique key (the event ID, or orderId + 'welcome-email'). If the insert fails, skip the work. This works across crashes and servers.
3. The provider's idempotency key. Stripe and many APIs accept one, so a repeated call does not charge twice.

// only the FIRST delivery sends the emailconst r = db.prepare('INSERT OR IGNORE INTO emails_sent (order_id) VALUES (?)').run(job.data.orderId)if (r.changes === 1) await sendEmail(job.data.orderId)

There is a gap: if the process crashes after the insert but before the email, the retry skips the email. Choose the order by what hurts more, a lost email or a double one. For money, use the provider's key too.

BullMQ also helps when adding jobs:

• jobId: adding a second job with the same jobId returns the existing job. Gotcha: only while the job still exists. With removeOnComplete: true I added the same jobId after the first job finished and it ran twice. With removeOnComplete: false it ran once.
• deduplication: { id, ttl } ignores new jobs with the same id for ttl ms (or until the first job finishes, if you give no ttl). I added three jobs with one id: one stayed and the deduplicated event fired twice. In BullMQ 6 the old debounce option is gone; use deduplication.

At-least-once, at-most-once, exactly-once

An acknowledgement (ack) is the consumer telling the broker "I finished, you may delete it". If the consumer crashes before the ack, the broker gives the message to someone else. That is why duplicates exist: the work may be done with only the ack missing.

Exactly-once delivery over a network is not possible in the general case, because the sender cannot tell "the message was lost" from "the answer was lost". What you can build is effectively once: at-least-once delivery plus an idempotent consumer. Kafka's own docs say to read the fine print on exactly-once claims. Its exactly-once covers reading from a topic and writing to another Kafka topic (with transactions). A write to your database or an email is outside it.

I proved the crash case. I started a worker with a 2-second lockDuration on a long job and killed it with kill -9. A second worker logged the job as stalled and started it again with attemptsStarted=2. Nothing was lost, and the job ran again.

Workers, concurrency and shutdown

Concurrency is the number of jobs one worker runs at once. It uses the Node.js event loop, so it helps with I/O-bound jobs (waiting on the network or database). I ran 10 jobs that each wait 200 ms: about 2,050 ms with concurrency: 1 and about 430 ms with concurrency: 5. The BullMQ docs call 100 to 300 normal for I/O jobs, but tune it in production.

For CPU-bound jobs (image resizing) concurrency does not help, because JavaScript runs on one thread. Add worker processes or machines, or use a sandboxed processor (your function runs in a separate process). Workers scale out because each job goes to only one worker.

Stalled jobs. A worker keeps a lock on its active job and renews it. The default lockDuration is 30 seconds. If the lock is not renewed (the process died, or a CPU loop blocked the event loop), the job is stalled and goes back to waiting. If it stalls more than maxStalledCount times (default 1), it fails with "job stalled more than allowable limit". So a long synchronous loop can make a healthy job look dead.

Graceful shutdown. On a deploy the platform sends SIGTERM (a polite "please stop"). Call worker.close(): it takes no new jobs and waits for the running ones. In my test it waited about 830 ms for a one-second job.

process.on('SIGTERM', async () => {  await worker.close()  process.exit(0)})

If a job outlasts the grace period (30 seconds by default in Kubernetes), the process is killed and the job stalls. Keep jobs short or split them into steps.

Also useful: limiter: { max: 10, duration: 1000 } limits the whole queue across all workers (some pages in the BullMQ docs still show the old QueueScheduler, which is not needed since BullMQ 2.0). priority runs important jobs first. delay runs a job at least that many ms later. The metric to watch is the age of the oldest waiting job: it shows when workers cannot keep up.

Scheduled jobs and cron

Typical cron work: nightly reports, cleanup, reminders. The hard parts are not the syntax:

• Time zones. Server cron often uses UTC. "Every day at 9" for users in Berlin moves when daylight saving time changes. Choose a zone on purpose.
• Many instances. setInterval or node-cron inside a web app that runs three copies fires three times. Run the scheduler in one place, use a lock, or let the queue do it.
• Overlap. If a run is longer than the interval, the next one starts while the first is still going.
• Missed runs. If the server is down at 03:00, plain cron does not run later. Process "everything since the last success", not just "yesterday".

In BullMQ use a Job Scheduler. It replaces the old "repeatable jobs", whose API is removed in BullMQ 6. upsertJobScheduler creates or updates by id, so deploying twice does not make two schedulers:

await queue.upsertJobScheduler(  'nightly-report',  { pattern: '0 0 3 * * *', tz: 'UTC' },   // BullMQ cron has a seconds field first  { name: 'report', data: {}, opts: { attempts: 3, removeOnFail: 1000 } },)

I ran one with every: 1000 and upserted it twice with the same id: getJobSchedulers() showed one scheduler and it made about one job per second. The docs note the next job is created when the last one starts, so a busy queue can run it less often than planned. The jobs go through normal workers, so retries and concurrency apply.

The outbox pattern

The problem: you save an order, then add a job to the queue. These are two systems and cannot share one transaction. A crash between them leaves an order with no email job. This is the dual write problem.

The outbox pattern fixes it. In the same database transaction that saves the order, also insert a row into an outbox table. A separate relay reads unsent rows, publishes them to the queue and marks them sent.

db.exec('BEGIN')db.prepare('INSERT INTO orders (id, total) VALUES (?, ?)').run(id, total)db.prepare('INSERT INTO outbox (type, payload) VALUES (?, ?)')  .run('order.created', JSON.stringify({ orderId: id }))db.exec('COMMIT')      // both rows, or neither// relay: publish, THEN mark as sentfor (const row of unsentRows) {  await queue.add(row.type, JSON.parse(row.payload), { jobId: `outbox-${row.id}` })  db.prepare("UPDATE outbox SET sent_at = datetime('now') WHERE id = ?").run(row.id)}

I ran this on SQLite. A failed order rolled back with no outbox row. When I made the relay "crash" after publishing row 1 and before marking it, the restart published row 1 again, the same jobId kept it to one job, and the consumer's unique key sent each email once. So the outbox gives at-least-once, and consumers must still be idempotent. The Meilisearch sync in Lesson 12 uses the same idea.

Trade-offs: a small delay (the relay polls), a table to clean, one more process. Alternative: change data capture (a tool reads the database log and publishes changes).

Webhooks: receiving them

A webhook endpoint is a public URL, so anyone can call it. Six rules:

1. Read the raw body. Signatures cover the exact bytes. I checked: a signature made over a pretty-printed body failed after JSON.stringify(req.body). In Express use express.raw({ type: 'application/json' }) on that route; in a Next.js Route Handler use await request.text().
2. Verify the HMAC with a constant-time comparison (crypto.timingSafeEqual), not ===. A normal comparison stops at the first different character, and its timing can leak information.
3. Check the timestamp. Providers often sign timestamp.body. Reject old timestamps (Stripe's default tolerance is 5 minutes). This stops replay: re-sending a captured valid request.
4. Answer fast with 2xx. Verify, save or queue the event, return 200, work in a worker. A slow answer counts as a failure and is retried.
5. Be idempotent. Store the event ID with a unique constraint. For one you have seen, return 200 (an error makes them retry).
6. Do not trust the order. Treat the event as a hint, then fetch the current object from the provider API and write that.

app.post('/webhooks/pay', express.raw({ type: 'application/json' }), async (req, res) => {  if (!verify(req.body, req.header('x-signature'))) return res.status(400).send('bad signature')  const event = JSON.parse(req.body.toString('utf8'))  try {    db.prepare('INSERT INTO processed_events (id) VALUES (?)').run(event.id)  // unique key  } catch {    return res.status(200).send('duplicate, ignored')  }  await queue.add(event.type, event, { jobId: `evt-${event.id}` })  res.status(200).send('queued')})

I ran five requests. A valid one gave 200 queued; the same again gave 200 duplicate, ignored; a tampered body, a one-hour-old timestamp and a missing header each gave 400. One job was queued. My verify parses t=<seconds>,v1=<64 hex>, checks the age, recomputes HMAC-SHA256(secret, t + '.' + body) and uses timingSafeEqual.

Sending webhooks makes you the provider. Sign every request and include a timestamp and a unique event ID. Send from a queue, with exponential backoff and jitter over hours, and a short timeout. Never call private IP addresses (SSRF: a customer could aim your server at your internal network). Disable endpoints that fail for days, and give customers a delivery log and a replay button.

RabbitMQ, Kafka, SQS: which one?

How I choose:

• BullMQ / Redis: Node app with jobs, delays, cron, retries. Fast to start. Risk: Redis is memory-first, so set persistence and noeviction.
• SQS: on AWS, no servers to run, with a DLQ and Lambda as consumer.
• RabbitMQ: complex routing (topic exchanges, fan-out), many languages, priorities.
• Kafka: an event log: many independent readers, replay, stream processing, very high volume. Overkill for "send an email later".

Two Kafka words: a consumer group is a set of consumers sharing a topic, where each partition goes to one member. An offset is a message's position in a partition. Since order exists only inside a partition, pick a key such as userId to keep related events together. I checked two defaults in the source: producer idempotence is on, and the consumer's enable.auto.commit is also on, which can commit a position before your work is done. Many teams turn it off and commit after processing.

For SQS, the visibility timeout (default 30 seconds) hides a received message from others. If you do not delete it in time, it reappears. Set it longer than your slowest job.

Serverless: where do background jobs run?

A serverless function lives for a request and has a time limit, so a long-lived BullMQ worker does not fit.

Common mistakes

• A worker that is not idempotent, so each retry or crash makes a duplicate email or charge.
• attempts with no backoff, which hammers a service that is already down. Retrying errors that can never succeed.
• Nobody looks at failed jobs or the DLQ.
• Redis with eviction on, no persistence, or finished jobs kept forever.
• Parsing the webhook body as JSON before verifying, comparing with ===, skipping the timestamp.
• Cron inside a multi-instance web app. Database write and queue add as two steps, with no outbox.
• Trusting webhook order.

Interview questions

Key takeaways

• Use a queue for slow, flaky, bursty or must-not-lose work. Return fast, work later.
• Every queue can deliver twice. Build at-least-once + idempotent ("effectively once"). Tools: natural idempotency, a unique key, the provider's key. jobId and deduplication only help when adding jobs.
• Retries need attempts, exponential backoff with jitter and UnrecoverableError for permanent failures, plus a dead-letter step that someone watches.
• Concurrency helps I/O jobs, not CPU jobs. Shut down with worker.close(). Do not block the event loop, or jobs stall.
• Cron: run it in one place or use upsertJobScheduler. Decide the time zone; handle overlap and missed runs.
• Outbox fixes the dual write problem: event row and business row in one transaction, then a relay publishes.
• Webhooks: raw body, HMAC with constant-time compare, timestamp check, fast 2xx, event-ID dedupe, never trust order.
• BullMQ for Node jobs, SQS for simple managed queues, RabbitMQ for routing, Kafka for event logs and replay.
• Serverless: after() for small lossy work, platform cron as best effort, an external queue or workflow service for durable work.

What's next

Libraries, Lesson 13 covers the tools beside queues: Redis caching, Stripe and TanStack Query. For a design that includes job queues with retries, see Design Problems, Lesson 07.