Design a Notification System

Step 1: Requirements

Functional:

1. Send notifications through several channels: mobile push (a message shown on the phone screen, for iOS and Android), SMS, email and in-app (a notification inbox inside the app, or live messages through a WebSocket, which is a connection that stays open).
2. Support transactional (OTP, order updates, security alerts) and marketing / bulk notifications (campaigns).
3. User preferences: the user can opt in or out (say yes or no) for each channel and category, and set quiet hours (times with no messages) and a language.
4. Templates (message text with blanks that are filled in for each user) with personalisation (Hi {{name}}, your order {{orderId}} has shipped) and localisation.
5. Scheduling: send at a set time, or at a set time in the user's own time zone.
6. Tracking: know if a message was sent, delivered, opened, clicked, failed or bounced.

Out of scope: building our own SMS or email delivery systems (we use providers), and the screen for designing campaigns.

Non-functional:

• Reliability: notifications must not be lost. Aim for at-least-once delivery with deduplication. At-least-once means every message arrives, but it may arrive twice. Deduplication removes the second copy, so users do not get duplicates (post 37).
• Low latency for critical messages: OTPs and security alerts must arrive within seconds.
• High throughput for bulk: millions of messages in minutes, without slowing down transactional traffic. (Throughput is how much work is done per second.)
• Scalable and extensible: it can grow, and it is easy to add a new channel (say, WhatsApp) or a new provider.
• Respect user choices and the law: opt-outs, consent (permission from the user) and anti-spam rules.

Step 2: Estimation

Assumptions:

• 50M users, about 10M push, 1M SMS and 5M email notifications on a normal day. (M means million.)
• Occasionally, a marketing campaign to 20M users within 10 minutes.

So this means:

• Normal load is modest. Campaign bursts are about 30 times higher. So we need queues (waiting lines for jobs) to soak up bursts, and separate lanes so bursts do not delay OTPs.
• Third-party providers have rate limits. We must throttle (slow down on purpose) what we send them. A rate limiter is a part that limits how many requests a caller can make in a period of time (post 43).
• Status logs grow quickly. So use storage split by time (one partition per day or month), and delete old data after a retention period.

Step 3: API and data model

Send API (the request that other internal services call):

POST /v1/notificationsIdempotency-Key: order-5521-shipped            ← a retry will not send the message twice{  "userId": "42",  "type": "order_shipped",                     ← points to the template, category and priority  "channels": ["push", "email"],  "data": { "orderId": "5521", "eta": "Thu" },  "priority": "high",  "sendAt": null}202 Accepted  { "notificationId": "ntf_8f2a" }   ← accepted, and will be delivered later (asynchronously)

Bulk sends use a campaign API. It points to an audience segment (a saved group of users) and does not need millions of single calls.

Core data:

users_contact:      user_id, email, phone, locale, timezonedevice_tokens:      user_id, platform (ios/android/web), token, app_version, last_seenpreferences:        user_id, category (marketing/transactional/social…), channel, enabled,                    quiet_hourstemplates:          type, channel, locale, subject/body with placeholders, versionnotifications_log:  notification_id, user_id, type, channel, status, provider_msg_id,                    timestamps (created, sent, delivered, opened), error

Step 4: High-level design

The flow for "order shipped":

1. The order service calls the Notification API with an idempotency key. This is a unique ID for the request. If the same key arrives again, the system knows it is a repeat.
2. The API removes duplicates, loads the user's preferences (email allowed? push enabled?), picks the channels, fills in the templates in the user's language, and applies quiet hours (messages that are not urgent wait until morning).
3. Messages go into the transactional queue for each channel.
4. Push workers (programs that take jobs from the queue) send to APNs (Apple Push Notification service) and FCM (Firebase Cloud Messaging, from Google) using the user's device tokens (an address for one app on one phone). Email workers call the email provider.
5. Providers report the delivery status (in their replies and through webhooks, which are calls that the provider sends to you), and the tracking service updates the log.
6. Failures are retried with backoff (waiting longer each time), and permanent failures go to a DLQ (dead-letter queue, a place for messages that cannot be processed, post 38).

Step 5: Deep dives

Deep dive 1: Priority lanes, so OTPs never wait behind marketing

• Separate queues or topics per priority (and per channel), each with their own workers. This is a bulkhead: like the walls in a ship, a problem in one part does not flood the others (post 42).
• Critical: OTPs, password resets, security alerts, payment confirmations. Low latency, highest priority, and watched by message age, which is how long the oldest message has waited ("the oldest OTP must be less than 10 seconds old").
• Transactional: order and delivery updates.
• Bulk / marketing: campaigns. Throttled, and it can be delayed or shed (dropped on purpose) when the system is busy (post 44).
• Provider capacity is shared, so keep part of each provider's rate limit for critical traffic only.

Deep dive 2: Reliability and no duplicates

• At-least-once delivery through queues (post 37). A worker only sends an acknowledgement (a "done" signal to the queue) after the provider accepts the message. If the worker crashes before that, the queue sends the job again.
• Remove duplicates at two levels:
  • Request level: the Idempotency-Key from the calling service (post 34).
  • Send level: a (notification_id, channel) record, marked as sent. Workers check it before sending, so a queue message that arrives again does not send twice.
• Pass an idempotency key to providers if they support it.
• Transactional outbox in calling services (post 37). An outbox is a database table where a service writes the messages it still has to send. The service saves the notification request in its own database in the same transaction as the order change. So "save order as shipped" and "request a notification" cannot get out of sync.

Deep dive 3: Third-party providers, limits and failover

• Respect provider rate limits. Use a token bucket per provider (a token bucket is a counter that refills at a steady rate, and each request spends one token; post 43) so the provider does not slow you down or block you.
• Retry transient errors (errors that may go away: timeouts; 5xx errors, which are replies with a status code from 500 to 599 that mean the server had a problem; and 429 too many requests) with backoff + jitter. Backoff means waiting longer each time. Jitter means adding a random bit to the wait, so retries do not all happen together. Do not retry permanent errors (invalid number, unsubscribed, bad token).
• Fail over (switch) to a backup provider for SMS and email when the main provider's error rate jumps. A circuit breaker for each provider (post 42) makes this automatic. A circuit breaker stops calls to a service that keeps failing.
• Handle provider feedback:
  • Email bounces and spam complaints: a hard bounce means the address does not exist. Stop emailing addresses that hard-bounce or complain. If you do not, your sender reputation gets worse, and delivery gets worse for everyone.
  • Invalid push tokens: APNs and FCM tell you when a device token is not valid any more (for example, the app was uninstalled). Delete those tokens.

Deep dive 4: Push notifications specifics

• Each device registers with APNs or FCM and gets a device token. The app sends the token to your backend, which saves it in device_tokens.
• Users have several devices (phone, tablet, web). So send to each active token, or only to the most recently used one, depending on the notification.
• Keep payloads (the data in the message) small, and do not put sensitive data in push text, because it may show on a locked screen. For example, write "You have a new message", not the message itself.
• Collapse keys / thread IDs stop floods. A collapse key is a label that tells the push service to replace an older waiting message that has the same label with the newer one. They group messages, so the user sees "5 new messages" instead of 5 separate alerts.

Deep dive 5: User experience and respect

• Preferences and consent: users choose categories and channels, and marketing requires opt-in (the user must say yes first) in many regions. Include an unsubscribe link in marketing emails.
• Per-user rate limits and digests: do not send 40 "someone liked your post" pushes in an hour. Group them into one digest (a summary message), for example "Ravi and 39 others liked your post".
• Quiet hours and time zones: send messages that are not urgent at sensible local times.
• Smart channel selection: if the user is active in the app right now, show an in-app notification instead of a push. If a push is not opened, you may send an email later.
• Localisation: one template per language, with a default language to use if one is missing.

Deep dive 6: Big campaigns

For 20M messages:

1. The campaign service works out the audience in batches (for example, 10,000 users per job), and leaves out users who have opted out.
2. It enqueues batches into the bulk lane.
3. Workers slow down to stay within provider limits, and spread the sends over the planned time window.
4. Monitor and pause the campaign if error rates or complaints jump. A kill switch is a control that turns something off at once (post 44).

Deep dive 7: Tracking and observability

• Status lifecycle: accepted → queued → sent → delivered → opened/clicked, or failed / bounced / suppressed (opted-out).
• Dashboards: delivery rate per channel and provider, latency (especially OTP end-to-end time), queue ages, DLQ size, bounce and complaint rates (posts 46–48).
• SLOs (service level objectives, which are reliability targets), for example "99% of OTPs delivered to the provider within 5 seconds" (post 47).

Deep dive 8: Security and abuse

• Only trusted internal services can call the send API. Each service proves who it is (service authentication, post 49).
• Protect OTP endpoints from abuse: rate limits per phone number, IP and device. Watch for SMS pumping fraud. Here bots trigger a large number of SMS messages to premium-rate numbers, so that the attackers earn money from the charges (post 43).
• Do not log full OTPs or sensitive message content (post 51).
• Check provider webhooks. The provider signs each status callback, and you verify the signature to be sure it is real (post 34).

Wrap-up

• Key decisions: one central, asynchronous system (it answers "202 Accepted" and sends later) with priority lanes per channel; preferences, templates and dedup applied centrally; channel workers with provider rate limits, retries, circuit breakers and failover; tracking via provider callbacks.
• Trade-offs: at-least-once plus dedup is reliable, but the dedup storage is complex. Batching and quiet hours give a better experience, but messages that are not urgent arrive later. Several providers make the system resilient, but cost more to connect and pay for.
• Next steps: choosing the best send time with machine learning, ranking channels, WhatsApp and other messaging channels, and a self-service campaign screen.