Design a Notification System (In-App, Push, Email)

A few words first. An event is a small message that says "something happened" (for example comment.created). A channel is one way to reach a person (the bell in the app, push, email or SMS). Delivery means handing a message to the channel. A queue is a waiting line that stores jobs until a worker takes them. A worker is a program that takes jobs from a queue and does them. Idempotent means "safe to repeat": doing it twice gives the same result as doing it once.

This problem touches almost every backend topic: queues, retries, removing duplicates, real-time connections, user settings and database design. It also has a frontend part: the bell icon.

1. Requirements

Functional

• Send notifications through many channels: in-app (the bell), web or mobile push, email and SMS.
• Start a notification from an event (a comment, a payment, a shipped order), from a schedule, or from a broadcast to many users.
• Users have preferences (per channel and per type, plus quiet hours). They can mark notifications as read.
• Show an unread count and a list. Both update in real time (within a second or two, without a page refresh) while the user is online.

Non-functional

• Reliable delivery: do not lose important notifications (at least once means every message is delivered one or more times, never zero). Also avoid duplicates, so the user feels it was sent exactly once.
• Low latency for in-app (a few seconds). Email can be slower.
• Scalable: handle spikes (a broadcast to a million users), keep channels separate from each other, and cope when a provider (an outside service such as an email or SMS company) is down.
• Respect the rate limits of providers (the most requests they allow per time period) and avoid user fatigue (the user gets so many messages that they stop reading or turn them off).
• Easy to watch and audit.

2. Estimates

10 million users, 20% daily active → 2 million DAU (daily active users)Avg 10 notifications per DAU per day  → 20 million in-app notifications / day ≈ 230 per second (peak ~2,000)Concurrent online users (open SSE or WebSocket connections, see section 3): ~5% of DAU at peak ≈ 100,000 open connectionsStorage: 20M/day × 365 × 500 B ≈ 3.6 TB/year → keep 90 days hot, archive the rest

What this tells us: the event pipeline is a medium load. The hard scaling problems are holding 100,000+ long-lived connections (connections that stay open for minutes or hours) and fan-out spikes (one message turning into many) for broadcasts.

3. Choosing the real-time transport

How does the server tell a browser that something happened? HTTP normally works only one way: the browser asks and the server answers. So we need one of these techniques.

EventSource is the browser object that opens an SSE stream. Last-Event-ID is a header the browser sends when it reconnects, holding the id of the last event it received.

For notifications, SSE is usually the best choice. The data goes one way (server to client). The browser reconnects by itself. Resuming from Last-Event-ID is built in. Choose WebSocket when the client also sends messages often (chat, shared editing). Use polling as a fallback and for simple, low-volume cases.

4. Architecture

 ┌───────────┐  events   ┌──────────────────┐   ┌─────────────────────┐ │ Services  │──────────▶│  Event bus       │──▶│ Notification service │ │ (orders,  │ (Kafka /  │  (Kafka/SNS/SQS) │   │  • resolve recipients│ │ comments) │ SQS/SNS)  └──────────────────┘   │  • check preferences │ └───────────┘                                   │  • dedupe / throttle │                                                 │  • render templates  │                                                 └───────────┬─────────┘                                                             │ one message per (user, channel)                    ┌───────────────┬─────────────┬──────────┴───────┐                    ▼               ▼             ▼                  ▼              In-app queue     Push queue    Email queue         SMS queue                    │               │             │                  │              store + realtime   FCM/APNs/     SES/SendGrid     Twilio/SNS                    │            Web Push                    ▼        Realtime gateway (SSE/WebSocket) ──▶ connected browsers

Main ideas:

• Use queues to separate the parts. This is called decoupling: each part can fail or be slow without stopping the others. A producer (the code that creates a message) publishes an event (comment.created) and moves on. It does not call email or SMS providers itself. So a slow provider cannot slow down checkout.
• Use one queue per channel. Then a failing email provider does not delay in-app messages. Each queue has its own scaling, rate limits and retry rules.
• The notification service turns one event into zero or more notification jobs. It finds the recipients (for example, the followers of a post). It applies preferences and quiet hours. It removes duplicates. It builds the message text (templates and translations).
• Workers (one group per channel) call the providers. FCM (Firebase Cloud Messaging) and APNs (Apple Push Notification service) are the services that deliver push messages to Android/Chrome and to Apple devices. SES, SendGrid (email) and Twilio (SMS) are provider companies. Kafka, SNS and SQS are messaging systems that carry events and queue jobs. They use retries and backoff (waiting longer after each failed try). They also save the delivery status.

5. Data model

CREATE TABLE notifications (  id          uuid PRIMARY KEY,  user_id     uuid        NOT NULL,  type        text        NOT NULL,                  -- 'comment.reply', 'order.shipped'  payload     jsonb       NOT NULL,                  -- ids and data needed to render; avoid copying large objects  dedupe_key  text,                                  -- e.g. 'comment.reply:post42:user9'  created_at  timestamptz NOT NULL DEFAULT now(),  read_at     timestamptz,  UNIQUE (user_id, dedupe_key)                       -- idempotency: the same event cannot create two rows);CREATE INDEX notif_user_created ON notifications (user_id, created_at DESC, id DESC);       -- the list, cursor-paginatedCREATE INDEX notif_user_unread  ON notifications (user_id) WHERE read_at IS NULL;            -- small partial index for unreadCREATE TABLE notification_preferences (  user_id uuid, channel text, type text, enabled boolean NOT NULL DEFAULT true,  quiet_start time, quiet_end time, PRIMARY KEY (user_id, channel, type));CREATE TABLE push_subscriptions (user_id uuid, platform text, token text, last_seen timestamptz, PRIMARY KEY (user_id, token));CREATE TABLE deliveries (notification_id uuid, channel text, status text, attempts int, provider_id text, error text, PRIMARY KEY (notification_id, channel));

• jsonb is a PostgreSQL column type that stores JSON in a binary form you can search. A dedupe key ("deduplication key") is a text value that is the same for the same event, so duplicates can be detected.
• The list: show the bell list with cursor pagination (loading one page at a time, where the "cursor" marks where the last page ended; see Lesson 03).
• Unread count: SELECT count(*) … WHERE user_id = $1 AND read_at IS NULL is cheap for most users because of the small partial index (an index that only covers unread rows). If a user has a very large unread number, keep a counter per user in Redis (a very fast in-memory data store). Update it when a notification is created or read. Cap the display at "99+".
• Read state: use PATCH /notifications/{id} with { read: true }, and POST /notifications/read-all. These updates are idempotent (doing them twice gives the same result as doing them once).
• Retention (how long to keep data): delete or archive old rows. Split the table by month (a partition is a smaller physical piece of one big table) and drop old partitions, which is much faster than deleting rows one by one. In PostgreSQL, a primary key or unique rule on a partitioned table must include the partition column, so you would use PRIMARY KEY (id, created_at) and add created_at to the dedupe rule (or keep dedupe keys in a small separate table).

6. Delivering in-app notifications in real time

The connection gateway

A gateway is a server whose only job is to hold open connections to browsers and pass messages through. A group of gateway servers holds the open SSE or WebSocket connections. The notification system must be able to reach the gateway that holds user 42's connection.

Browser ──SSE──▶ Gateway instance 3          Redis: user:42 → [gateway-3:conn-9, gateway-5:conn-2]                         ▲In-app worker ──publish "user:42" event──▶ Redis Pub/Sub (or NATS/Kafka) ──▶ gateways subscribed for connected users

• When a user connects, the gateway checks who they are (this is authentication, using a cookie or token). Then it subscribes to that user's channel. When the connection drops, it removes the subscription.
• To deliver a message, the worker publishes it to user:{id} on a pub/sub system. Pub/sub (publish/subscribe) is a messaging style where publishers send a message to a named topic and every subscriber of that topic receives it. Redis Pub/Sub does not store messages, so a subscriber that is offline misses them; this is why we also save to the database. Any gateway that holds a connection for that user then pushes the message.
• Keep business logic out of the gateway. It should only pass messages along. Scale it by the number of connections. A well-tuned server can hold tens of thousands, because idle connections are cheap. Set the load balancer for long-lived connections. Send heartbeats (a small comment line, every 15 to 30 seconds) so proxies do not close idle streams. A proxy is a server between the browser and your app.

SSE endpoint (server) and client

// Express SSE endpointapp.get("/v1/notifications/stream", requireAuth, async (req, res) => {  res.set({ "Content-Type": "text/event-stream", "Cache-Control": "no-cache, no-transform", Connection: "keep-alive", "X-Accel-Buffering": "no" });  res.flushHeaders();  const send = (n: Notification) => res.write(`id: ${n.id}\nevent: notification\ndata: ${JSON.stringify(n)}\n\n`);  const unsubscribe = bus.subscribe(`user:${req.user.id}`, send);   // subscribe first, so nothing is lost while we replay  const heartbeat = setInterval(() => res.write(": ping\n\n"), 25_000);  req.on("close", () => { clearInterval(heartbeat); unsubscribe(); });  const lastId = req.header("Last-Event-ID");                       // the browser sends this on reconnect  if (lastId) for (const n of await replayMissed(req.user.id, lastId)) send(n);   // catch up from the DB (rows newer than that id)});

// Browserconst es = new EventSource("/v1/notifications/stream", { withCredentials: true });es.addEventListener("notification", (e) => {  const n = JSON.parse((e as MessageEvent).data);  addToList(n); incrementBadge();});es.onerror = () => { /* the browser reconnects automatically and sends Last-Event-ID */ };

Every event has an id. When the client reconnects, it sends Last-Event-ID. Then the server replays the messages the client missed from the database. Because the server subscribes before it replays, a message can arrive twice, so the client ignores ids it has already shown. This is how you get reliability on top of a connection that can break. The database is the source of truth (the one place whose data we trust when copies disagree). The stream is only the fast path.

7. Reliability: at least once, plus idempotency

Most queues promise at least once delivery: a message is never lost, but it may arrive more than once. A worker may crash after it sends an email but before it confirms the message. Then the queue sends the same message again. To make the user feel it was sent exactly once, do these things:

• Remove duplicates when you create the notification. Give each notification a dedupe_key that is always the same for the same event (for example order.shipped:{orderId}:{userId}). The UNIQUE (user_id, dedupe_key) rule rejects a second row for the same event. Insert with ON CONFLICT (user_id, dedupe_key) DO NOTHING so the repeat is skipped quietly instead of raising an error.
• Make senders idempotent. Save the delivery status for (notification_id, channel) before or while you send. Skip the send if the status is already SENT. Pass an idempotency key (a unique id you send with a request, so the provider can spot a repeat and ignore it) to providers that support it.
• Retry temporary failures with exponential backoff (the wait doubles each time) and jitter (a small random extra wait). Use a dead-letter queue (a separate queue for messages that keep failing) and set an alert on it (Lesson 07).
• Respect provider limits. Slow your workers down with a token bucket (a rate-limit method where each send takes one token and tokens refill at a steady speed; Lesson 02). Then a burst does not cause 429 errors (the HTTP status "Too Many Requests") from the email or push provider.
• Order: the order of notifications per user usually does not matter much. If it does, split the queue by user_id (so one user's messages go in one lane, in order).
• Provider failover (switching to a backup when the main one fails): keep a second email or SMS provider for critical messages.

8. User preferences, batching and fatigue

• Preferences by channel and type. Always allow critical messages (security alerts, receipts), whatever the user chose. Follow the law (unsubscribe links, SMS opt-in).
• Quiet hours use the user's own time zone. Delay push or email until the morning.
• Batching and digests: send "5 people liked your post" instead of five pushes. A digest is one message that sums up several events. Group messages with the same key in a short time window (like a "debounce for notifications"; debounce means waiting for a quiet moment before acting). Hold the first one, collect the rest, then send one summary.
• Frequency caps limit messages per user per hour or day. Priority levels let urgent messages skip batching.
• Channel choice and fallback: send in-app first. If it is still unread after N minutes, send an email. Do not push to a device the user is using right now (check presence, meaning whether they are online).

9. Scenario: a user has 3 tabs open. How many connections do you hold?

The simple answer is three SSE connections, one per tab. Each one receives every notification. This is fine at small scale. If 100,000 users all had three tabs open, that would be 300,000 connections. Your options:

1. Accept it. Idle connections are cheap and it is simple.
2. Leader tab: the tabs talk to each other with BroadcastChannel (a browser API for sending messages between tabs of the same site) or a SharedWorker (one background script shared by all tabs). Only one tab holds the connection and passes events to the other tabs.
3. Use HTTP/2 to avoid the browser limit per origin (an origin is scheme + host + port, such as https://example.com). The six-connection limit on HTTP/1.1 is a real problem when several tabs use SSE.
4. Set a maximum number of connections per user on the gateway and close the oldest one.

Also keep the read state on the server. Then marking a notification as read in one tab updates the other tabs (through the same stream or a BroadcastChannel).

10. Scenario: broadcast to a million users

A broadcast creates a million rows and a million jobs. Fan-out means one message spreading to many receivers. Ways to handle it:

• Fan-out on write: insert one row per user. Reads are fast, but the writes spike. Use batched workers that go through users page by page, and limit their speed to protect the database.
• Fan-out on read: store one broadcast record. Merge it into each user's feed when the user reads the feed. Writes are cheap. Reads need a merge and a read-state for each user.
• A hybrid: use fan-out on write for normal notifications. Use fan-out on read for large broadcasts and for "celebrity" accounts (accounts with huge numbers of followers).
• For push, use topics (FCM topic messaging, where devices subscribe to a named topic and you send once to the topic) instead of sending to each device, where you can.
• Throttle (limit the speed of) and spread out the sends. This avoids a thundering herd (many clients acting at the same moment, for example everyone opening the app after one push).

11. Offline users

• In-app notifications are saved. They show up on the next page load or reconnect (through the Last-Event-ID replay or the list endpoint).
• Push notifications are messages the operating system or browser shows even when your page is closed. Web Push is the browser standard for this: a service worker (a script the browser runs in the background for your site) receives the message, and VAPID (a pair of keys that proves the message comes from your server) identifies the sender. FCM and APNs do the same for mobile apps. Take care of the token lifecycle. A push token is an address that identifies one device or browser for push. Remove invalid tokens when a provider returns an "unregistered" error. Refresh the token each time the app opens. Ask for permission at a good moment, not on the first page load.
• Email and SMS are slower fallbacks.

12. The frontend: the bell

• On load, fetch the list (one page) and the unread count. Subscribe to the stream. Merge new items into the list. Keep the badge in sync.
• Optimistic update: show the item as read in the UI right away, before the server answers, then send the request. If the request fails, undo the change.
• Group and shorten long lists. Show relative times ("5 min ago"). Link each notification to the thing it is about.
• Accessibility: use aria-live="polite" (an HTML attribute that makes a screen reader announce changes in that area when it is idle) so new notifications are announced without being too noisy. Make the dropdown reachable with the keyboard. Manage focus.
• Fallbacks: if the stream keeps failing, switch to polling with backoff (waiting longer between each try). Pause when the tab is hidden. Do not overload the server (Debugging, Lesson 05).

Observability

Observability means being able to see what the system is doing from its logs, numbers (metrics) and traces. Track these numbers: events in, notifications created, deliveries by channel and status, end-to-end latency (from event to delivered), queue depth and age, provider error and bounce rates (a bounce is an email that could not be delivered), DLQ size (a DLQ, dead-letter queue, holds messages that failed too many times), open connections, reconnect rate and unread backlog. Put the notification ID and dedupe key in every log line (Backend, Lesson 05).

Common mistakes

1. Calling providers directly from business code. This ties checkout to an email API.
2. No idempotency, so retries send duplicates.
3. One shared queue for all channels, so one slow provider blocks the rest.
4. Polling every second from every client.
5. Treating the stream as the source of truth, with no replay on reconnect.
6. Ignoring preferences, quiet hours and batching. Users get tired of the messages and opt out.
7. Fan-out per recipient inside the request for broadcasts.
8. Never removing dead push tokens or old notifications.

Interview questions

Key takeaways

• Events in, decisions in the middle, deliveries out. Use queues to separate the parts, with one queue per channel.
• SSE (with Last-Event-ID replay) is a practical real-time channel for server-to-client notifications. The database is the source of truth.
• At-least-once plus idempotency (dedupe keys, delivery records, provider keys) gives the feeling of exactly-once. Use DLQs and backoff.
• Scale the connection gateway on its own. Connect it to the workers with pub/sub keyed by user.
• Plan for preferences, batching, quiet hours, broadcast fan-out and offline users.

What's next

Lesson 06: Design Autocomplete / Typeahead. In this real-time feature the frontend and backend design matter equally: debouncing, caching, accessibility and a trie.