Design a Notification System

The problem

Every product sends notifications: "your order shipped", "your code is 482913", "new message from Sam", "20% off this weekend". A notification system is the one service the rest of the company calls to send them, across push (iOS and Android), email and SMS, without each team rebuilding retries, preferences and provider integrations.

It looks like plumbing, but it is full of real trade-offs: urgent and bulk traffic share infrastructure, every send goes through an outside provider with its own limits, and a duplicate or a message at 3 a.m. annoys a real person.

Step 1: Requirements

Try it first: what would you put on each list?

Functional:

• Other services send a notification for a user and an event type ("order_shipped", "password_reset").
• Delivery over push, email and SMS, chosen by event type and user preferences.
• Users can opt out per type and channel; marketing must honor unsubscribe.
• Templates per event type and language.
• Delivery status is tracked (sent, delivered, failed, opened where available).

Non-functional:

• Transactional messages (password resets, security codes) arrive within seconds, even during a big campaign.
• No duplicates from retries; no lost messages.
• Scales to tens of millions of notifications a day with spikes.
• Respects provider rate limits and keeps sender reputation healthy.

Step 2: Estimates

• Volume: 50 million notifications a day ≈ 600 per second on average.
• Spikes: a campaign of 10 million emails sent within an hour is ≈ 2,800 per second on top of normal traffic.
• Mix: most messages are push; email is the most bulky; SMS is the most expensive per message, so it is reserved for codes and critical alerts.
• Storage: a status row per notification (~500 bytes) is 25 GB a day; keep detailed history for weeks, aggregates longer.

The point of the numbers: the system must absorb bursts many times the average without delaying the urgent few.

Step 3: Core entities and API

notifications  (id, user_id, type, channel, status, idempotency_key, created_at)
preferences    (user_id, type, channel, enabled, quiet_hours)
devices        (user_id, platform, push_token, last_seen)
templates      (type, channel, locale, body)

POST /notifications
  { "user_id": "u42", "type": "order_shipped", "data": { "order": "A-1001" },
    "idempotency_key": "order:A-1001:shipped" }
  → 202 Accepted { "id": "n_789" }

The API returns 202 Accepted: the request is stored and will be sent asynchronously. Callers do not wait on providers.

Step 4: High-level design

The API checks the request, looks up preferences (is this type enabled on this channel?), records it, and enqueues it. Workers render the template in the user's language and call the provider. That alone decouples callers from slow or failing providers.

Step 5: Deep dive — priorities and isolation

Bad: one queue for everything. A 10-million-message campaign sits in front of every password reset. Transactional messages wait for the backlog to drain, and users cannot log in.

Good: separate queues per priority. Transactional messages get their own queue and workers; marketing gets another. A backlog in one cannot delay the other.

Great: separate by priority and by channel, with limits per provider. Email, push and SMS each have their own queues and worker pools, because each provider fails and rate-limits independently: if the SMS provider slows down, push is unaffected. Workers enforce each provider's sending rate (a token bucket per provider) and the campaign tier is throttled deliberately, so a campaign never pushes a provider into rejecting transactional traffic. This is the bulkhead idea from Failures, Retries and Rate Limits, applied to queues.

Step 6: Deep dive — duplicates, retries and expiry

Duplicates. Callers retry on timeouts, and queues deliver at least once. So the API requires an idempotency key per event (for example order:A-1001:shipped) and stores keys it has accepted; a repeat returns the original response instead of creating a second notification. Workers mark each notification as sent before acknowledging the queue message, and check that mark before sending, so a redelivered message is skipped. Push providers also offer a collapse identifier (apns-collapse-id on APNs; collapse keys on FCM) so a newer update replaces an older one on the device.

Retries. Provider calls fail for two kinds of reasons:

• Temporary (timeouts, rate limiting, 5xx): retry with exponential backoff and jitter, then send to a dead-letter queue after a few attempts.
• Permanent (invalid address, unsubscribed, or an invalid push token, which FCM reports as NotRegistered and APNs with status 410): do not retry. Mark the token or address invalid so it is never used again.

Expiry. A "your driver has arrived" push is useless an hour later. FCM, for example, stores undelivered messages for up to 28 days by default. Set a short time-to-live on time-sensitive messages so the provider drops them rather than delivering them late; the same goes for your own queues (skip messages that are older than their deadline).

Going deeper: preferences, quiet hours and email reputation

Check preferences twice: when accepting (cheap rejection) and again just before sending, since a user may opt out while a campaign is queued. Quiet hours (no marketing pushes at night in the user's time zone) mean some messages are scheduled rather than sent: store a send-after time and have a scheduler enqueue them later.

Email deliverability is a shared asset: if users mark campaigns as spam, transactional mail from the same domain starts landing in spam too. Teams often send transactional and marketing mail from different subdomains or IP pools. Gmail requires senders of more than 5,000 messages a day to support one-click unsubscribe (RFC 8058) on marketing and subscribed messages, and honoring it promptly matters.

Step 7: Scaling and the final design

• Workers are stateless and scale with queue depth, separately per queue.
• The status store is write-heavy: partition by time (and user), keep recent detail, aggregate the rest.
• Device tokens and preferences are read on every send: cache them, invalidate on change.

What interviewers expect at each level

• Early career: an API, a queue, workers and providers; preferences and templates mentioned; asynchronous sending explained.
• Mid-level (2–5 years): priority separation, idempotency keys, retries with backoff and a dead-letter queue, handling invalid tokens, and rough capacity numbers.
• Senior: isolation per channel and provider with rate limits, expiry for time-sensitive messages, preference checks at send time, quiet hours and scheduling, email reputation, and the metrics that matter (time to deliver per priority, provider error rates, queue age).

Remember this

• Accept fast (202), send asynchronously, and never let a campaign share a queue with password resets.
• Idempotency keys stop retries from becoming duplicate notifications.
• Retry temporary failures with backoff; never retry permanent ones such as invalid tokens.
• Give time-sensitive messages a short time-to-live.
• Isolate channels and providers, and rate-limit each one, so one slow provider cannot stall the rest.