Notification System — Principal-Level Deep Dive (The Stops)¶
The "never be blind" companion to the base Notification System docs. The system as a staircase of stops — smallest defensible design first, each bigger stop with its exact trigger — ending in the top-stop (max-scale / deliberately over-engineered) architecture plus a reconciliation table for when each heavy component is overkill and should be removed.
Read the base first: Requirements · Architecture · Deep Dives · Trade-offs.
Say this in an interview: "I'll start with an async send behind a queue and grow it one bottleneck at a time. I know the multi-region, multi-provider, ML-ranked, fully-instrumented version — but each piece only earns its place when a specific ceiling forces it."
The stops (climb one lever at a time)¶
| Stop | Scale target | Shape | Trigger that forces the NEXT stop |
|---|---|---|---|
| 1 · Inline send | tiny; one service, one channel | call the provider SDK in the request | provider latency/failures block callers; no retries; can't burst |
| 2 · Async + workers | ~1k sends/s | ingestion API (202) → queue → workers → provider adapters; retries → DLQ |
broadcasts amplify; OTPs stuck behind bulk; provider throttling |
| 3 · Fan-out + lanes (base target) | ~50k sends/s peak | fan-out stage + per-channel + per-priority queues + preference cache + provider rate limiting + dedup | provider outages; status-write volume; notification fatigue; global users |
| 4 · Multi-provider, multi-region, ranked (top stop) | millions/s bursts, global | multi-provider orchestration + multi-region ingestion/queues + delivery-tracking stream pipeline + aggregation/ML-ranked digests + strict per-tenant isolation | (ceiling; a true ranked feed becomes a feed system, not a notifier) |
Most interviews want Stop 2–3. Reach into Stop 4 only when the prompt says "global / multi-provider / real-time analytics / billions of notifications." Name the trigger each time.
Top-stop architecture (max-scale)¶
events → [ Ingestion API (multi-region) ] validate·dedup(event_id)·persist → 202
│ enqueue (region-local durable queue / Kafka)
[ Fan-out fleet ] stream recipients in batches · prefs/quiet-hours/opt-out · render (i18n)
│ route by CHANNEL × PRIORITY × REGION
┌──────────────┼───────────────┬───────────────┐
push lanes email lanes sms lanes in-app (×2 transactional|bulk each)
│ │ │ │
channel workers (rate-limited per provider, circuit breaker)
│ PROVIDER ORCHESTRATION: primary→failover, cost/deliverability-based routing
▼
APNs/FCM · SES/SendGrid/Mailgun · Twilio/SNS/MessageBird (multiple per channel)
│ delivery/bounce/open webhooks
▼
Delivery-tracking STREAM pipeline: Kafka → Flink → ClickHouse (status, deliverability, dashboards)
│ feedback → suppression lists, provider-health-based routing, ML send-time/ranking
Per-tenant isolation + quotas + fatigue caps
Deep dives that only matter at the top stop¶
Multi-provider orchestration (Stop 3 → 4 trigger: provider outage / rate-limit ceiling / deliverability)¶
- Why: a single provider is an outage SPOF and a hard rate ceiling; deliverability varies by provider/region.
- Design: several providers per channel behind the uniform channel abstraction; route by health (circuit breaker), cost, and deliverability; fail over automatically; split load to stay under each provider's quota.
- Over-engineering watch: only when a single provider's outages/limits actually bite. Early on, one provider + buffered retries is simpler; remove the orchestration layer below that.
Multi-region ingestion & queues (trigger: 99.999% + global senders/recipients)¶
- Why: regional SPOF; global producers want low-latency ingestion; some data must stay in-region (GDPR).
- Design: region-local ingestion + queues; providers are global; the durable queue means accepted notifications aren't lost on region failure; route ingestion by geo with failover.
- Over-engineering watch: only for five-nines / global / data-residency. Single-region multi-AZ is plenty for most; remove multi-region below that.
Delivery-tracking stream pipeline (trigger: status-write volume + real-time deliverability)¶
- Why: at millions of sends, provider webhooks (delivered/bounced/opened) are a huge write + analytics stream; you need real-time deliverability/bounce signals to protect sender reputation and drive failover.
- Design: webhooks → Kafka → Flink → ClickHouse; feed suppression lists (auto-suppress hard bounces/ complaints) and provider-health routing; dashboards for delivery rate. Keep it off the send critical path.
- Over-engineering watch: only when status volume + real-time deliverability matter. Below that, async status writes to an append store + TTL suffice; remove the streaming pipeline.
Aggregation / ML ranking (trigger: notification fatigue at scale)¶
- Why: high-frequency notifications spam users → disengagement, cost, provider load.
- Design: digest/aggregation windows ("10 people liked your post"), per-user fatigue caps, and — at the extreme — ML-ranked send-time optimization (when is this user reachable/receptive). Digests via a buffering window or the job scheduler.
- Over-engineering watch: only when fatigue is a measured problem. Simple category prefs + rate caps go a long way; a full ML-ranked pipeline is a feed system, not a notifier — if you need that, build the feed. Remove ML ranking below real fatigue signals.
Strict per-tenant isolation & quotas (trigger: multi-tenant, noisy neighbor)¶
- Why: one tenant's 50M campaign must not degrade another tenant's OTPs.
- Design: per-tenant rate limits, quotas, and (at the extreme) dedicated lanes/pools for large tenants; fairness scheduling across the shared channel capacity.
- Over-engineering watch: only for real multi-tenant contention. Single-tenant/internal use doesn't need it; remove the isolation layer.
Reconciliation table — what forces each heavy component (and when to remove it)¶
| Component | Base docs (Stop 2–3, ~50k/s) | Top stop (Stop 4) | Trigger to ADOPT | When it's OVERKILL → remove |
|---|---|---|---|---|
| Providers | primary + one failover | multi-provider health/cost/deliverability routing | single provider's outages/limits bite | one provider + retries is enough |
| Regions | single-region multi-AZ | multi-region ingestion/queues | 99.999% / global / data-residency | 99.99% target |
| Delivery tracking | async writes → append store + TTL | Kafka→Flink→ClickHouse stream | status volume + real-time deliverability | modest volume; batch status is fine |
| Fatigue control | category prefs + per-user caps | digests + ML send-time/ranking | measured notification fatigue | low frequency; prefs suffice |
| Tenant isolation | per-tenant rate limits | dedicated lanes + fair scheduling | multi-tenant noisy-neighbor | single/trusted tenant |
| Queue | durable queue per lane | Kafka partitioned lanes | replay / very high fan-out | plain queue is enough |
Lead with Stop 2–3 (async fan-out + separate channel/priority lanes + dedup + provider rate limiting). When pushed: "for global, multi-provider, billions-of-notifications with real-time deliverability and anti-fatigue, here's Stop 4 — each piece maps to a ceiling, and below that ceiling I'd remove it."
The 60-second Principal summary¶
"It's an async fan-out pipeline. I start with an ingestion API that dedups and returns 202, a queue, and workers calling provider adapters with retries and a DLQ. Next I add the fan-out stage, per-channel + per-priority lanes so an OTP never waits behind a marketing blast, a preference/quiet-hours/opt-out check at fan-out, provider rate limiting, and a dedup key for exactly-once effect — that's ~50k sends/s. Pushed to global scale I climb one lever at a time: multi-provider orchestration when a single provider's outages or limits bite, multi-region ingestion for five-nines and data residency, a Kafka→Flink→ClickHouse delivery-tracking pipeline when status volume and real-time deliverability matter, and aggregation/ML ranking only when fatigue is a measured problem — at which point it's really a feed system. Each piece has a named trigger, and below it I'd remove it."
← Back to Notification System overview · Base deep dives · cheatsheet