Notification System — Async Pipeline (Priority Queue)

The async pipeline with priority queue architecture is the industry-standard approach for notification systems handling thousands of notifications per second. It solves the two fundamental problems that make the naive synchronous approach unworkable at scale: thread pool saturation from slow third-party calls and lack of retry coordination for transient failures. This is the design that interviewers at companies like Twilio, SendGrid, Slack, and every major mobile-first platform expect candidates to propose after identifying the synchronous bottleneck.

The core architectural shift is decoupling notification acceptance from delivery. Instead of calling FCM or SendGrid synchronously in the API request path (holding a thread for 200-500ms), the service validates the request, checks user preferences, renders templates, and publishes a notification event to Kafka. The API returns 202 Accepted in approximately 50ms — the caller knows the notification has been accepted for delivery, but delivery itself happens asynchronously in background workers. This separation of concerns — acceptance vs. delivery — is a foundational distributed systems pattern.

This decoupling provides three critical benefits. First, API throughput increases from ~500 RPS to 10,000+ RPS because threads are held for only ~50ms (preference check + template render + Kafka publish) instead of 300ms (third-party call). With 800 threads across 8 pods, the throughput ceiling rises dramatically. Second, retry coordination is centralized in the workers. PushWorker and EmailWorker implement exponential backoff on transient failures (FCM token refresh, SendGrid rate limit) and route permanent failures (invalid token, bounced email) to a dead-letter queue. The caller never needs to implement retry logic — the system guarantees delivery or explicit failure via DLQ. Third, the system is resilient to third-party outages — Kafka buffers notifications during outages (millions of messages with multi-day retention), and workers deliver them when the third party recovers. No notifications are lost.

The priority partitioning design is a critical differentiator for notification systems. Without priority separation, a 10-million-recipient marketing blast fills the Kafka topic for minutes. An auth code (time-critical, single recipient) would wait behind millions of marketing messages — a phenomenon known as head-of-line blocking. Priority partitioning solves this by reserving Kafka partitions 0-3 for transactional notifications (auth codes, payment alerts) with dedicated worker capacity that is never shared with marketing. Bulk marketing notifications go to partitions 4-15 with separate consumer groups. This ensures a 2FA code is delivered within seconds regardless of whether a massive marketing campaign is running concurrently.

The system handles 100 million notifications per day at peak throughput of 10,000 per second. Push delivery via FCM/APN takes approximately 200ms per message; email delivery via SendGrid/SES takes approximately 500ms per batch. User preferences (opt-out, quiet hours, channel selection) are cached in Redis with a 98% hit rate, adding only 2ms to the API path. Templates are rendered at API time, so Kafka messages carry the final rendered content and workers do not need database access. Idempotent delivery via a Redis dedup set (notification_id + recipient, 24-hour TTL) prevents duplicate notifications even if Kafka consumer offsets fail.

This architecture is used by virtually every production notification system at scale. The pattern generalizes beyond notifications to any system that needs to decouple fast API responses from slow downstream processing — email, webhooks, data pipeline ingestion, and event-driven microservices all use the same Kafka-backed async pipeline pattern. Interviewers expect candidates to arrive at this design after identifying the synchronous delivery bottleneck in the naive approach and proposing async processing as the core optimization.

The async pipeline system uses 8 components organized into three layers: an API layer that accepts and validates requests, a Kafka event stream for async decoupling, and channel-specific workers for delivery. The components are: Client, API Gateway, Load Balancer, NotifyService, UserPrefsCache (Redis), TemplateDB (PostgreSQL), NotifyStream (Kafka), PushWorker, and EmailWorker.

All traffic enters through the API Gateway (Amazon API Gateway, REST mode), which authenticates service-to-service JWT tokens (~3ms) and enforces per-service rate limits (15K RPS cap). The Load Balancer (AWS ALB) distributes requests across 8 NotifyService pods using round-robin. The LB supports 25K RPS — 2.5x headroom above the 10K peak, ensuring the load balancer is never the bottleneck.

NotifyService is the orchestration layer running on AWS ECS Fargate (4 vCPU, 8 GB per pod, 8 pods with 100 threads each = 800 concurrent threads). For notification sends, it executes a three-step pipeline: (1) check user preferences in UserPrefsCache (Redis) — 2ms cache hit, 5ms cache miss — to enforce opt-out, quiet hours, and channel selection; (2) fetch and render the notification template from TemplateDB (PostgreSQL) with the caller's variables, substituting {{variable}} placeholders with actual values; (3) publish the rendered notification event to NotifyStream (Kafka) with the appropriate priority partition based on the notification type (transactional vs marketing). The API returns 202 Accepted in approximately 50ms. For status queries, it reads the delivery_log table from PostgreSQL read replicas.

UserPrefsCache (Amazon ElastiCache for Redis, 2 nodes, 13 GB each) stores per-user notification preferences with a key pattern prefs:{user_id}. Each entry contains enabled channels, quiet hours window, and opt-out flag. The 98% cache hit rate keeps the average lookup under 2ms. Preferences change infrequently (days/weeks), so the 1-hour TTL rarely causes cache misses for active users. Cache invalidation happens on explicit preference updates via cache-aside write-through. The Zipfian access pattern (alpha 0.99) reflects that a small set of active users receive most notifications.

TemplateDB (Amazon RDS PostgreSQL, db.r7g.large, 2 read replicas) stores two tables. The templates table is small and read-heavy (~1K rows) with high buffer cache hit rate — fetched on every notification send for template rendering. The delivery_log table is append-only, growing at approximately 10K rows per second at peak traffic. PushWorker and EmailWorker write delivery status (sent, delivered, failed) after each delivery attempt. Two read replicas serve status queries without loading the primary, with ~5ms replication lag providing acceptable eventual consistency for status reads.

NotifyStream (Amazon MSK, kafka.m7g.large) is the critical decoupling layer between API acceptance and delivery. 16 partitions with priority separation: partitions 0-3 are reserved for transactional notifications (auth codes, payment alerts) with dedicated worker capacity that is never shared with marketing workloads. Partitions 4-15 handle bulk marketing notifications with separate consumer groups. Messages carry the fully rendered content (not template references), so workers do not need TemplateDB access. Partitioned by recipient_id for per-user delivery ordering. Message retention is set to 7 days, enabling replay if consumers crash.

PushWorker (10 ECS Fargate instances, 2 vCPU/4 GB each) consumes from Kafka and delivers push notifications via FCM for Android and APN for iOS (~200ms per external call). EmailWorker (4 instances, 1 vCPU/2 GB each) delivers email via SendGrid/SES (~500ms per batch, batching up to 10 emails per API call for efficiency). Both workers implement idempotent delivery using a notification_id + recipient dedup key in Redis with 24-hour TTL, retry with exponential backoff (initial 1s, max 5 minutes, 3 retries), and DLQ for permanent failures such as invalid FCM tokens or bounced emails.