Notification System — Multi-Channel Fanout

The multi-channel fanout architecture is the production-grade approach to notification systems that adds three critical capabilities missing from the single-stage Async Pipeline variant: in-app real-time delivery via WebSocket, per-channel independent scaling, and broadcast notification expansion. This is the architecture used by platforms like Slack, Discord, and WhatsApp where sub-second in-app delivery is a core product requirement.

The most significant addition is the WebSocket channel for in-app notifications. The Async Pipeline variant supports only push (FCM/APN) and email (SendGrid/SES) — both of which require the user to not be actively using the app. When a user is online and looking at the application, the fastest delivery mechanism is a WebSocket push directly to the open browser tab or mobile app. WebSocketService maintains persistent connections with 30-second heartbeats, tracks user presence in a dedicated Redis cache (PresenceCache), and delivers notifications in under 200ms. This is 25x faster than push notifications (5 seconds) and 150x faster than email (30 seconds).

The two-stage Kafka pipeline is the architectural innovation that enables this multi-channel delivery. Stage 1: NotifyService publishes raw notification events to RawStream (Kafka) — each event contains the rendered content, recipient list, and requested channels. The API returns 202 in ~50ms. Stage 2: FanoutWorker consumes from RawStream and expands each event per-recipient and per-channel. For each recipient, it checks user preferences (UserPrefsCache) for opt-out and channel selection, checks presence (PresenceCache) for online/offline status, and publishes per-channel events to the appropriate ChannelStream topic (push-events, email-events, or inapp-events). This two-stage design keeps the API thin (one Kafka message per request) while enabling sophisticated per-recipient routing.

Presence-aware routing is a key differentiator. When the FanoutWorker expands a notification for a specific recipient, it checks PresenceCache to determine if the user is currently connected via WebSocket. If online, the notification is routed to inapp-events for sub-second WebSocket delivery. If offline, it goes to push-events for FCM/APN delivery. This avoids the annoyance of receiving a push notification when you are already looking at the app, and it reduces unnecessary push delivery costs. Users can configure per-channel preferences to receive both channels if desired.

The per-channel architecture enables independent scaling. PushWorker (10 instances) handles ~8K pushes/sec with 200ms per FCM call. EmailWorker (4 instances) handles ~5K emails/sec with batching. WebSocketService (6 pods) supports ~200K concurrent connections. Each channel has its own Kafka consumer group, its own DLQ, its own retry policy, and its own scaling triggers. A SendGrid outage does not affect push or in-app delivery. A spike in push volume does not delay email delivery. This isolation is impossible with a single-stage pipeline where all channels share the same Kafka topic and consumer group.

Broadcast notification expansion is another critical capability. When a notification targets a segment (e.g., all users in a region, all premium subscribers), the raw event contains a segment ID rather than individual user IDs. The FanoutWorker resolves the segment to individual users and expands accordingly — a single raw event can expand to millions of per-channel events. Rate limiting at the FanoutWorker level (50K expansions/sec) prevents this expansion from overwhelming the ChannelStream topics. The Async Pipeline variant does not support broadcast — it requires the caller to enumerate individual recipients.

This is the architecture that senior candidates should propose after discussing the Async Pipeline variant. The progression from sync to async to multi-channel fanout demonstrates deepening understanding of notification system trade-offs. The key additions — WebSocket for real-time, two-stage pipeline for fanout, presence for intelligent routing — are exactly what interviewers at communication platforms like Slack, Twilio, and Discord expect.

The multi-channel fanout system uses 12 components organized into a two-stage Kafka pipeline with per-channel delivery and WebSocket for real-time in-app notifications. The components are: Client, API Gateway, Load Balancer, NotifyService, UserPrefsCache (Redis), TemplateDB (PostgreSQL), RawStream (Kafka), FanoutWorker, PresenceCache (Redis), ChannelStream (Kafka with 3 topics), PushWorker, EmailWorker, and WebSocketService.

The API layer is similar to the Async Pipeline variant but optimized for higher throughput. Client requests enter through the API Gateway (JWT auth, 20K RPS rate limit), pass through the Load Balancer (40K RPS capacity, 2.5x headroom), and reach NotifyService (10 ECS Fargate pods, 100 threads each, 1000 total concurrent capacity). NotifyService validates the request, renders the template from TemplateDB with the caller's variables, and publishes a raw notification event to RawStream (Kafka). The API returns 202 Accepted in approximately 50ms. Notably, NotifyService does NOT check user preferences — preference resolution is deferred to the FanoutWorker where per-recipient expansion happens. This keeps the API thin and fast.

RawStream (Amazon MSK, 8 partitions) carries raw notification events from NotifyService to FanoutWorker. Each event contains the rendered content, recipient list (which may be individual user IDs or a segment ID for broadcast notifications), and requested channels. Partitioned by notification_id for ordering. This is Stage 1 of the two-stage pipeline — the raw, unexpanded notification event that has not yet been resolved to individual per-channel delivery tasks.

FanoutWorker (12 ECS Fargate instances, 4 vCPU/8 GB each) is the core expansion engine that transforms raw notifications into per-channel delivery events. For each raw event consumed from RawStream, it executes four steps: (1) resolves segment IDs to individual user IDs if the notification targets a user segment rather than explicit recipients; (2) checks per-user preferences in UserPrefsCache (Redis) via pipelined MGET for opt-out flags, quiet hours windows, and enabled channel selection; (3) checks presence in PresenceCache (Redis) to determine whether each user is currently online via WebSocket; (4) expands each event into per-recipient x per-channel events and publishes to the appropriate ChannelStream topic (push-events, email-events, or inapp-events). A single raw event targeting 1,000 recipients across 2 channels expands to approximately 2,000 channel events. Rate-limited to 50K expansions per second to prevent overwhelming ChannelStream during large broadcast notifications.

ChannelStream (Amazon MSK, 32 partitions distributed across 3 topics) carries per-channel events to the appropriate delivery workers. Three topics: push-events (16 partitions for the highest-volume channel), email-events (8 partitions), inapp-events (8 partitions). Each topic has its own independent consumer group, enabling independent scaling, independent backpressure handling, and independent dead-letter queues. This is Stage 2 of the pipeline — the expanded, routed notification events ready for final delivery to end users.

PushWorker (10 instances) consumes push-events and delivers via FCM for Android and APN for iOS (~200ms per external call). EmailWorker (4 instances) consumes email-events and delivers via SendGrid/SES (~500ms per batch, batching up to 10 emails per API call for throughput efficiency). Both implement idempotent delivery using a Redis dedup set and exponential backoff retry with DLQ for permanent failures. WebSocketService (6 ECS Fargate pods, 200 threads each, supporting ~200K concurrent connections) consumes inapp-events and pushes notifications over persistent WebSocket connections (~10ms delivery latency). It manages the full connection lifecycle: upgrade, heartbeat (30-second interval), reconnection, and graceful disconnect. On each connect and heartbeat, it updates PresenceCache.

PresenceCache (Amazon ElastiCache for Redis, dedicated 2-node cluster, 13 GB each) tracks which users are currently connected via WebSocket. Key pattern: presence:{user_id} with a 60-second TTL that auto-expires on missed heartbeats. This enables presence-aware routing in FanoutWorker: online users receive in-app delivery via WebSocket (sub-200ms), offline users receive push delivery via FCM/APN (2-5 seconds). The separate Redis cluster prevents volatile presence updates (written every 30 seconds per connected user) from evicting long-lived user preferences stored in UserPrefsCache with 1-hour TTL.