Notification System — Naive (Synchronous Send)

Designing a notification system is one of the most commonly asked system design interview questions because it touches on asynchronous processing, multi-channel delivery, retry strategies, and priority management. The naive synchronous approach is the natural starting point — it establishes the baseline that makes the improvements in async pipeline architectures measurable and concrete. Companies like Twilio, SendGrid, Slack, OneSignal, and virtually every mobile-first platform ask this question because notification delivery is a core infrastructure challenge.

The core challenge is delivering notifications across multiple channels (push, email, SMS) reliably and at scale. In the naive approach, the NotifyService receives a notification request, validates the payload, calls the appropriate third-party API (Firebase Cloud Messaging for push notifications, SendGrid for email) synchronously, waits for the response, records the delivery status in PostgreSQL, and returns the result to the caller. This is the simplest possible implementation — no message queues, no background workers, no retry coordination. The caller sends a request and gets back a definitive answer: the notification was delivered, or it was not.

The synchronous delivery model's fatal flaw is thread pool saturation. Each notification request holds a service thread for 200-500 milliseconds while waiting for the third-party API to respond. FCM calls average 200ms; SendGrid calls average 500ms. With 4 pods running 50 threads each (200 threads total) and a 300ms average hold time, the theoretical ceiling is approximately 666 requests per second. In practice, the ceiling is lower (~500 RPS) due to thread scheduling overhead and the occasional slow third-party response (p99 can reach 2 seconds). Above this threshold, requests queue behind occupied threads, and API latency climbs from 350ms to multiple seconds. The system does not fail gracefully — it degrades continuously, with every pending request contributing to the backlog.

The second critical weakness is the lack of retry coordination. Third-party APIs experience transient failures — FCM returns 503 during capacity issues, SendGrid rate-limits at high volume, APN tokens expire and need refresh. In the synchronous model, these failures are returned directly to the caller as 502 errors. The notification is lost unless the caller implements its own retry logic with idempotency keys, exponential backoff, and maximum retry limits. This pushes retry complexity to every upstream service that sends notifications, violating the single-responsibility principle and leading to inconsistent retry behavior across callers. The Async Pipeline variant centralizes retry with exponential backoff and dead-letter queues, ensuring uniform delivery guarantees regardless of which service triggered the notification.

The third weakness is cascading failures during third-party outages. If SendGrid goes down for 5 minutes, every email notification request during that window fails with a 502 error and occupies a thread for the full timeout duration (typically 5-10 seconds). This thread exhaustion cascades to push notifications too, since they share the same thread pool. A single-channel outage becomes a total system outage — users cannot receive push notifications because all threads are stuck waiting for email timeouts. The Async Pipeline variant uses Kafka to buffer notifications during outages, delivering them when the third-party recovers, and separate worker pools per channel isolate failures.

This template makes these failure modes visible and quantifiable. Run the simulation at increasing RPS and watch thread pool utilization climb toward 100%. Inject a simulated third-party outage and observe how the entire service degrades within seconds. The comparison with the Async Pipeline and Multi-Channel Fanout variants provides the concrete numbers to support the discussion of sync-vs-async trade-offs that interviewers expect. Understanding why synchronous delivery fails is the first step toward designing production-grade notification infrastructure.

The naive notification system is a five-component linear architecture: Client, API Gateway, Load Balancer, NotifyService, and PostgreSQL database. There is no message queue, no background workers, no cache layer, and no separation between the notification acceptance path and the delivery path. This simplicity is both the strength and the weakness of the design.

All traffic enters through the API Gateway (Amazon API Gateway, REST mode), which authenticates service-to-service JWT tokens (~3ms) and enforces per-service rate limits (2K RPS cap). Authenticated requests are forwarded to the Load Balancer (AWS Application Load Balancer), which distributes across NotifyService pods using round-robin. The Load Balancer supports 5K RPS — well above the system's actual ceiling, which is determined by thread pool saturation in NotifyService. Both notification sends and status queries flow through the same gateway, LB, and service — there is no CQRS split, no separate read/write paths, and no routing by operation type.

NotifyService is a stateless REST API running on 4 AWS ECS Fargate pods (2 vCPU, 4 GB each) with 50 threads per pod (200 total threads). It handles two operations: (1) notification send — validate the request payload, determine the target third-party API based on channel type, call FCM (for push, ~200ms) or SendGrid (for email, ~500ms) synchronously, wait for the complete response, record delivery status in PostgreSQL, and return the result to the caller; (2) status query — read delivery records from PostgreSQL by notification_id and return the current status. The critical bottleneck is the send operation: each thread is occupied for the full duration of the third-party call plus the database write, making thread count divided by per-request hold time the hard throughput ceiling. No connection pooling to third-party APIs is implemented — each request opens a fresh HTTP connection.

PostgreSQL (Amazon RDS, db.r7g.large) stores a single table: notifications. Each notification send results in an INSERT containing the notification_id (UUID primary key), channel, recipient, subject, body, status (sent or failed), error_message (on failure), created_at, and delivered_at timestamps. Status queries use a B-tree index on notification_id for O(log N) lookups. At low volume (under 500 writes per minute), the database is not a bottleneck — it handles both reads and writes with substantial headroom. A single read replica serves status queries to avoid loading the primary with read traffic. The table grows at approximately 30 MB per day at 500 notifications per minute, including index overhead.

The system has minimal redundancy. There is no cache layer — every status query hits PostgreSQL directly. There is no message queue — every notification is delivered synchronously in the request path. There is no deduplication mechanism — if a client retries a timed-out request, the recipient may receive duplicate notifications because the service has no way to check whether the first attempt succeeded. There is no user preference checking — the caller is entirely responsible for determining the correct channel, verifying the recipient has not opted out, and respecting quiet hours. These missing features are exactly what the Async Pipeline variant adds.

The concrete scaling ceiling is approximately 500 RPS sustained. At this point, 200 threads with a 300ms average hold time support 666 theoretical RPS, but thread scheduling overhead, garbage collection pauses, and occasional slow third-party responses (p99 reaching 2 seconds) reduce the practical limit. Above 500 RPS, thread pool utilization hits 100%, incoming requests queue behind occupied threads, and p99 latency climbs from 800ms to 5+ seconds. The system does not fail gracefully — it degrades continuously as more requests pile up, and there is no backpressure mechanism to reject excess load cleanly.