Messaging & Streaming

Pub/Sub (publish-subscribe) and message queues are two fundamental asynchronous messaging patterns. Pub/Sub fans out every message to all subscribers, enabling event-driven architectures where producers are fully decoupled from consumers. Message queues deliver each message to exactly one consumer, enabling work distribution and load leveling. Choosing the right pattern shapes your system's coupling, scalability, and failure semantics.

Apache Kafka is a distributed, durable, high-throughput event streaming platform built around an append-only commit log. Its architecture of brokers, topics, partitions, and consumer groups enables millions of events per second with strong ordering guarantees per partition, configurable durability, and the ability to replay the full event history.

Amazon SQS (Simple Queue Service), SNS (Simple Notification Service), and EventBridge are three AWS messaging services that serve different roles. SQS is a fully managed message queue for point-to-point work distribution. SNS is a pub/sub notification service for fan-out. EventBridge is a serverless event bus with content-based routing. Understanding when to use each -- and how to combine them -- is essential for AWS-based architectures.

Delivery semantics define how many times a message is guaranteed to be delivered and processed: at-most-once (may lose messages), at-least-once (may duplicate messages), or exactly-once (neither lost nor duplicated). Understanding these guarantees is critical for designing reliable distributed systems, as they determine whether your application needs deduplication, idempotency, or can tolerate data loss.

An idempotency key is a unique identifier attached to a request or message that allows the receiver to detect and safely handle duplicates. If the same key is seen twice, the receiver returns the cached result of the first processing instead of executing the operation again. Idempotency keys are the practical mechanism that turns at-least-once delivery into effectively exactly-once processing.

Ordering guarantees determine whether messages are delivered and processed in the same order they were sent. Total ordering means all consumers see all messages in the same global sequence. Partition ordering means messages with the same key are ordered within a partition. No ordering means messages may arrive in any order. Stronger ordering guarantees reduce parallelism and throughput.

A Dead Letter Queue (DLQ) is a special queue that receives messages that cannot be successfully processed after a configured number of retry attempts. Instead of blocking the main queue or silently dropping failed messages, DLQs capture them for inspection, debugging, and eventual reprocessing. DLQs are a critical reliability pattern in any production messaging system.

The Outbox Pattern ensures reliable message publishing by writing the message to an 'outbox' table in the same database transaction as the business operation. A separate process reads the outbox table and publishes messages to the message broker. This eliminates the dual-write problem where a database commit succeeds but the message publish fails (or vice versa), ensuring atomicity between state changes and event publication.

Windowing divides an unbounded event stream into finite, time-bounded segments (windows) for aggregation and analysis. The three fundamental window types are tumbling (fixed, non-overlapping), hopping (fixed, overlapping), and session (dynamic, activity-based). Windowing is essential for computing time-based metrics like 'clicks per minute,' 'average latency over 5 minutes,' or 'purchases per user session.'