Payment System — Idempotent Ledger + Saga (Kafka Orchestration)

The idempotent ledger with saga orchestration is the industry-standard architecture for payment processing systems handling tens of thousands of transactions per second. It solves the two critical problems that make the naive synchronous approach unworkable at scale: the thread-blocking card network bottleneck and the absence of idempotency guarantees that cause double charges.

The key architectural shift is making the card network call asynchronous. Instead of blocking a request thread for 200-500ms waiting for Visa or Mastercard to respond, the charge API writes the payment record to the ledger with status 'pending' and publishes an event to Kafka. A dedicated CardNetworkWorker consumes the event and makes the external call asynchronously. The charge API returns in under 200ms with a payment_id and status 'pending' — the merchant either polls for the result or receives a webhook when the card network responds.

This async processing model transforms the throughput ceiling. The naive approach caps at ~450 RPS because each thread is blocked for 300ms. The saga approach frees the thread immediately after the ledger write (~50ms) and Kafka publish (~5ms), allowing each thread to handle ~15 charges per second instead of ~3. With 15 pods running 100 threads each (1,500 threads), the system sustains 100K+ RPS — a 200x improvement over the naive approach using the same thread model.

The idempotency guarantee is equally critical. Every charge request carries a client-supplied Idempotency-Key header. Before any processing begins, the ChargeService stores this key in Redis using SETNX (set-if-not-exists) — an atomic operation that succeeds only if the key does not already exist. If a merchant retries a timed-out charge, the SETNX fails because the key already exists, and the service returns the previous result. This guarantees at-most-once charging regardless of how many times the merchant retries. Critically, the idempotency key is stored BEFORE the ledger write — if the key were checked after processing, a retry during processing could create a second charge.

The saga pattern provides consistency guarantees across the distributed workflow. When a charge is created, the saga coordinates: idempotency check (Redis) then ledger write (PostgreSQL) then Kafka publish. If the card network declines the charge, the saga compensates: ledger status updates to 'failed', any merchant balance hold is released, and a payment_failed webhook is sent. Each step is independently retryable and idempotent. The saga guarantees that partial failures never leave the system in an inconsistent state — a charge is either fully processed or fully compensated.

This architecture is how Stripe, Adyen, and PayPal actually process payments. The idempotency-first design, async card network routing, and saga compensation are not theoretical — they are battle-tested patterns processing billions of dollars in daily transaction volume. Interviewers at payment companies expect candidates to arrive at this architecture as the natural evolution from the naive approach, and to articulate exactly why each component exists.

The saga-based payment system uses 10 components organized into a synchronous charge path (MerchantClient, ApiGateway, MainLB, ChargeService, IdempotencyCache, LedgerDB) and an asynchronous processing pipeline (PaymentStream/Kafka, CardNetworkWorker, SettlementWorker, WebhookWorker). The RefundService handles refund processing as a separate service.

The charge critical path begins at the MerchantClient, which sends a POST request with an Idempotency-Key header. The ApiGateway authenticates the merchant API key and enforces rate limits (120K RPS cap). MainLB distributes across ChargeService pods. ChargeService executes the synchronous portion: (1) check IdempotencyCache — Redis SETNX with 24-hour TTL, returning the cached result if the key exists; (2) write the charge record to LedgerDB with status 'pending'; (3) publish a charge_created event to PaymentStream (Kafka). The HTTP response returns immediately with payment_id and status 'pending'. This entire flow completes in under 200ms.

The asynchronous pipeline begins when CardNetworkWorker consumes the charge_created event from Kafka. It routes to the appropriate card network (Visa, Mastercard, Amex) based on the card BIN prefix. The external call takes 200-500ms. On success, CardNetworkWorker updates LedgerDB status to 'succeeded' and publishes a payment_result event. On failure, it updates to 'failed' and triggers saga compensation. WebhookWorker consumes payment_result events and delivers them to merchant webhook URLs with HMAC-SHA256 signatures and exponential backoff retry (up to 72 hours). SettlementWorker runs daily batch reconciliation matching charges against card network settlement files.

LedgerDB is the single source of truth — a PostgreSQL database sharded by merchant_id across 64 partitions. The sharding strategy ensures that most queries (which are scoped to a single merchant) hit a single shard. At 100K TPS across 64 shards, each shard handles approximately 1,500 TPS — well within PostgreSQL capacity. The ledger stores payments, refunds, and an append-only audit log retained for 7 years per financial regulations.

IdempotencyCache (Redis) is on the critical path of every charge. Redis SETNX completes in under 1ms, compared to a database check at 10-15ms. At 100K RPS, the Redis approach saves over 1 billion ms of aggregate latency per day. The 24-hour TTL on keys balances storage cost (8.6B keys/day at 100 bytes each is approximately 860GB at peak) against retry safety (most retries happen within seconds, but 24 hours covers edge cases).

The PaymentStream (Kafka) provides durable, ordered, at-least-once delivery. 64 partitions partitioned by merchant_id support 1M messages per second. Consumer groups allow CardNetworkWorker, SettlementWorker, and WebhookWorker to scale independently without interfering with each other. Kafka replay capability means a worker crash does not lose any payment events.