Payment System — Multi-Region Distributed (Active-Active with Fraud ML)

The multi-region distributed payment system is the production-grade architecture used by platforms processing billions of dollars in daily transaction volume — Stripe, Adyen, PayPal, and Square. It solves three critical limitations of the single-region saga approach: single-region failure vulnerability, absence of fraud detection, and lack of PCI-compliant card data isolation.

The most impactful addition is the FraudService, which scores every charge using a gradient-boosted ML model trained daily on historical fraud data. The model evaluates velocity features (how many charges on this card in the last hour), geo anomaly features (is the card being used in a different country than its billing address), amount deviation features (is this charge significantly larger than the cardholder's typical spend), and device fingerprint features (has this device been associated with previous fraud). Each scoring takes 20-50ms — an acceptable trade-off when it blocks 2-5% of charges as fraudulent. At $50 average charge and 100K TPS, that prevents $500K to $1.25M in fraud per day. The ROI on 30ms of latency is extraordinary.

PCI-DSS Level 1 compliance requires that any system storing or transmitting raw card numbers undergoes a full compliance audit — expensive, time-consuming, and scope-expanding. The CardVault solves this by isolating all card data in a dedicated service with HSM-backed encryption. Raw card PANs enter CardVault and never leave; all other services handle only non-reversible tokens. This means only CardVault (one small service) is in PCI scope, while the entire rest of the payment platform operates outside PCI scope using tokens. This architectural decision saves millions in annual compliance costs.

The double-entry accounting ledger ensures financial integrity. Instead of recording a single 'charge' entry, every financial movement creates two entries: a debit from the customer's account and a credit to the merchant's account. The sum of all entries in the ledger is always zero — any imbalance is a critical alert indicating a bug or fraud. This is not optional for regulated financial services; it is a legal requirement in most jurisdictions. The trade-off is 2x write amplification: 100K charges per second becomes 200K ledger entries per second.

Multi-region active-active deployment ensures the system survives a full region failure. Traffic is routed to the nearest healthy region by the API Gateway. LedgerDB uses cross-region async replication with conflict resolution. The idempotency layer (Redis) prevents duplicate charges even when the same request is routed to different regions during a failover. The cost is operational complexity: 12 components across 2 regions with distributed saga coordination, ML model deployment, and cross-region data consistency.

This is the architecture senior candidates are expected to propose in system design interviews at payment companies. The progression from naive (synchronous, no safety) to saga (async, idempotent) to distributed (multi-region, fraud-aware, PCI-compliant) demonstrates the layered thinking that distinguishes senior engineers from mid-level candidates.

The distributed payment system uses 11 components organized into a synchronous saga path (MerchantClient, ApiGateway, MainLB, PaymentOrchestrator, CardVault, FraudService, LedgerDB) and an asynchronous processing pipeline (PaymentStream/Kafka, NetworkWorker, WebhookWorker, ReconciliationDB). The architecture is deployed active-active across us-east-1 and us-west-2.

The charge critical path begins at the ApiGateway, which authenticates the merchant API key, enforces rate limits (250K RPS), and routes to the nearest healthy region. MainLB distributes across PaymentOrchestrator pods. The orchestrator executes a four-step distributed saga. Step 1: send the raw card PAN to CardVault for tokenization. CardVault encrypts the PAN with AES-256-GCM using HSM-managed keys and returns a non-reversible token. Step 2: send charge details to FraudService for ML scoring. The gradient-boosted model returns a risk score (0-100) and decision (allow/block/review). Charges scoring above 80 are blocked immediately, and the saga compensates by releasing the card token. Step 3: write double-entry ledger entries to LedgerDB — DEBIT customer_account and CREDIT merchant_account in a single SQL transaction. Step 4: publish a charge_authorized event to PaymentStream (Kafka). The HTTP response returns with payment_id and status 'pending'.

The asynchronous pipeline processes the card network authorization. NetworkWorker consumes charge_authorized events, calls CardVault's detokenize endpoint to get the card BIN for network routing, and makes the external card network call (200-500ms). On success, it updates LedgerDB and publishes payment_result. WebhookWorker delivers events to merchant webhook URLs with HMAC-SHA256 signatures for authenticity verification. ReconciliationDB ingests daily card network settlement files and runs batch matching against LedgerDB records — flagging discrepancies (amount mismatches, chargebacks, missing charges) for manual review.

LedgerDB is sharded by merchant_id across 128 partitions with 3 synchronous replicas per shard. The double-entry journal (ledger_entries table) is the authoritative financial record — every debit has a corresponding credit, and the sum of all entries is always zero. Cross-region async replication enables read queries from both regions, while writes are routed to the primary region for the shard. Conflict resolution uses idempotency keys for charge deduplication and last-writer-wins for status updates.

CardVault is a deliberately small, tightly scoped service. It handles only two operations: tokenize (raw PAN in, token out) and detokenize (token in, BIN and network out — never the full PAN). It runs in an isolated VPC with no internet access, mTLS on all connections, and HSM-backed key management. This isolation means only CardVault undergoes PCI-DSS audit, saving the rest of the platform from PCI scope.

FraudService runs on 10 pods with 100 threads each, processing 100K+ scorings per second. The ML model (gradient-boosted trees, not deep learning — inference runs on CPU without GPU) is retrained daily on a labeled dataset of confirmed fraud cases. Velocity features are computed from an in-memory sliding window; geo features use a Redis-backed IP geolocation cache. The 30ms average scoring time is the primary latency addition compared to the Saga variant.