Distributed Cache — Consistent Hashing

Distributed caching is one of the most frequently asked system design interview topics because it underpins the performance of virtually every large-scale web application. Companies like Facebook, Twitter, and Netflix rely on distributed caches to serve billions of requests per day with sub-millisecond latency. Without a caching layer, databases would buckle under the read load, and user-facing latency would spike from single-digit milliseconds into the hundreds.

The core challenge is distributing cache entries across multiple nodes while maintaining balanced load, minimizing disruption when nodes are added or removed, and preventing catastrophic failure scenarios like thundering herds. A naive modulo-based sharding strategy rehashes every key when the cluster size changes, causing a massive cache miss storm that can take down the origin database. Consistent hashing solves this by only remapping approximately 1/N of keys when a single node changes, but introduces its own complexity around virtual nodes, ring management, and hot key handling.

At production scale, a distributed cache cluster might handle 10 million operations per second across hundreds of terabytes of aggregate memory. The system must maintain a 92% or higher hit rate to keep database load manageable, deliver sub-millisecond p99 latency on cache hits, and gracefully degrade when individual shards fail. Additional concerns include cache warming on cold starts, TTL-based expiration, and write-through versus cache-aside consistency patterns.

This template models a Memcached/Redis-style cache with a proxy-based consistent hash ring, three independent Redis shards, an origin PostgreSQL database for cache misses, and a warmup worker for cold start mitigation. It covers the full spectrum of caching concerns that interviewers expect candidates to address.

The architecture follows a proxy-based consistent hashing model with a clear separation between the routing layer, the cache data layer, and the origin database. Client applications send cache GET, SET, and DELETE requests through an API Gateway that handles authentication and rate limiting, then through a load balancer that distributes traffic to the CacheProxyService fleet.

The CacheProxyService is the brain of the system. It maintains a consistent hash ring with 150 virtual nodes per physical shard (450 total), ensuring balanced key distribution within 5% variance. When a request arrives, the service hashes the key using MurmurHash3, locates the owning shard on the ring, and routes the request accordingly. On a cache hit (approximately 92% of reads), the value is returned in under 1 millisecond. On a miss, the proxy falls through to OriginDB (PostgreSQL), fetches the authoritative value, backfills the cache shard, and returns the result. Single-flight coalescing ensures that if 100 concurrent requests miss on the same key, only one database query is issued.

The three Redis shards (cache.r7g.xlarge, 26 GB each) are deployed across different availability zones for fault isolation. Each shard operates independently, holding roughly 33% of the keyspace. There is no cross-shard replication, which means a shard failure results in temporary cache misses for that partition until the replacement warms up. The AdminStream (Kafka) carries operational events such as shard additions and warmup requests. The WarmupWorker consumes these events and pre-populates cache shards by scanning OriginDB for hot keys, reducing the cold-start miss storm from minutes to seconds.

The write path follows the cache-aside pattern: the upstream application writes to OriginDB first, then the CacheProxyService invalidates or updates the corresponding cache entry. This approach is simpler and more resilient than write-through, because if the cache is temporarily unavailable, reads simply fall through to the database without data loss.