Hotspot Mitigation

In any partitioned system, the theoretical promise of uniform load distribution often collides with the reality of skewed access patterns. A hotspot forms when one partition receives significantly more read or write traffic than its peers, becoming a throughput bottleneck that limits the scalability of the entire system. The system is only as fast as its slowest partition, so a single hotspot can negate the benefits of partitioning across dozens or hundreds of otherwise healthy nodes. Hotspots are one of the most common and impactful operational problems in distributed databases, caches, and message queues.

The causes of hotspots fall into three main categories. First, the celebrity problem (also called the thundering-herd or viral-content problem): when a single key receives massively disproportionate traffic. A tweet from a celebrity with 100 million followers generates millions of read requests for a single partition key. A viral product listing on an e-commerce site concentrates all traffic on the partition owning that product ID. Second, temporal patterns: in range-partitioned systems, all writes for 'today' go to the partition holding the current time range, while historical partitions sit idle. Third, poor shard key selection: choosing a low-cardinality key (country code, status enum) or a highly correlated key (department_id in a company where 80% of employees are in engineering) creates permanent structural imbalance.

Salting is the most common write-side mitigation technique. By appending a random suffix (e.g., a number from 0-9) to the key before hashing, writes for a single logical key are scattered across multiple partitions. Instead of all writes to celebrity_tweet_123 hitting one partition, writes go to celebrity_tweet_123_0 through celebrity_tweet_123_9 across 10 partitions. The tradeoff is that reads must now query all 10 salted variants and merge the results. This is acceptable when writes are the bottleneck (which they usually are for hot keys) and reads can tolerate the fan-out overhead. Instagram used this technique to handle time-bucketed partition hotspots by adding a shard suffix to their partition keys.

Dynamic partition splitting is the system-level solution. CockroachDB detects hot ranges by monitoring per-range query rates and automatically splits them, distributing the load across two new ranges on different nodes. DynamoDB's adaptive capacity feature (introduced in 2019) isolates hot partition keys by splitting the underlying physical partition and dedicating throughput to the hot key. On the read side, caching is the first line of defense: placing a cache (Redis, Memcached) in front of the database absorbs repeated reads for hot keys before they reach the partition. Application-level routing takes this further by maintaining a list of known hot keys and routing them to dedicated read replicas or specialized caching tiers, keeping hot-key traffic entirely separate from normal-path queries.

A grocery store has 10 checkout lanes. Normally, customers distribute themselves evenly. But one day, a celebrity walks in and everyone rushes to lane 5 to take photos, blocking the lane completely. The other 9 lanes are empty, but the store is effectively at a standstill because lane 5 has a 200-person queue. The store manager has several options: (1) Salting -- hand out random lane tickets so fans are spread across all 10 lanes. (2) Splitting -- open 3 additional express lanes just for the celebrity crowd. (3) Caching -- put up a photo of the celebrity at the entrance so people take their selfie there instead of blocking the checkout. (4) Routing -- create a dedicated VIP lane for the celebrity, keeping normal lanes clear. Each solution trades some convenience for better overall throughput.