Load Shedding

Load shedding is the practice of intentionally rejecting incoming requests when a system is at or approaching its capacity limits. The fundamental insight is counterintuitive: it is better to reject some requests outright and serve the remaining requests well than to accept all requests and serve them all poorly. Without load shedding, an overloaded system exhibits a characteristic failure pattern: as load increases beyond capacity, response latency increases for all requests, timeouts begin firing, clients retry (adding more load), latency increases further, and the system enters a death spiral where it spends most of its resources on requests it will never finish. Load shedding breaks this pattern by maintaining a clear boundary between 'in capacity' (serve well) and 'over capacity' (reject fast).

The simplest load shedding strategy is random rejection: when the system detects overload (CPU > 80%, queue depth > threshold), it randomly rejects a percentage of incoming requests with HTTP 503 or 429, along with a Retry-After header. This is fair -- no client is systematically penalized -- but undiscriminating: a critical checkout request and a low-priority analytics ping are equally likely to be rejected. Priority-based shedding improves on this by assigning priority levels to different request types and shedding from the lowest priority first. Batch jobs are shed before search queries, search queries before product browsing, and product browsing before checkout. This ensures the highest-value user journeys remain functional even during overload.

More sophisticated approaches use queue theory to make shedding decisions. LIFO (Last In, First Out) queue processing ensures that when the queue is deep, the oldest requests (which have likely already timed out on the client side) are the ones discarded, while fresh requests are processed first. This avoids wasting compute on requests whose callers have already moved on. CoDel (Controlled Delay), developed at Google and inspired by networking research, takes a queue-aware approach: it monitors how long requests spend waiting in the queue, and when the minimum queue wait time exceeds a target (e.g., 5ms) for a sustained period, it begins dropping requests that have waited too long. CoDel is self-tuning -- it adapts to the system's actual capacity rather than requiring manually configured thresholds.

Effective load shedding requires client cooperation. The server should return machine-readable rejection responses: HTTP 429 (Too Many Requests) with a Retry-After header telling the client when to try again, or HTTP 503 (Service Unavailable) for transient overload. Well-behaved clients respect the Retry-After header and implement exponential backoff with jitter. Without client cooperation, rejected requests are immediately retried, adding to the load and defeating the purpose of shedding. The client-server contract around load shedding -- 429/503 responses, Retry-After headers, and backoff strategies -- is as important as the shedding algorithm itself.

A nightclub has a fire-code capacity of 500 people. When 500 people are inside, the bouncer stops letting people in -- even if there is a long line. This is load shedding. The bouncer is not being mean; they are preventing overcrowding that would make the experience terrible for everyone inside and potentially dangerous. If the bouncer let everyone in, the club would be uncomfortably packed, the bar would be overwhelmed, and nobody would have a good time. By maintaining a capacity boundary, the 500 people inside have a great experience, and people in line know they need to wait rather than being packed into an unpleasant situation.