Rate Limiter

A Rate Limiter is a component that controls the rate of incoming requests to protect backend services from overload. It enforces maximum request rates per client, IP, or API key using algorithms like token bucket, sliding window, or fixed window. When a client exceeds the limit, requests are rejected with HTTP 429 status codes. Rate limiters are essential in system design for DDoS protection, fair multi-tenant resource sharing, and preventing cascade failures during traffic spikes.

Token bucket allows controlled bursts — tokens accumulate up to a maximum (burst capacity) and are consumed per request. A client can send a burst of requests equal to the bucket size, then must slow to the refill rate. Sliding window counts requests in a continuously rolling time window with no burst allowance, providing smoother rate enforcement. Token bucket is better for APIs where short bursts are acceptable; sliding window is better when strict, uniform rate enforcement is required.

Distributed rate limiting requires shared state across all rate limiter instances. The standard approach uses Redis with atomic INCR and EXPIRE commands — each request increments a counter keyed by client ID and time window, and EXPIRE ensures automatic cleanup. For token bucket, Redis stores the token count and last refill timestamp. Lua scripting in Redis ensures atomicity of multi-step operations. Alternative approaches include local rate limiting with gossip protocol synchronization, though these sacrifice accuracy for lower latency.

The fixed window boundary problem occurs because counters reset at interval boundaries. If the limit is 100 requests per minute, a client could send 100 requests at 0:59 and 100 more at 1:01 — effectively 200 requests in 2 seconds while technically respecting the per-minute limit. Sliding window algorithms solve this by tracking request counts across a continuously moving time window. A common compromise is the sliding window counter, which interpolates between the current and previous window counts.

Rate limiters and client retries can create dangerous feedback loops. When a rate limiter rejects requests with 429 responses, clients that retry immediately amplify the overload. Best practice is for the rate limiter to include a Retry-After header indicating when the client should retry. Well-behaved clients implement exponential backoff with jitter based on this header. In Vetora's simulator, you can model this interaction to observe how aggressive retry policies cause sustained overload while conservative backoff allows graceful recovery.