Timeouts and Deadline Propagation

Timeouts are the single most important reliability mechanism in distributed systems. A network call without a timeout is a resource leak waiting to happen. When a downstream service becomes unresponsive -- due to a network partition, a deadlocked process, or an overwhelmed server -- a call without a timeout will hold its thread, connection, and memory indefinitely. If this happens to enough concurrent requests, the calling service runs out of threads and becomes unresponsive itself, cascading the failure upstream. Setting appropriate timeouts on every network call is non-negotiable: it is the first line of defense against cascading failure.

Timeouts come in several layers, each protecting against a different failure mode. Connection timeout governs the TCP handshake phase -- how long to wait for the remote server to accept the connection. This is typically 1-5 seconds; a server that cannot accept a connection within 5 seconds is likely unreachable. Read timeout (or socket timeout) governs how long to wait for the response body after the connection is established. This depends on the expected operation time: a simple key-value lookup might have a 500ms read timeout, while a complex aggregation query might need 10 seconds. Total timeout (or request timeout) caps the entire operation end-to-end, including connection establishment, request sending, server processing, and response reading. This is the ultimate safety net and should include time for any retries.

Deadline propagation takes timeouts from a per-call mechanism to a system-wide coordination mechanism. In a call chain where service A calls B, and B calls C, without deadline propagation each service sets its own independent timeout. If A has a 500ms timeout, B has a 1-second timeout, and C has a 2-second timeout, then B might initiate a call to C that takes 1.5 seconds -- well within B's and C's timeouts but long past A's deadline. A has already returned an error to the user, and B and C are doing wasted work. Deadline propagation solves this by passing the remaining time budget through the call chain. When A calls B with 500ms remaining, B knows it has at most 500ms for everything, including its own processing and the call to C. B might allocate 50ms for itself and pass 450ms to C as the deadline.

Context cancellation is the complementary mechanism that cleans up in-flight work when a deadline expires. When A's timeout fires, it should cancel the request to B, which should cancel its request to C. Go's context.Context propagates both deadlines and cancellation signals through the call chain. gRPC propagates deadlines natively via metadata headers, automatically canceling server-side processing when the client deadline expires. In HTTP-based systems, teams typically implement deadline propagation manually using a custom header (e.g., X-Deadline or X-Request-Timeout) that each service reads, subtracts its own processing time, and passes downstream. Without context cancellation, timed-out requests continue consuming resources on downstream services even after the upstream caller has moved on.

Imagine ordering food at a restaurant. You tell the waiter you need to leave in 30 minutes (your deadline). The waiter writes down 25 minutes for the kitchen (reserving 5 minutes for serving). The kitchen checks: can we prepare this in 25 minutes? If yes, they start cooking. If the dish requires 45 minutes of prep, they immediately tell the waiter instead of starting a dish that cannot be finished in time. If you leave after 30 minutes, the waiter cancels the order (context cancellation) so the kitchen stops cooking food nobody will eat. Without this coordination, the kitchen would keep cooking, wasting ingredients and burner time, for a customer who has already left.