Three Pillars of Observability

• 1Logs are high-cardinality, high-detail event records. Use structured logging (JSON) with consistent field names (trace_id, service, operation, duration_ms) to make them queryable. Unstructured printf-style logs become unusable past a few hundred RPS.
• 2Metrics are pre-aggregated time series. They excel at dashboards, alerting, and capacity planning. The four golden signals (latency, traffic, errors, saturation) form the minimum viable metrics set for any service.
• 3Traces follow a single request across service boundaries using context propagation (W3C Trace Context or B3 headers). Each span records service, operation, duration, status, and arbitrary tags. Head-based or tail-based sampling controls cost.
• 4No single pillar is sufficient. Metrics tell you the 99th percentile is high; logs tell you which error is occurring; traces tell you which downstream service is the bottleneck for a specific slow request.
• 5Correlation is the key to modern observability. Embedding trace_id in log lines, exemplar links in metrics, and service.name across all three lets operators pivot seamlessly between pillars.
• 6Cost grows differently per pillar. Metrics: O(cardinality × retention). Logs: O(volume × retention). Traces: O(sample_rate × span_count × retention). Most organizations spend 60-80% of their observability budget on log storage.

Debugging a Slow Checkout

Spotify's Migration to OpenTelemetry

• ⚠Logging without structure: Printf-style logs like 'Error processing order 12345' are impossible to query at scale. Without structured fields (order_id, error_type, service), log search degenerates into regex over terabytes of text. Use structured JSON logging with a consistent schema, including trace_id, service, operation, and error_type in every log line, enforced via a shared logging library.
• ⚠Alerting on logs instead of metrics: Log-based alerts are expensive (scan terabytes per query), brittle (break when log format changes), and slow (minutes of lag). Metrics-based alerts fire in seconds. Extract counters from logs at write time (e.g., log-to-metric rules in Loki or Datadog), alert on metric thresholds, and use logs only for root-cause investigation after an alert fires.
• ⚠No sampling strategy for traces: Storing 100% of traces at 50K RPS produces ~4 TB/day of span data. Cost explodes and query performance degrades, and engineers disable tracing entirely, losing the debugging benefit. Use head-based sampling at 1-5% for normal traffic and add tail-based sampling to always capture error and high-latency traces -- this captures 99% of debugging value at 5% of cost.
• ⚠Treating observability as an afterthought: Adding instrumentation to a 50-service system post-launch means months of PRs, inconsistent schemas, and missing coverage during the riskiest period (first launch). Bake observability into the service template, auto-instrument via OpenTelemetry agents, and make trace_id propagation a required middleware, not an opt-in library.