Log Aggregation

In a distributed system with hundreds of service instances, logs are generated on ephemeral containers that may be destroyed within minutes. Without centralized aggregation, debugging requires SSH-ing into individual hosts and grepping files -- impossible at scale and useless after containers are recycled. Log aggregation solves this by collecting all logs into a durable, searchable store.

The canonical pipeline has four stages. Generation: services emit structured JSON logs to stdout with consistent fields (timestamp, service, level, trace_id, message). Collection: agents (Fluentd, Vector, Filebeat, OTel Collector) running on each node tail container logs and forward them. Processing: a buffer (Kafka) absorbs burst traffic and a processor enriches logs (add Kubernetes metadata, parse stack traces, redact PII). Storage: an indexed backend (Elasticsearch, Grafana Loki, ClickHouse) stores logs and serves queries. A web UI (Kibana, Grafana) provides search, filtering, and visualization.

The choice of backend defines the architecture's trade-offs. Elasticsearch (ELK stack) creates a full-text inverted index on every field, enabling fast arbitrary queries ('show me all logs where user_id=12345 AND latency>500ms'). But this indexing is expensive: each GB of raw logs requires 1.5-3 GB of index storage, and ingestion throughput is limited by indexing speed. Loki (Grafana stack) takes the opposite approach: it indexes only a small set of labels (service, level, pod) and stores log lines as compressed chunks in object storage. Queries that match labels are fast; queries that require full-text search scan chunks and are slower. Loki is 10-20x cheaper to operate than Elasticsearch for the same log volume but less powerful for ad-hoc analysis.

Structured logging is the foundation of effective log aggregation. Unstructured logs ('Error: failed to process order') require regex extraction at query time, which is slow and fragile. Structured JSON logs ({"level": "error", "service": "orders", "order_id": 12345, "error_type": "payment_declined", "trace_id": "abc-123"}) are parsed once at ingestion and queryable by any field. The shift from printf-style to structured logging is the single highest-leverage observability improvement a team can make.

Cost management dominates log aggregation operations. A moderate-size microservice fleet (100 services, 10K RPS total) generates 50-100 GB of logs per day. At Elasticsearch's storage amplification (3x for index + replica), that is 150-300 GB/day of indexed storage. With a 30-day retention, that is 4.5-9 TB. Cloud-hosted Elasticsearch (AWS OpenSearch, Elastic Cloud) charges $0.10-0.20/GB/month, making log storage a $500-1800/month expense for a modest fleet. Strategies to reduce cost include: log level gating (INFO in production, DEBUG only when investigating), sampling verbose log sources, tiered storage (7 days hot SSD, 30 days warm HDD, archive to S3), and replacing high-volume log lines with metric counters.

Before: logger.info(f'Order {order_id} payment failed: {error}'). This produces: '2024-01-15 14:03:07 INFO Order 12345 payment failed: card_declined'. Searching for all payment failures requires regex: /payment failed/. After: logger.info('payment_failed', order_id=12345, error_type='card_declined', amount_cents=5999, trace_id='abc-123'). This produces: {"timestamp": "2024-01-15T14:03:07Z", "level": "info", "msg": "payment_failed", "order_id": 12345, "error_type": "card_declined", "amount_cents": 5999, "trace_id": "abc-123"}. Now you can query: error_type='card_declined' AND amount_cents > 5000 -- impossible with the unstructured version.