Logging Pipeline — Kafka + Indexing Pipeline (Hot/Warm/Cold)

The Kafka-based logging pipeline is the industry-standard architecture used by every major observability platform — Datadog, Splunk, Elastic, and Grafana all use variations of this pattern. It solves the two fundamental problems that make the naive direct-to-database approach unworkable at scale: the write throughput ceiling and the search performance degradation.

The first problem is write throughput. Production systems generate log data at rates that no single database can absorb synchronously. A fleet of 10,000 hosts generating 10,000 log lines per second each produces 100 million lines per second at peak. The naive approach's 500 lines per second ceiling is seven orders of magnitude too low. The solution is asynchronous ingest via Kafka: agents publish log batches to Kafka topics, which provides sub-5ms acknowledgment regardless of downstream processing capacity. Kafka acts as a deep buffer (50 million messages) that absorbs traffic bursts during incidents and deployments while parser workers process at a sustainable rate.

The second problem is search performance. SQL LIKE on a PostgreSQL text column is an O(N) sequential scan that becomes unusable beyond a few million rows. The solution is Elasticsearch (or OpenSearch), which maintains an inverted index mapping every term to the documents containing it. This enables sub-second full-text search across billions of documents. The trade-off is that indexing is not instant — there is a delay between ingest and searchability while parser workers consume from Kafka, parse and enrich the logs, and bulk-index into Elasticsearch.

The third problem is storage economics. At 100 million lines per second with an average 4KB per batch, the system generates approximately 10 petabytes per day. Storing all of this in Elasticsearch would cost approximately $500,000 per day in compute and storage. The solution is tiered storage: a hot tier (Elasticsearch, 7 days, ~$2,300/TB/month) for recent searchable data, and a cold tier (S3, 7 years, ~$0.01/TB/month) for long-term archival. Roll-off workers continuously move expired data from hot to cold, achieving a 100x cost reduction for 95 percent of the data.

The architecture splits into two paths: the ingest path (agents to Kafka to parser workers to Elasticsearch and S3) and the query path (engineers through an API gateway to a search service that queries Elasticsearch for recent data and S3 for historical data). Redis caches pre-computed dashboard metrics to avoid expensive Elasticsearch aggregation queries on every dashboard refresh.

This architecture handles the scale requirements that the naive approach cannot: 100M lines per second ingest, sub-5-second search on recent data, 60-second search on historical data, and 7-year retention at manageable cost. The operational complexity is significantly higher (10 components versus 3), requiring expertise in Kafka partition management, Elasticsearch cluster sizing, S3 lifecycle policies, and worker auto-scaling.

Logging pipeline design at this level appears in interviews at Datadog, Splunk, Elastic, AWS CloudWatch, Google Cloud Logging, and Grafana Labs. Interviewers expect candidates to explain the Kafka buffering pattern, the Elasticsearch inverted index advantage, and the tiered storage cost optimization.

The Kafka-based logging pipeline uses 10 components organized into separate ingest and query paths with shared data layers. The ingest path handles the write-heavy workload (100M lines per second); the query path handles search and dashboard reads (60K queries per second).

The ingest path starts with AgentClient, representing millions of log agents streaming batched log lines via gRPC. Batches of approximately 1,000 lines in protobuf format (~4KB per batch) are sent to IngestLB (AWS NLB), which distributes across IngestService pods using round-robin. IngestService validates the batch, adds tenant_id and ingest_timestamp, and publishes to IngestStream (Kafka). The HTTP response returns immediately — indexing is fully async. At 100M lines per second, Kafka's 256 partitions and 10M msg/sec capacity provide massive headroom.

ParserWorker consumes from IngestStream. For each batch, it parses structured (JSON) and unstructured (grok pattern) logs, enriches with metadata (service name, host, trace_id), computes sliding-window metrics for MetricsCache, and bulk-indexes into IndexDB (Elasticsearch/OpenSearch). 100 workers process in parallel with bulk batches of 1,000 documents per index call. The hot tier retains 7 days of indexed data.

RolloffWorker runs continuously, reading expired data (older than 7 days) from IndexDB and archiving it to ArchiveStore (S3) in compressed Parquet format. After successful archive, the hot tier data is deleted. Cold tier retains data for 7 years. 20 workers handle the continuous roll-off at terabytes per hour.

The query path starts with QueryClient (engineers and dashboards). Requests flow through ApiGateway (AWS API Gateway — auth, rate limiting) to SearchService. For recent queries (last 7 days), SearchService queries IndexDB (Elasticsearch, ~500ms p99). For historical queries, it reads from ArchiveStore (S3 Select on Parquet, ~30s p99). MetricsCache (Redis) stores pre-computed aggregations (error rates, p99 latencies) for dashboard reads (~2ms).

The architecture provides clear separation of concerns: Kafka handles burst absorption, ParserWorker handles parsing and indexing, Elasticsearch handles search, S3 handles archival, and Redis handles dashboard metrics. Each component scales independently. The primary limitation is the lack of streaming alerting — anomaly detection requires polling Elasticsearch or building metrics from the Redis cache, adding minutes of delay compared to the streaming approach in the Multi-Tier variant.