Logging Pipeline — Multi-Tier with Alerting & Distributed Query

The multi-tier logging platform with streaming alerting represents the state of the art in production observability — the architecture behind Datadog, Splunk Cloud, Elastic Observability, and AWS CloudWatch at their largest scale. It addresses three limitations of the standard Kafka-plus-Elasticsearch pipeline that become critical at FAANG scale: the lack of streaming alerting, the two-tier storage gap between hot and cold, and the absence of a unified query interface across storage tiers.

The first limitation is alerting latency. The standard pipeline (v1) indexes logs and pre-computes dashboard metrics but has no mechanism for streaming alerting. Detecting that the 5xx error rate for the checkout service exceeds 1 percent in a 5-minute window requires either polling Elasticsearch periodically (adding minutes of delay) or building custom aggregations from the Redis metrics cache (limited to pre-defined aggregations). In a production incident, minutes of alert delay translate directly to minutes of customer impact. The multi-tier variant adds a dedicated AlertEngine that evaluates rules on every parsed log record as it flows through the pipeline, providing sub-5-second alert latency.

The second limitation is the storage gap between hot and cold tiers. The standard pipeline has two tiers: hot (Elasticsearch, 7 days, ~$2,300/TB/month) and cold (S3, 7 years, ~$0.01/TB/month). Data transitions directly from hot to cold with no intermediate tier. This means that querying logs from 8 days ago requires a cold-tier S3 scan at 30-60 seconds latency, even though 8-day-old data is frequently accessed during incident investigations. The multi-tier variant introduces a warm tier: compressed Parquet files in S3 Standard ($23/TB/month) retaining 90 days of data with ~10-second query latency via S3 Select. This fills the gap between expensive-fast and cheap-slow.

The third limitation is the lack of a unified query interface. In the standard pipeline, engineers must know which tier contains the data they need and use different tools for each tier. Recent data requires an Elasticsearch query; historical data requires S3 Select or Athena. The multi-tier variant adds a distributed QueryEngine that accepts a single query with a time range and automatically determines which tiers to query, executes sub-queries in parallel, and merges results transparently. This is how production platforms like Datadog present a seamless search experience across petabytes of data spanning years.

Additional capabilities in the multi-tier variant include schema-on-read parsing (auto-detecting JSON, logfmt, and grok patterns without requiring upfront schema definition), sampling for high-cardinality metrics (keeping 100 percent of ERROR, 10 percent of WARN, 1 percent of DEBUG to reduce indexing cost 100x while preserving statistical accuracy), and a columnar hot index (ClickHouse or Apache Druid) that provides 10x faster aggregation queries than Elasticsearch for dashboard workloads.

The operational complexity is significantly higher than the standard pipeline: 11 components across 3 storage tiers, a streaming alert engine with sliding-window state management, a distributed query engine with cross-tier query planning, and a warm compactor managing continuous hot-to-warm and warm-to-cold data lifecycle. This architecture requires a dedicated SRE team to operate reliably. The trade-off is justified at FAANG scale where the combination of sub-second alerting, transparent cross-tier queries, and optimized storage economics directly reduces mean time to detection and mean time to resolution for production incidents.

The multi-tier logging platform uses 11 components organized into ingest, processing, alerting, query, and storage layers. The design prioritizes three properties that the standard pipeline lacks: streaming alerting (sub-5-second anomaly detection), transparent cross-tier queries (one query spans hot, warm, and cold), and 3-tier storage economics (columnar hot, Parquet warm, Glacier cold).

The ingest path begins with LogAgent, representing millions of hosts streaming batched log lines via gRPC to IngestGateway (AWS NLB). IngestGateway validates tenant headers, applies per-tenant rate limiting, and publishes to IngestStream (Kafka with 512 partitions). The API returns immediately — all downstream processing is async. At 100M lines per second, Kafka's deep buffer (100M messages) absorbs burst traffic during incidents and deployments.

ParserWorker (200 instances) consumes from IngestStream and is the central routing hub. For each log batch, it applies schema-on-read parsing: auto-detecting JSON, logfmt, and unstructured text patterns, extracting structured fields (service, level, trace_id, user_id), and enriching with metadata. It then applies sampling rules (100 percent ERROR, 10 percent WARN, 1 percent DEBUG for indexing; all data preserved in warm/cold tiers). Parsed records are routed to three destinations: (1) bulk index into HotIndexDB, (2) forward to AlertEngine for real-time rule evaluation, and (3) buffer for WarmCompactor.

HotIndexDB is a columnar store (ClickHouse or Apache Druid on OpenSearch) retaining the last 24 hours of parsed data. Columnar storage provides 10x faster aggregation queries than Elasticsearch for time-series workloads: GROUP BY service, level, minute over 24 hours completes in approximately 100ms. The trade-off is weaker full-text search (bloom filters instead of inverted indexes), which is acceptable for the 24-hour hot tier where most queries are aggregation-heavy (dashboards, alerts, SLO monitors). 6 data nodes with 128 partitions and 2 replicas.

AlertEngine (50 pods) receives parsed log records from ParserWorker and evaluates them against active rules stored in AlertDB (PostgreSQL). Rules define conditions on sliding windows: for example, '5xx rate exceeds 1 percent in a 5-minute window for service:checkout.' Each worker maintains in-memory sliding-window counters per rule per service. When a threshold is breached, AlertEngine writes the fired alert to AlertDB and triggers notifications (webhook, PagerDuty, Slack). Sub-5-second latency from log emission to alert firing.

WarmCompactor (30 workers) manages two lifecycle transitions: hot-to-warm and warm-to-cold. For hot-to-warm, it reads data older than 24 hours from HotIndexDB, compresses into columnar Parquet files organized by tenant/date/hour, and writes to WarmStore (S3 Standard, ~$23/TB/month). For warm-to-cold, it transitions data older than 90 days from WarmStore to ColdArchive (S3 Glacier Deep Archive, ~$0.00099/GB/month) via S3 lifecycle policies.

QueryEngine (20 pods, 100 threads each) is the unified search interface. It receives queries from QueryClient, determines which tiers contain relevant data based on the time range, and executes sub-queries in parallel: (1) HotIndexDB for the last 24 hours (~500ms), (2) WarmStore via S3 Select on Parquet for 1-90 days (~10s), and (3) ColdArchive via Glacier restore for 90 days to 7 years (minutes to hours). Results are merged, sorted by timestamp, and returned. Dashboard metric queries are served directly from HotIndexDB aggregations at sub-500ms latency.