Feature Stores

The feature store emerged as an ML infrastructure primitive around 2017-2018, pioneered by Uber (Michelangelo), Airbnb (Zipline), and LinkedIn (Frame). The core insight was that feature engineering -- the process of transforming raw data into model inputs -- is the most time-consuming part of applied ML (often 60-80% of effort), and most of that work is duplicated across teams. Different teams re-implement the same features (e.g., 'user's average session duration over 30 days') with slight variations, leading to inconsistencies between training and production.

The feature store solves three fundamental problems. First, train-serve skew: when training features are computed in batch (e.g., Spark SQL over a data lake) but serving features are computed in real-time (e.g., Java code in the serving path), subtle differences in logic, precision, or data freshness cause the model to receive inputs in production that differ from what it learned during training. A feature store ensures both paths use the same transformation definition. Second, feature reuse: once a team defines 'user_30d_avg_session_duration', any other team can discover and use it without re-implementing the computation. At Uber, this increased feature reuse from ~0% to over 50%. Third, point-in-time correctness: when building a training dataset, you must join features as they existed at the time of each training example, not as they exist today. Otherwise, you introduce data leakage (the model trains on future information). Feature stores automate point-in-time joins across multiple feature tables.

Architecturally, a feature store has four components: (1) a feature registry that catalogs feature definitions, owners, and metadata; (2) a transformation engine that computes features from raw data (batch transforms via Spark/Flink, streaming transforms via Kafka/Flink, on-demand transforms at request time); (3) an offline store (data warehouse or data lake) for training data retrieval; and (4) an online store (Redis, DynamoDB, or Bigtable) for low-latency serving. Materialization pipelines sync computed features from the offline store to the online store on a schedule.

The distinction between batch, streaming, and on-demand features is critical. Batch features (recomputed hourly or daily) are the simplest and cheapest -- most features fall here. Streaming features (updated in near-real-time from event streams) are needed for fraud detection, real-time recommendations, and dynamic pricing. On-demand features (computed at request time from the request payload, like 'distance between user location and merchant') cannot be precomputed because they depend on request-time data. A production feature store must support all three modes.

A fraud detection model uses a feature 'user_7d_transaction_count'. During training, a data scientist computes this using a SQL query over the data warehouse: COUNT(*) with a 7-day window. In production, an engineer re-implements the same logic in Java, but uses a 7-day window based on UTC midnight boundaries instead of a rolling 7-day window. The model's accuracy drops 8% in production because the feature values differ. With a feature store, both training and serving use the same registered transformation definition, eliminating the discrepancy.