Distributed File System — Metadata + Distributed Block Storage

The metadata-plus-block-storage architecture is the industry-standard approach to distributed file systems, pioneered by Google File System (GFS) and adopted by HDFS, Amazon S3, and virtually every production object storage system. It solves the fundamental limitations of the single-server naive approach by separating the metadata plane from the data plane and distributing data across many storage nodes.

The key architectural insight is that metadata and data have fundamentally different characteristics. Metadata is small (approximately 100 bytes per object: key, block list, checksums, permissions) and accessed frequently — every read, write, list, and delete operation starts with a metadata lookup. Data is massive (objects range from bytes to terabytes) and accessed less frequently. By separating them, each plane can be optimized independently: metadata fits in memory and can be cached aggressively (Redis at 95% hit rate), while data is distributed across hundreds of storage nodes for parallel I/O.

The data plane uses a block-based storage model. Objects are split into fixed-size blocks (64 MB), each stored with 3x replication across different fault domains (availability zones). When a client uploads a 640 MB file, it is split into 10 blocks. Each block is written to 3 different BlockStore nodes in different AZs — 30 block writes total, many of which happen in parallel. This provides both durability (any 2 replicas can fail without data loss) and read performance (the client reads from the nearest replica). The 64 MB block size is a carefully chosen trade-off: smaller blocks would mean more metadata rows per object (increasing metadata DB load), while larger blocks would reduce parallelism opportunities and waste space for small objects.

The metadata plane consists of a MetadataService backed by MetadataDB (DynamoDB) and MetadataCache (Redis). On write, MetadataService allocates block IDs, selects storage nodes across fault domains, writes the object-to-block mapping to both DB and cache, and returns block locations to the client for direct data upload. On read, MetadataService looks up the block list in cache (95% hit rate, 2ms) or DB (15ms on miss), and returns block locations for direct data download. This two-phase approach — metadata lookup followed by direct data transfer — means the MetadataService handles only lightweight metadata operations, not the heavy data I/O.

A RepairWorker runs continuously in the background, monitoring block health via heartbeats from storage nodes. When a node fails or a block is corrupted (checksum mismatch), the worker reads a healthy replica, writes a new replica to a different node, and updates the metadata. This maintains 3x replication for 11-nines durability (99.999999999%). The repair process is asynchronous and prioritized — blocks with only 1 remaining replica are repaired before those with 2 replicas, preventing data loss during cascading failures. It does not impact client-facing performance because repair I/O is rate-limited to avoid saturating storage nodes.

The storage overhead of 3x replication is the primary cost concern at scale. For 1 PB of user data, the system stores 3 PB of raw block data. This overhead motivates the erasure-coded v2 variant, which achieves the same durability at only 1.4x overhead — a 53% reduction in raw storage cost.

This architecture appears in every system design interview for storage roles. Interviewers expect candidates to explain the metadata/data separation, articulate the block replication strategy, reason about the RepairWorker's role in durability, and identify the MetadataService as a potential bottleneck at extreme scale — which motivates the erasure-coded variant's use of Raft consensus for distributed metadata.

The distributed file system uses 8 components organized into separate metadata and data planes with an audit and repair pipeline. The components are: StorageClient, ApiGateway, MainLB, MetadataService, MetadataCache (Redis), MetadataDB (DynamoDB), BlockStore (S3-modeled distributed storage), AuditStream (Kafka), and RepairWorker.

All traffic enters through the ApiGateway, which performs IAM-style signature authentication (approximately 3ms), enforces per-bucket rate limits (800K RPS capacity), and routes to the MainLB. The Load Balancer distributes requests across 25 MetadataService pods using round-robin — metadata operations are stateless. The ApiGateway adds approximately 1.5ms of transformation latency for request normalization and provides a centralized point for rate limiting, authentication, and routing.

The MetadataService is the brain of the system. It handles four operations: GET (resolve object to block mapping), PUT (allocate blocks, select replica nodes, write mapping), DELETE (mark tombstone, schedule garbage collection), and LIST (prefix scan). For GET, it checks MetadataCache first — with a 95% hit rate, most lookups return in 2ms. On cache miss, it falls back to MetadataDB at approximately 15ms. For PUT, it writes to both MetadataDB (durable) and MetadataCache (fast subsequent reads), ensuring read-after-write consistency. The service runs 25 pods with 100 threads each for 312K sustained RPS.

MetadataCache is a 12-node Redis cluster caching object-to-block mappings. Each entry is approximately 100 bytes (key, block IDs, replica locations, size, checksum). With billions of cached entries and a 95% hit rate, the cache reduces MetadataDB load by 20x. Write-through on PUT ensures newly stored objects are immediately readable from cache. LRU eviction with 1-hour TTL balances memory usage with freshness. The cache uses a single-flight pattern to prevent thundering herd on cache misses — if multiple concurrent requests miss the same key, only one query hits the database while others wait for the result.

MetadataDB is DynamoDB on-demand with 128 partitions and 3 replicas. It stores the authoritative object-to-block mappings with strong consistency mode for read-after-write guarantees. The partition key is the bucket/key hash, distributed across partitions for uniform load. It handles approximately 25K reads/sec (cache misses) and 50K writes/sec. Latency follows a log-normal distribution with a tail at approximately 30ms for reads and 60ms for writes, reflecting the realistic behavior of distributed databases under production load.

BlockStore represents the distributed data storage fleet — modeled as S3 Standard with 128 partitions and 3 replicas. Objects are split into 64 MB blocks, each stored with 3x replication across different fault domains. Clients read and write blocks directly (bypassing MetadataService for data plane operations), providing horizontal scalability: adding more storage nodes increases both capacity and throughput linearly. Each block has a SHA-256 checksum stored in MetadataDB, verified on every read to detect silent data corruption (bit rot).

AuditStream is a 64-partition Kafka cluster logging every write operation (PUT and DELETE). RepairWorker consumes these events to track newly written blocks and monitor their health. It continuously scans for under-replicated blocks (fewer than 3 healthy replicas due to node failures or corruption) and creates new replicas from healthy copies. The worker runs 30 instances processing approximately 1K repair operations per second, prioritizing blocks by remaining replica count. This background repair process is the durability guarantee engine — it maintains 3x replication despite hardware failures, achieving 11-nines durability.