Distributed Filesystem — Metadata + Block Storage

Distributed file storage is a cornerstone of modern cloud infrastructure, underpinning services from Amazon S3 to Google Cloud Storage to HDFS-based data lakes. It is a frequent system design interview topic at companies that operate large-scale storage platforms, because it tests a candidate's understanding of data durability, replication strategies, metadata management, and the trade-offs between consistency and availability at exabyte scale.

The central challenge is achieving extreme durability (11 nines, or 99.999999999%) while maintaining high throughput and low latency for diverse object sizes ranging from tiny configuration files to multi-terabyte data lake partitions. A single bit flip or disk failure must never result in data loss, which requires replicating every block of data across multiple independent fault domains and continuously monitoring for and repairing any degradation.

The architecture must separate the metadata plane from the data plane because they have fundamentally different characteristics. Metadata is small (roughly 100 bytes per object: key, block list, checksums) and fits in memory even for trillions of objects. Data is massive (exabytes) and must be distributed across hundreds or thousands of storage nodes. Coupling these two planes would mean that metadata operations (list, head, permission checks) compete with bulk data I/O, degrading performance for both. This separation is the same pattern used by HDFS (NameNode vs DataNode) and Amazon S3 internally.

At production scale, the system must handle 500K metadata reads per second, 100K block reads per second, and 50K writes per second at peak. Objects are split into 64MB blocks, each replicated three times across different failure domains. A background repair worker continuously scans for under-replicated blocks and re-replicates them to maintain durability guarantees. The interview version of this problem expects candidates to address block placement strategy, consistency guarantees (read-after-write), garbage collection of deleted objects, and the operational complexity of running a storage system at this scale.

The architecture follows the HDFS/S3 pattern of separating the lightweight metadata plane from the heavy data plane. All client requests enter through the API Gateway (IAM-style authentication, per-bucket rate limiting) and load balancer, then route to the MetadataService, which serves as the coordination layer for all storage operations.

For object reads, MetadataService first checks MetadataCache (a 12-node Redis cluster) for the object-to-block mapping. With a 95% cache hit rate, most metadata lookups complete in approximately 2ms. On cache miss, MetadataService falls back to MetadataDB (DynamoDB with strong consistency, 128 partitions, 3 replicas), which responds in approximately 15ms. The response includes block IDs and replica locations, and the client then reads block data directly from the nearest BlockStore replica, bypassing MetadataService for the data plane transfer.

For object writes, MetadataService allocates block IDs, selects three storage nodes across different fault domains for replicas, writes the object-to-block mapping to both MetadataDB (durable) and MetadataCache (for immediate read-after-write consistency), and returns the block locations to the client. The client then writes 64MB block data directly to the three BlockStore replicas in parallel. The API returns success only after all three replicas acknowledge, ensuring durability before the caller proceeds. Every write operation also generates an audit event in the Kafka-based AuditStream for compliance and repair tracking.

The RepairWorker is the durability guarantee engine. Running continuously with 30 worker instances, it consumes audit events from AuditStream, periodically checks block health via checksum verification (SHA-256), and re-replicates any under-replicated blocks. When a storage node fails, the worker reads a healthy replica from BlockStore and writes a new replica to a different node, then updates MetadataDB with the new location. Repair is prioritized by urgency: blocks with only one remaining replica are repaired before blocks with two replicas. This asynchronous approach spreads repair load over hours rather than creating an I/O burst that would degrade client-facing performance.