Distributed File System — Erasure-Coded Multi-Region Storage

The erasure-coded multi-region architecture represents the production-grade solution to distributed file storage — the architecture behind Amazon S3, Google Cloud Storage, and Azure Blob Storage. It solves two critical limitations of the metadata-plus-replication approach (v1): the 3x storage overhead of replication and the single-region failure domain.

Reed-Solomon erasure coding is the key innovation. Instead of storing 3 complete copies of every block (3x overhead), the system splits each block into 10 data shards and computes 4 parity shards using Reed-Solomon coding — 14 total shards at 1.4x storage overhead. Any 10 of the 14 shards can reconstruct the original data. This means the system tolerates up to 4 simultaneous shard losses (node failures, disk corruption) while maintaining full data availability — superior fault tolerance to 3x replication (which tolerates only 2 failures) at less than half the storage cost.

The numbers make the case compelling. For 1 PB of data: 3x replication requires 3 PB of raw storage at approximately $69,000/month (at $0.023/GB). Erasure coding (10+4) requires only 1.4 PB at approximately $32,200/month — a 53% cost reduction with better fault tolerance. At exabyte scale, this difference is hundreds of millions of dollars annually. Every major cloud provider uses erasure coding for their object storage tier.

The metadata layer uses a 5-node Raft consensus cluster instead of a single database. Raft provides strongly consistent metadata operations even during node failures: the leader processes all writes, replicating to at least 2 followers before acknowledging. This survives up to 2 metadata node failures while maintaining strong consistency. The previous variant's single MetadataDB was a potential single point of failure for metadata; Raft eliminates this vulnerability.

Multipart upload solves the large-object problem. Instead of uploading a 10 GB file as a single monolithic request (fragile, not resumable), the client splits it into parts (64 MB each) that are uploaded independently. Each part is erasure-coded and stored on ChunkServers. Parts can be uploaded in parallel and resumed on failure — only the failed part needs retransmission. The server stitches parts into a complete object only after all parts are verified.

Copy-on-write versioning provides data protection and audit compliance. When an object is overwritten, the previous version's shards are not deleted — a new version with new shards is created, and metadata tracks all versions. This enables instant rollback, audit trails, and protection against accidental deletion. Lifecycle policies automatically delete old versions after a configurable retention period to control storage growth.

Lifecycle tiering automatically moves cold data to cheaper storage. Objects not accessed for 30 days transition to infrequent-access (IA) tier at 40% lower cost. After 90 days without access, they move to cold storage (S3 Glacier equivalent) at 90% lower cost. A 1 PB dataset with typical access patterns costs approximately $6,000/month with lifecycle tiering vs $23,000/month on standard tier — a 74% reduction. The LifecycleWorker continuously evaluates policies and executes transitions.

Multi-region replication provides disaster recovery with RPO under 1 minute. Every write publishes a replication event to Kafka, consumed by a cross-region replicator that copies shards and metadata to the DR region. A full region outage (rare but possible) is survivable with automatic DNS failover to the DR region in approximately 5 minutes.

This is the architecture interviewers expect senior candidates to arrive at after discussing the naive and replication approaches. The progression from single-server to distributed replication to erasure coding demonstrates deepening understanding of storage trade-offs.

The erasure-coded system uses 11 components organized into metadata, encoding, storage, replication, and lifecycle management tiers. The key additions over the replication-based v1 variant are the ErasureEncoder (Reed-Solomon codec), Raft consensus MetadataDB, IndexCache (replacing MetadataCache), ReplicationStream (cross-region Kafka), LifecycleWorker, and ColdStore (glacier-equivalent archive).

The request path starts at the ApiGateway, which authenticates IAM signatures (approximately 3ms) and routes through MainLB to ObjectService. The ObjectService is the orchestration hub: it coordinates metadata operations (IndexCache and MetadataDB), encoding operations (ErasureEncoder), storage operations (ChunkServers), and replication (ReplicationStream). It runs on 50 pods with 200 threads each for 625K sustained RPS. The ApiGateway supports 3M RPS capacity with rate limiting, providing substantial headroom above the 2M peak.

The write path is a multi-step pipeline. ObjectService receives the PUT request, sends data to ErasureEncoder which produces 14 shards (10 data + 4 parity), writes all 14 shards to 14 different ChunkServers in parallel, commits metadata to MetadataDB via Raft leader (replicated to 2+ followers before ack), updates IndexCache for fast subsequent lookups, and publishes a replication event to ReplicationStream. For objects larger than 64 MB, multipart upload is triggered automatically — each part goes through this pipeline independently. The write completes only after the Raft leader confirms the metadata commit, ensuring durability before the client receives a success response.

The read path is optimized for speed. ObjectService checks IndexCache for the shard map (94% hit rate, 2ms). On cache miss, it queries MetadataDB Raft leader for a strongly consistent read (approximately 15ms). With the shard map, it fetches any 10 of the 14 shards from ChunkServers in parallel (approximately 15ms). If all 10 data shards are available, ErasureEncoder concatenates them (approximately 1ms — no decode needed). If some data shards are missing or corrupted, parity shards enable reconstruction via Reed-Solomon decode (5-15ms depending on missing count). The system is resilient to up to 4 simultaneous shard failures per object without any data loss or read degradation.

The MetadataDB is a 5-node PostgreSQL Raft cluster with 256 partitions. The leader handles all writes, replicating to at least 2 followers before acknowledging (majority quorum). This provides strong consistency while tolerating up to 2 node failures. Leader election takes 1-5 seconds on leader failure, during which writes are unavailable but reads can be served from followers with slightly stale data.

IndexCache is a 16-node Redis cluster caching object-to-shard mappings at approximately 150 bytes per entry. With 94% hit rate, it reduces MetadataDB load by 16x. Write-through on PUT ensures read-after-write consistency. The cache uses a 3600-second TTL with LRU eviction, and the single-flight pattern prevents thundering herd on concurrent misses for the same key.

ChunkServers store erasure-coded shards across 256 partitions distributed via consistent hashing. Shards are immutable — overwrites create new shard sets (copy-on-write). Two storage tiers are supported: standard (io2 EBS, high IOPS for frequently accessed data) and infrequent-access (gp3, lower IOPS at reduced cost for data not accessed in 30+ days). Per-shard SHA-256 checksums are verified on every read to detect silent data corruption (bit rot).

ReplicationStream is a 128-partition Kafka cluster handling cross-region replication events. Every write publishes object metadata and shard references. A cross-region consumer replicates to the DR region for disaster recovery (RPO < 1 minute). The Kafka topic is partitioned by object_key to maintain per-object ordering of operations.

LifecycleWorker continuously evaluates lifecycle policies with 20 worker instances processing approximately 5K transitions per second. It transitions cold objects from standard to IA (30 days without access) and from IA to ColdStore (90 days). ColdStore is an S3 Glacier-equivalent archive with retrieval times ranging from minutes (expedited) to hours (bulk). The worker also handles garbage collection of orphaned shards from deleted objects and abandoned multipart uploads, reclaiming storage that would otherwise leak indefinitely.