VetoraDropbox / File Sync — Naive (Whole-File Upload + Polling)
The simplest possible file sync architecture: a single SyncService backed by PostgreSQL metadata and S3 file storage. Entire files are re-uploaded on any change — a 1-byte edit to a 1GB file re-uploads the full gigabyte. No chunking, no resume, no dedup, no WebSocket. Demonstrates why block-level chunking and content-addressed dedup are essential for file sync at any meaningful scale.
Problem Statement
File sync is one of the most commonly asked system design interview questions because it combines large-file transfer optimization, content deduplication, conflict resolution, real-time notification, and storage durability into a single problem. Companies like Dropbox, Google Drive, OneDrive, Box, and iCloud all ask variants of this question because it directly maps to their core product engineering challenges.
The naive approach uses the simplest possible architecture: a single SyncService backed by PostgreSQL for metadata and S3 for file content. When a user modifies a file, the client uploads the entire file via PUT /api/v1/files/{path} — even if only a single byte changed. For a 1GB file with a 1-byte edit, this means re-uploading the full gigabyte over the network, which takes 80+ seconds on a 100 Mbps connection. There is no chunking, no SHA-256 hashing, no content-addressed blocks, and no block-level dedup.
The lack of resume capability means that a network failure at 99% of a 1GB upload wastes 990MB of bandwidth and requires starting from byte 0. This is the fundamental limitation of using a single HTTP PUT for the entire file body — the request is atomic, so it either completes fully or fails entirely. S3 multipart upload exists but is not used in the naive approach because it adds complexity (managing upload IDs, part ETags, and abort cleanup).
Without content-addressed dedup, the same file uploaded by N users is stored N times in S3. If 100 users upload a common 500MB installer, S3 stores 100 copies totaling 50GB — versus 500MB with content-addressed blocks. At enterprise scale with shared libraries, dependencies, and common assets, storage costs can be 2-5x higher than with dedup. This is the storage equivalent of the N+1 query problem in databases.
Sync notifications use HTTP polling: every connected client calls GET /api/v1/sync/changes?cursor=<timestamp> every 5 seconds. At 10K users, this generates 2K queries/sec to the sync_log table in PostgreSQL — 95% of which return empty results (no new changes since last poll). This poll-heavy pattern is identical to the naive chat approach: wasted bandwidth, wasted database connections, and delayed notification (0-5 second lag).
Interviewers expect candidates to identify the whole-file upload as the primary bandwidth bottleneck, propose block-level chunking as the solution, discuss content-addressed dedup and its storage savings, reason about polling versus push notification, and explain how resume capability requires breaking files into independently uploadable chunks.
Architecture Overview
The naive file sync system is a four-component architecture: Client, SyncLB (Load Balancer), SyncService, and MetadataDB (PostgreSQL) with FileStorage (S3). There is no chunking service, no dedup index, no event stream, no WebSocket service, and no CDN.
All traffic arrives at SyncLB (AWS ALB), which distributes requests across SyncService pods using round-robin. The Load Balancer adds approximately 1.5ms of routing latency and can handle up to 5K RPS — well above the system's actual limits, which are constrained by the database and S3 transfer times. The Load Balancer is never the bottleneck.
SyncService handles four types of requests. File uploads (20% of traffic): the client sends PUT /api/v1/files/{path} with the entire file body. SyncService streams the request body directly to S3 as a single object keyed by user_id/file_path, then writes metadata (version, size, last_modified) to the files table in MetadataDB and appends a row to the sync_log table. File downloads (30%): the client sends GET /api/v1/files/{path} and SyncService streams the entire file from S3. Metadata reads (20%): lightweight queries to MetadataDB for file version, size, and timestamp. Sync polling (30%): the client polls GET /api/v1/sync/changes every 5 seconds, triggering a SELECT on the sync_log table.
MetadataDB is a single PostgreSQL instance with two tables: files (file metadata keyed by user_id + path) and sync_log (append-only change log). The sync_log table grows unboundedly — every file change appends a row. At 200 uploads/sec, this table adds 17M rows per day. Without TTL or partitioning, the table eventually becomes too large for efficient scanning, degrading poll query performance.
FileStorage (S3) stores each file as a single object. No content addressing — the S3 key is user_id/file_path. Overwriting a file replaces the entire S3 object. S3 provides 99.999999999% durability but storage costs scale linearly with user count due to lack of dedup. At 1M users with an average of 10GB per user, total storage is 10 PB — versus 3-5 PB with content-addressed dedup.
The fundamental limitations are: (1) whole-file upload wastes bandwidth proportional to file size regardless of change size, (2) no resume means large uploads on unreliable connections may never complete, (3) no dedup means storage scales as O(users x files) instead of O(unique_content), and (4) polling wastes database resources with empty queries.
Architecture Preview
Key Design Decisions
Whole-File Upload (Single HTTP PUT)
Choice
PUT /api/v1/files/{path} with entire file body for every change
Rationale
The simplest upload model — no chunking logic, no hash computation, no block tracking. The client sends the file bytes in a single HTTP PUT, and SyncService streams them to S3. The cost is catastrophic bandwidth waste: a 1-byte edit to a 1GB file re-uploads the entire gigabyte. With 4MB block chunking (V1), the same edit uploads only 4MB — a 250x improvement. With delta sync (V2), only the changed bytes (~100 bytes) are uploaded — a 10,000,000x improvement.
No Resume Capability
Choice
Atomic HTTP PUT — upload completes or fails entirely
Rationale
Resume requires tracking upload progress at the server side — either via S3 multipart upload (managing upload IDs and part ETags) or chunked transfer encoding with server-side state. The naive approach skips this for simplicity. The consequence is devastating for large files on unreliable connections: a network failure at 99% of a 1GB upload wastes 990MB of bandwidth. Mobile users on cellular connections may find it impossible to upload large files.
No Content Dedup (Per-User S3 Keys)
Choice
S3 key = user_id/file_path — each user gets a separate copy
Rationale
Content-addressed dedup requires SHA-256 hashing on the client, a dedup lookup table on the server, and reference counting for garbage collection. The naive approach stores each file as a separate S3 object per user. If 100 users share a 500MB file, S3 stores 50GB instead of 500MB. The storage overhead is proportional to user count for shared content. Production systems like Dropbox report 30-50% storage savings from dedup.
HTTP Polling for Sync Notifications
Choice
GET /api/v1/sync/changes?cursor=<timestamp> every 5 seconds
Rationale
Polling is the simplest notification model — any HTTP client can poll. No WebSocket library, no persistent connection, no sticky sessions. The cost is 0-5 second notification delay and massive wasted bandwidth: 95% of polls return empty results. At 100K users, polling generates 20K QPS of mostly-empty responses, overwhelming the database connection pool. WebSocket push (V2) eliminates this entirely.
Scale & Performance
Target RPS
~1K sustained (ceiling at database + S3 transfer)
Latency (p99)
80s+ for 1GB upload, 12-60ms for metadata, 0-5s sync delay
Storage
10 PB at 1M users / 10GB each (no dedup)
Availability
~99% (single DB instance, no redundancy)
Database Schema (HLD)
File metadata table mapping user+path to current version, size, S3 key, and timestamps. Updated on every file upload. Single PostgreSQL instance, no sharding. Growth is proportional to total files across all users.
Indexes: PK on file_id, UNIQUE on (user_id, path), idx_files_user ON (user_id)
Simple metadata table — no block hashes, no dedup references. Each file maps to a single S3 object. Version is incremented on each upload but no version history is retained — only the current version exists.
Append-only change log recording every file create, update, and delete. Scanned by every poll query (WHERE created_at > cursor). Grows unboundedly at ~200 rows/sec. No TTL or partitioning — eventually becomes the dominant source of database bloat and query degradation.
Indexes: PK on log_id, idx_sync_log_user_time ON (user_id, created_at)
This table is the polling bottleneck. Every connected client scans it every 5 seconds. At 10K users: 2K queries/sec on this table. At 100K users: 20K queries/sec. Growth rate of 17M rows/day means the index grows proportionally, degrading query performance over weeks.
Solution Comparison
| Variant | Tier | Latency | Throughput | Cost | Complexity | Reliability |
|---|---|---|---|---|---|---|
| V0: Naive (Whole-File Upload + Polling) | T1 | 80s+ upload (1GB), 0-5s sync delay | ~1K RPS | $500/month | Low | 99% (single DB) |
| V1: Block-Level Chunked (SHA-256 + Kafka) | T2 | 0.3s upload (changed block), <5s sync | 25K RPS peak | $3,000/month | Medium | 99.9% (multi-AZ) |
| V2: Delta Sync + Dedup (WebSocket + CDN) | T3 | <0.1s delta upload, <2s sync | 25K RPS peak | $8,000/month | High | 99.95% (multi-AZ, CDN) |
This template is for educational and illustration purposes only. It may not represent the optimal production design for this problem. Real-world systems involve additional considerations (compliance, specific cloud provider constraints, organizational requirements) not captured here. Use this as a starting point for discussion, not as a production blueprint.
Frequently Asked Questions
Why is file sync one of the most commonly asked system design questions?
File sync combines five core distributed systems challenges: (1) large-file transfer optimization — chunking and delta encoding for bandwidth efficiency, (2) content deduplication — SHA-256 content addressing to avoid storing identical blocks multiple times, (3) conflict resolution — handling concurrent edits to the same file from multiple devices, (4) real-time notification — push-based sync via WebSocket or long-polling for sub-second change propagation, and (5) storage durability — ensuring 99.999999999% data durability for user files. Dropbox, Google, Microsoft, Apple, and Box all ask this question because it maps directly to their products. Uber, Meta, and Amazon ask it because it tests fundamental distributed storage concepts.
Why does whole-file upload fail at scale?
A 1-byte edit to a 1GB file re-uploads the entire gigabyte — wasting 999,999,999 bytes of bandwidth. On a 100 Mbps connection, this takes 80+ seconds versus 0.03 seconds for a delta upload. For a user editing a Photoshop file 10 times per hour, the naive approach consumes 10GB/hour of upload bandwidth versus 40MB/hour with block-level chunking. At enterprise scale with 10K users editing large files, the aggregate bandwidth waste exceeds the capacity of most internet connections.
How much storage does content-addressed dedup save?
Dropbox reported 30-50% storage savings from content-addressed dedup in their early years. Common scenarios: (1) OS files and system libraries shared across users — identical on every machine, stored once. (2) Version history — editing a 1GB file 10 times stores 10GB without dedup but only the unique blocks (~1.5GB) with dedup. (3) Shared project files — a team of 50 all syncing the same repository stores 50x with naive approach versus 1x + metadata with dedup. The savings compound at scale: at 1M users, dedup typically reduces total storage by 40%.
What happens when two users edit the same shared file simultaneously?
In the naive approach, the last upload wins — there is no conflict detection or merge. If User A and User B both edit a shared file, User A uploads first (version 2), then User B uploads (version 3), overwriting User A's changes entirely. The naive approach has no version history, no conflict detection, and no merge capability. The V1 variant detects conflicts via version numbers and creates a 'conflicted copy'. The V2 variant adds version history for rollback.
At what scale should you migrate from naive to chunked upload?
Migrate when any of these thresholds are hit: (1) average file size exceeds 10MB — at 10MB, whole-file upload takes 0.8 seconds on a 100 Mbps connection, which is tolerable; at 100MB it takes 8 seconds, which is not. (2) Upload failure rate exceeds 5% — indicates network reliability issues that resume capability would solve. (3) Storage costs exceed $10K/month — dedup would cut this by 30-50%. (4) Sync notification delay exceeds user tolerance — polling every 5 seconds is too slow for collaborative workflows. In practice, most teams outgrow the naive approach within 6 months of launch.
Related Templates
Discussion
Sign in to join the discussion.
Ready to design your own Dropbox / File Sync?
Open the simulator, place components on the canvas, wire them up, and run a traffic simulation to see how your architecture performs under real load.
Open Simulator