Web Crawler — Distributed URL Frontier

The web crawler is a classic system design interview question favored by companies like Google, Amazon, and LinkedIn because it tests a candidate's understanding of distributed task processing, data deduplication at massive scale, and polite interaction with external systems. The core problem — systematically fetching web pages, extracting links, and storing content — sounds simple, but the constraints at internet scale introduce challenges in queue management, duplicate detection, failure recovery, and storage that require careful architectural reasoning.

At production scale, a web crawler like Googlebot must fetch billions of pages per month while respecting the politeness conventions of the web. The robots.txt protocol and per-domain rate limiting (typically one request per second per domain) mean that raw throughput is constrained not by compute or bandwidth but by the need to distribute crawl pressure across millions of domains. A crawler that hammers a single domain with hundreds of concurrent requests will get blocked, damage the target site's performance, and violate widely accepted crawling norms.

Deduplication is another major challenge. As the crawler follows links across pages, it encounters the same URLs repeatedly — approximately 60% of extracted links point to already-crawled pages. Without efficient deduplication, the crawler wastes resources re-fetching known content. At a corpus of 10 billion URLs, a naive hash set would consume hundreds of gigabytes of memory. Probabilistic data structures like Bloom filters reduce this to roughly 10GB with a controlled false positive rate, trading a tiny fraction of missed pages for dramatic memory savings.

This template models the complete crawler architecture: a CrawlService for seed URL injection and admin operations, a Kafka-based URL frontier partitioned by domain for politeness enforcement, a pool of CrawlWorker pods that fetch pages and extract links, a Redis Bloom filter for URL deduplication, PostgreSQL for crawl metadata and scheduling, and S3 for raw HTML content storage. The simulation demonstrates how worker count affects crawl throughput, how Bloom filter false positive rates impact coverage, and how Kafka's partitioned architecture naturally enforces per-domain rate limiting.

The web crawler architecture follows a producer-consumer pattern with Kafka as the central URL frontier. The system operates as a recursive loop: seed URLs enter the frontier, workers consume and fetch them, extracted links are published back to the frontier for the next crawl round, and the cycle continues until the frontier is exhausted or a depth limit is reached.

The crawl begins when an administrator injects seed URLs via the CrawlService API. CrawlService validates the URLs, checks robots.txt compliance, and publishes them to the UrlFrontier (Amazon MSK / Kafka), partitioned by domain. Domain-based partitioning is a deliberate architectural choice: all URLs for a given domain land on the same Kafka partition, which means a single CrawlWorker (or worker group) handles that domain's queue. This naturally enforces per-domain politeness — the worker processes one URL at a time from that domain's partition with a configurable delay between requests.

CrawlWorker pods consume URLs from Kafka and execute a multi-step processing pipeline for each URL. First, the worker checks the DedupCache (Redis Bloom filter) to determine whether the URL has already been crawled. If the Bloom filter returns positive, the URL is skipped. If negative, the worker fetches the page via HTTP, respecting per-domain rate limits. After fetching, the worker parses the HTML to extract outbound links and page metadata (title, status code, content hash).

The worker then performs three write operations: raw HTML content is stored in ContentStore (S3) keyed by URL hash, page metadata is written to CrawlDatabase (PostgreSQL) for admin queries and freshness-based re-crawl scheduling, and the URL is marked as seen in the DedupCache. Finally, extracted outbound links are published back to the UrlFrontier, completing the recursive loop. Each link carries a depth counter that is incremented from its parent page, enabling the system to enforce a maximum crawl depth.

The architecture is naturally horizontally scalable. Adding more CrawlWorker pods and Kafka partitions increases parallelism proportionally. Kafka's offset-based consumption model provides crash recovery: if a worker dies mid-crawl, its uncommitted offsets are replayed by another worker, ensuring no URLs are lost. The 20-worker configuration sustains approximately 400 pages per second (each worker handles roughly 20 pages per second due to politeness delays), yielding one billion pages per month at steady-state throughput.