Web Crawler — Producer-Consumer (Kafka Frontier)

The producer-consumer web crawler is the standard production approach for crawling at moderate to large scale. It solves the three fundamental problems that make the naive single-threaded approach unworkable: sequential throughput (5 pages/sec ceiling), lack of deduplication (infinite re-crawling via link cycles), and no crash recovery (in-memory queue lost on restart).

The key architectural shift is decoupling URL discovery from URL fetching via a durable distributed queue. Instead of a single service maintaining an in-memory queue, the system uses Kafka as a distributed URL frontier. Seed URLs and extracted links are published to Kafka topics partitioned by domain. A pool of 20 CrawlWorkers consume URLs from Kafka partitions in parallel — each worker independently fetches pages, checks dedup, stores content, and publishes extracted links back to Kafka. This producer-consumer loop is the core of the recursive crawl.

The parallelism unlocked by this architecture is transformative. Where the naive approach processes 1 URL at a time (5 pages/sec), 20 workers processing in parallel achieve 400 pages/sec — an 80x improvement. Each worker handles approximately 20 pages/sec, limited by politeness delays (1 request/sec per domain) rather than compute. Adding more workers and Kafka partitions scales throughput linearly until network or storage becomes the bottleneck.

URL deduplication is handled by a Redis Bloom filter shared across all workers. Before fetching a URL, each worker checks the Bloom filter. At 10B URLs, the filter uses approximately 10GB RAM with O(1) lookups and a 0.1% false positive rate. The false positive rate means approximately 10M URLs per 10B are incorrectly skipped — acceptable for general crawling but not for exhaustive archival. The Bloom filter rejects approximately 60% of extracted links as already-seen, saving significant fetch overhead.

Crash recovery is handled by Kafka's offset-based consumption model. Each worker tracks its position in the Kafka partition. If a worker crashes, Kafka replays uncommitted offsets to a replacement worker — no URLs are lost, no re-crawling of already-committed pages. This is a fundamental improvement over the in-memory queue, which loses all state on crash.

The architecture uses domain-based partitioning in Kafka: all URLs for a given domain are routed to the same partition, which is consumed by the same worker or worker group. This enables per-domain politeness enforcement — the worker can maintain a per-domain rate limiter ensuring no more than 1 request per second to any single domain. Without domain partitioning, concurrent workers could inadvertently DDoS a domain by crawling it in parallel.

This template is the answer interviewers expect when they ask 'How would you scale a web crawler?' The progression from naive (sequential, no dedup) to producer-consumer (parallel, Bloom filter, Kafka durability) demonstrates understanding of the fundamental scaling techniques.

The producer-consumer web crawler uses six components organized into a pipeline: SeedClient, CrawlService, UrlFrontier (Kafka), CrawlWorker pool, DedupCache (Redis), CrawlDatabase (PostgreSQL), and ContentStore (S3). The architecture separates URL discovery (CrawlService + LinkExtraction within workers) from URL fetching (CrawlWorkers) via Kafka, enabling parallel processing and crash recovery.

Seed URLs enter through SeedClient via the CrawlService API. CrawlService is a lightweight orchestration layer running on 2 ECS Fargate pods — it validates URLs, checks robots.txt compliance, and publishes seed URLs to the UrlFrontier (Kafka topic). CrawlService also serves an admin stats API that reads from CrawlDatabase. At approximately 10 RPS for seed injection, CrawlService is dramatically over-provisioned — its primary role is the admin interface, not the crawl engine.

The UrlFrontier is Amazon MSK (Managed Kafka) with 32 partitions, partitioned by domain. Each message contains a URL, domain, crawl depth from seed, and optional priority score. Kafka serves as both the URL queue (durable, partitioned) and the crash recovery mechanism (offset-based replay). At 400 URLs/sec consumption rate, Kafka is well within its throughput limits. Messages are retained for 7 days to support re-crawl scenarios.

The CrawlWorker pool is the crawl engine — 20 ECS Fargate workers consuming from Kafka partitions. Each worker processes URLs sequentially within its assigned partitions. For each URL: (1) check DedupCache for the URL hash — if seen, skip; (2) fetch the page via HTTP with politeness delay; (3) parse HTML to extract outbound links; (4) store raw HTML in ContentStore (S3); (5) write page metadata to CrawlDatabase; (6) mark the URL as seen in DedupCache; (7) publish extracted links back to UrlFrontier. Each worker processes approximately 20 pages/sec, limited by the 1 req/sec/domain politeness constraint across the domains assigned to its partitions.

DedupCache is Amazon ElastiCache for Redis running a Bloom filter for URL deduplication. Three nodes with 26 GB memory support 10B URLs with O(1) lookups. The Bloom filter is checked on every URL before fetching — approximately 60% of extracted links are rejects (already-seen URLs). Workers also cache per-domain robots.txt in this Redis cluster with 1-hour TTL.

ContentStore is Amazon S3 storing raw HTML. Each page is stored as an object keyed by URL hash. At 400 pages/sec with 50KB average page size, S3 ingests approximately 20 MB/sec (approximately 50 TB/month). S3 lifecycle policies move content to Infrequent Access after 30 days.

CrawlDatabase is Amazon RDS PostgreSQL storing crawl metadata — URL, domain, title, status code, timestamp, and outbound link count. Indexed by domain and fetched_at for freshness queries and admin dashboards. At 400 INSERTs/sec, a single primary handles the write load.