Web Crawler — Distributed (Bloom + Priority Frontier)

The distributed web crawler with priority frontier and content deduplication is the production-grade architecture used by large-scale search engines crawling billions of pages. It builds on the Producer-Consumer variant by solving three additional problems: lack of priority scheduling (all URLs treated equally), lack of content deduplication (same article stored 100 times across different domains), and DNS overhead (50-100ms per page for domain resolution).

The most impactful addition is the priority-based URL frontier. In the Producer-Consumer variant, URLs are consumed in FIFO order within each Kafka partition — a new, unimportant page from a spam blog is treated with the same urgency as a stale page from a high-authority news site. The priority frontier assigns each URL a score based on domain authority, page freshness (time since last crawl), and link context (anchor text relevance, position on the source page). Workers consume higher-priority URLs first within each partition, ensuring the most valuable pages are always crawled first.

Content deduplication via SimHash addresses a major storage inefficiency. The Producer-Consumer variant deduplicates URLs but not content. The same news article syndicated to 100 aggregator sites has 100 different URLs and is stored 100 times in S3. SimHash generates a 64-bit fingerprint per page; two pages with Hamming distance less than or equal to 3 are classified as near-duplicates. Workers compute SimHash after fetching a page and check the dedup cache before storing. At scale, approximately 30% of unique-URL pages are near-duplicates of already-stored content, saving approximately 15TB/month in storage costs.

DNS caching eliminates a hidden per-page overhead. Every HTTP fetch requires resolving the domain to an IP address. DNS resolution takes 50-100ms on average — at 1000 pages/sec, that is 50-100 seconds of aggregate DNS latency per second. Caching DNS results in Redis with 1-hour TTL eliminates redundant lookups: at steady state, approximately 95% of domains have cached resolutions. The DNS cache also provides a single point for blocklisting known spam and malware domains.

The dedicated LinkExtractor service separates link parsing and scoring from page fetching. In the Producer-Consumer variant, workers parse HTML and publish extracted links inline — this ties CPU-intensive link scoring to the I/O-bound fetch loop. The dedicated LinkExtractor pool receives raw HTML from workers, parses it, scores each outbound link by domain authority and contextual relevance, and publishes scored links back to the priority frontier. This decoupling allows workers to focus on fetching and allows link scoring to scale independently.

This architecture represents the senior-level answer in system design interviews. Candidates who can articulate the priority scheduling trade-offs, explain SimHash content dedup mechanics, reason about DNS cache staleness, and discuss the link scoring pipeline demonstrate the depth of understanding expected for staff-level positions at search companies.

The distributed web crawler uses nine components organized into a priority-aware crawl pipeline: SeedClient, CrawlService, UrlFrontier (Kafka with priority), CrawlWorker pool (50 workers), LinkExtractor (10 workers), DedupCache (Redis Bloom filter + SimHash), DnsCache (Redis), CrawlDatabase (PostgreSQL), and ContentStore (S3).

Seed URLs enter through SeedClient via CrawlService, which validates URLs, checks robots.txt compliance, assigns initial priority scores based on domain authority lookup, and publishes to the UrlFrontier. CrawlService runs on 3 ECS Fargate pods for high availability and also serves admin stats including content dedup metrics, Bloom filter size, and DNS cache hit rates.

The UrlFrontier is Amazon MSK (Managed Kafka) with 64 partitions, partitioned by domain with priority headers. Each message includes URL, domain, crawl depth, priority score (0-1), and source URL. Workers consume from assigned partitions using a priority-aware consumer that processes higher-priority URLs first within each partition. The priority score combines domain authority (0.4 weight), freshness decay (0.3 weight), and link context score from LinkExtractor (0.3 weight).

The CrawlWorker pool consists of 50 ECS Fargate workers — 2.5x the Producer-Consumer variant — each processing approximately 20 pages/sec. For each URL, the worker follows a multi-step pipeline: (1) check DedupCache Bloom filter for URL dedup, (2) resolve domain via DnsCache, (3) fetch page with per-domain politeness delay, (4) compute 64-bit SimHash fingerprint, (5) check DedupCache for content near-duplicates (Hamming distance <= 3), (6) store unique HTML in ContentStore (S3), (7) write metadata to CrawlDatabase, (8) mark URL as seen and store SimHash in DedupCache, (9) pass HTML to LinkExtractor for link scoring.

The DedupCache is a 5-node Amazon ElastiCache Redis cluster with 52 GB memory serving dual purposes: URL Bloom filter (10B URLs, ~10GB, 0.1% false positive rate) and SimHash fingerprint store (10B fingerprints, ~80GB with metadata). Workers check both before storing content. The URL Bloom filter has no TTL (URLs never expire); SimHash entries also persist permanently.

The DnsCache is a 2-node ElastiCache Redis cluster with 13 GB memory caching domain-to-IP mappings with 1-hour TTL. At steady state, approximately 95% of domains are cached. Cache misses trigger actual DNS resolution (50-100ms) and backfill the cache. The single-flight pattern prevents multiple workers from simultaneously resolving the same domain. DNS entries are invalidated on repeated HTTP connection errors.

CrawlDatabase is Amazon RDS PostgreSQL with 16 partitions and 3 replicas. Two tables: pages (url_hash, url, domain, title, status_code, fetched_at, priority_score, simhash, content_dedup flag) and domains (domain, robots_txt, authority_score, last_crawled, page_count). Partitioned by crawl date for efficient freshness queries. At 1000 INSERTs/sec, the primary is moderately loaded.

ContentStore is Amazon S3 storing unique (non-duplicate) HTML. After SimHash content dedup filters out approximately 30% of pages as near-duplicates, the net storage rate is approximately 600 unique pages/sec at 50KB average = 30MB/sec = approximately 75TB/month. S3 lifecycle policy transitions to Infrequent Access at 30 days and Glacier at 90 days.