Interview Walkthrough: RAG-Powered Search

Retrieval-Augmented Generation (RAG) has become one of the most commonly asked system design interview questions since 2024, reflecting the rapid adoption of large language models in production systems. RAG combines information retrieval with LLM-powered generation to answer questions grounded in a specific knowledge base, eliminating the hallucination problem of pure LLM-based answers by providing citations to source documents.

The architecture splits into two pipelines: ingestion and query. The ingestion pipeline processes source documents through a series of stages. First, documents are parsed from their source format (PDF, HTML, Markdown, database records) into plain text. Next, the text is split into chunks -- contiguous segments of a few hundred tokens each. Chunking strategy is a critical design decision: fixed-size chunks (e.g., 512 tokens with 50-token overlap) are simple and predictable, while semantic chunking (splitting at paragraph or section boundaries using NLP) preserves the coherence of each chunk but produces variable-length segments. Each chunk is then passed through an embedding model (e.g., OpenAI text-embedding-3-small, or open-source models like E5-large or BGE-large) to produce a dense vector representation (typically 768 to 1536 dimensions). The embeddings are stored in a vector database alongside the original chunk text and metadata.

The query pipeline processes user questions in real time. The query text is embedded using the same embedding model used for documents, producing a query vector. This vector is used for approximate nearest neighbor (ANN) search against the vector database, retrieving the top-K most semantically similar chunks (typically K=10-20). ANN search uses algorithms like HNSW (Hierarchical Navigable Small World) or IVF (Inverted File Index) to find approximate nearest neighbors in sub-millisecond time, even across millions of vectors. The retrieved chunks are then optionally reranked using a cross-encoder model that scores each query-chunk pair more accurately than the embedding similarity alone, reducing the set to the top 3-5 most relevant chunks. Finally, the selected chunks are inserted into a prompt template along with the user's question, and an LLM generates a natural-language answer with citations referencing the source chunks.

Hybrid search is an important advanced pattern. Pure vector search excels at semantic matching (finding conceptually similar content) but can miss exact keyword matches that a traditional BM25 index would catch. Hybrid search runs both a BM25 keyword search and a vector similarity search in parallel, then fuses the results using reciprocal rank fusion (RRF). RRF computes a combined score for each result based on its rank in each search, giving credit to results that appear highly in either or both. This approach handles both the case where the user asks a conceptual question (vector search dominates) and the case where they search for a specific error message or product name (keyword search dominates).

A RAG system works like a research assistant in a library. When you ask a question, the assistant does not try to answer from memory alone (which might be inaccurate or outdated). Instead, they walk to the shelves (vector database), find the most relevant books and pages (retrieval), photocopy the key passages (context), and then write a summary answer that cites each source (generation). If the assistant only used their memory, they might confidently state incorrect facts (hallucination). By grounding every claim in a specific source, the answer is verifiable and trustworthy.