Semantic Search & RAG

Retrieval-augmented generation (RAG) is the most common production LLM pattern: find the right evidence, put it in the prompt, and make the model answer from it with citations. Underneath the acronym it is an information-retrieval system (parsing, chunking, embeddings, inverted indexes, approximate nearest-neighbour graphs, rerankers) bolted onto a generator. Interviewers use it to see whether you can reason about the whole system: where quality is won or lost, what each index costs in memory and latency, how permissions and freshness work, and how you would measure and debug it. This page teaches each stage at the level of mechanism and trade-off.

Why RAG (and when not)

An LLM's parametric knowledge is frozen at training time, is lossy (it compresses trillions of tokens into weights), cannot be scoped per user, and cannot point to where a fact came from. RAG fixes these by moving the knowledge into an external, queryable store and conditioning generation on retrieved passages. The original formulation combined a dense retriever with a seq2seq generator and treated the retrieved document as a latent variable Lewis+ 2020. Today "RAG" usually means the simpler engineering pattern: retrieve, stuff the prompt, generate.

RAG vs long context vs fine-tuning

Context windows of hundreds of thousands to millions of tokens make "just put everything in the prompt" viable for small corpora. Anthropic's own guidance is that a knowledge base under about 200k tokens (roughly 500 pages) can simply be included in the prompt, especially with prompt caching Anthropic 2024. Beyond that size, or when cost per query, latency, permissions or freshness matter, retrieval wins. Studies comparing the two find long context can match or beat RAG on quality when resources allow, while RAG is far cheaper, which motivates hybrids that route between them Li+ 2024. Retrieval and long context also combine well: retrieval helps even long-context models Xu+ 2023. Models also use long contexts unevenly; information in the middle of a long prompt is used less reliably than at the start or end Liu+ 2023.

Fine-tuning is the wrong tool for injecting facts. A controlled comparison found RAG consistently outperformed unsupervised fine-tuning for knowledge injection, for both existing and new knowledge Ovadia+ 2023. Fine-tune for behaviour: output format, tone, domain jargon, tool-use style, or to make a model better at using retrieved context. Fine-tuning the embedder or reranker on your domain is a different, often very worthwhile, intervention (see below).

The end-to-end pipeline

A RAG system has an offline (indexing) path and an online (query) path. Treat them as two services with a shared contract: the chunk schema (text, embedding, sparse terms, metadata, ACL, source pointer, version).

Document parsing

"Garbage in" is the single biggest cause of bad RAG, and parsing is where it enters. Text from HTML or Markdown is easy. PDFs are hard because a PDF is a list of positioned glyphs, not a document tree: there is no guaranteed reading order, tables are just lines and text runs, multi-column layouts interleave, and scanned PDFs contain no text at all.

Docling, for example, combines a layout analysis model and a table-structure model to convert PDFs into structured output Auer+ 2024; Nougat is a visual transformer that converts academic PDF pages into markup Blecher+ 2023.

Practical rules

• Tables: keep each table intact as one chunk (or row-groups with the header repeated). Serialise to Markdown or HTML; models read those well. Optionally add an LLM-written one-line summary for retrieval while returning the full table to the generator.
• Figures and charts: caption them (VLM description) so they are retrievable by text, and keep the image reference for multimodal generation.
• Boilerplate: strip repeated headers, footers, page numbers and navigation; they pollute embeddings and BM25 statistics.
• Keep provenance: page number, bounding box, section path. You need them for citations and highlighting.
• Route by document type: cheap text extraction for born-digital text, layout pipeline for complex PDFs, OCR/VLM for scans. Detect "no text layer" automatically.
• Validate: sample parsed output by eye, track characters-per-page and table-detection rates, and diff against a small hand-checked set. VLM transcription in particular should be spot-checked for paraphrase.

Chunking strategies

A chunk is the unit you embed, index and retrieve. Chunking sets a basic tension:

• Small chunks have focused embeddings (one topic per vector, so higher precision) but lose context: "It increased 12% year over year" doesn't say what "it" is.
• Large chunks carry context but their embedding averages several topics (diluted similarity), they waste context-window tokens, and fewer fit in the prompt.

Embedding models also have maximum input lengths (often 512 tokens for older BERT-style models, up to 8k or more for newer ones), and quality on long inputs is not uniform even when allowed.

Late chunking

Normally you chunk first, then embed each chunk independently, so a chunk's vector knows nothing about its neighbours. Late chunking reverses the order: encode the full document (up to the model's context length) once, get contextualised token embeddings, then mean-pool the token embeddings inside each chunk's span. Because attention ran over the whole document, a chunk that says "the city" carries information about which city Günther+ 2024. It costs nothing extra at index time but requires an embedder that exposes token embeddings and handles long inputs.

Contextual retrieval

Anthropic's variant uses an LLM: for each chunk, prompt the model with the whole document plus the chunk and ask for a short context to situate it, typically 50–100 tokens, which is prepended to the chunk before both embedding and BM25 indexing. With prompt caching (the document is cached across all its chunks) they report a one-time cost of about $1.02 per million document tokens. On their benchmarks, measured as 1 − recall@20, contextual embeddings cut retrieval failures by 35% (5.7% → 3.7%), adding contextual BM25 brought it to 49%, and adding a reranker to 67% (to 1.9%) Anthropic 2024. Those are the vendor's own numbers on their datasets, but the direction matches common practice: chunk-level context fixes the "orphaned chunk" problem.

Choosing sizes

There is no universal best size; it depends on document type and query type. Common starting points are a few hundred tokens (for example 256–512) with 10–20% overlap, structure-aware where possible, then tune against a golden set. Fact-lookup queries favour smaller chunks; "explain/summarise" queries favour larger units or parent expansion. Always store metadata (title, section path, date, author, doc type, ACL) alongside the chunk, since it powers filters and gives the LLM context cheaply.

Embeddings

Bi-encoders

A bi-encoder maps query and document independently to vectors \(\mathbf{q}, \mathbf{d} \in \mathbb{R}^D\) with a transformer encoder followed by pooling (mean of token embeddings, the [CLS] token, or the last token for decoder-based embedders). Relevance is a cheap similarity, \(s(q,d)=\mathbf{q}^\top\mathbf{d}\) or cosine. Because documents are encoded offline, query time costs one encoder forward pass plus a nearest-neighbour search. Sentence-BERT popularised siamese fine-tuning of BERT for this Reimers+ 2019, and DPR showed dense retrieval beating BM25 on open-domain QA Karpukhin+ 2020.

The cost of independence: the query and document never attend to each other, so all of a document's meaning must be squeezed into one vector before the query is known. That is why cross-encoders (joint encoding) are more accurate and why rerankers exist.

Contrastive training and InfoNCE

Embedders are trained so that a query is closer to its relevant document than to others. With a batch of \(B\) (query, positive) pairs, every other positive in the batch acts as a negative ("in-batch negatives"). The InfoNCE loss van den Oord+ 2018 for query \(i\) is:

This is softmax cross-entropy where the "correct class" is the positive. The temperature \(\tau\) (often around 0.01–0.05 with cosine similarity) sharpens the distribution. Key training levers:

• Batch size: more in-batch negatives make the task harder and the gradient more informative, so embedders train with very large batches (thousands to tens of thousands of pairs; gradient caching helps).
• Hard negatives: passages that look relevant but are not (mined with BM25 or an earlier model). They teach fine distinctions but risk false negatives (unlabelled positives), so teams filter them with a cross-encoder.
• Data: modern recipes do weakly-supervised contrastive pre-training on huge mined pair corpora (title–body, question–answer), then fine-tune on curated labelled sets Wang+ 2022 (E5). LLM-generated synthetic queries are now standard.
• Instruction/prefix tokens: many models expect "query: …" / "passage: …" or a task instruction. Using the wrong prefix measurably degrades retrieval.

Cosine, dot product, normalisation

Cosine similarity is \(\cos(\mathbf{q},\mathbf{d}) = \frac{\mathbf{q}^\top \mathbf{d}}{\lVert\mathbf{q}\rVert\,\lVert\mathbf{d}\rVert}\). If you L2-normalise every vector, cosine equals dot product, and squared Euclidean distance is a monotone function of it: \(\lVert\mathbf{q}-\mathbf{d}\rVert^2 = 2 - 2\,\mathbf{q}^\top\mathbf{d}\). So all three metrics give the same ranking on normalised vectors, and you can use whichever the index implements fastest (inner product usually). Use the metric the model was trained with; some models are trained with unnormalised dot product, where vector norm carries signal (for example popularity or confidence), and normalising them changes rankings.

Dimensions, Matryoshka and quantization

Typical dimensions run from 384 (small MiniLM-class models) through 768 and 1024 (base and large encoders) to 1536–4096 for API and LLM-based embedders. Memory is \(N \times D \times\) bytes per value: 10M chunks × 1024 dims × 4 bytes (float32) ≈ 41 GB of raw vectors, before index overhead. Three ways to shrink it:

• Matryoshka Representation Learning (MRL): train so that every prefix of the vector (first 64, 128, 256, … dims) is itself a good embedding, by summing the contrastive loss over several nested prefix lengths Kusupati+ 2022. You can then truncate (and re-normalise) to trade quality for memory, or do a fast coarse search with short prefixes and rescore with full vectors. Several commercial and open embedders expose a "dimensions" option built on this idea HF blog 2024.
• Scalar (int8) quantization: map each float to 8 bits using per-dimension ranges, 4× smaller.
• Binary quantization: keep only the sign of each dimension, 32× smaller, and similarity becomes Hamming distance (XOR + popcount, extremely fast). Hugging Face's experiments report binary retaining roughly 92.5% of retrieval quality, rising to about 96% when the top candidates are rescored with the float query, and int8 retaining about 99% with rescoring, with speedups of ~25× (binary) and ~3.7× (int8) on average HF blog 2024. The usual pattern: binary or int8 index for candidate generation, then rescore the top few hundred with higher-precision vectors.

Higher dimension does not automatically mean better; quality is model-dependent, and many large-dimension models truncate well thanks to MRL.

Multilingual and domain adaptation

Multilingual embedders map different languages into a shared space so that an English query can retrieve a German document. BGE-M3, for example, supports over 100 languages and produces dense, sparse and multi-vector (ColBERT-style) outputs from one model Chen+ 2024. Cross-lingual quality varies a lot by language pair, so test low-resource languages explicitly.

Domain fine-tuning of the embedder is one of the best levers for specialised corpora (legal, medical, internal jargon, code). Recipe: generate synthetic queries per chunk with an LLM (or mine real query logs and click data), mine hard negatives with the current retriever, filter false negatives with a cross-encoder, fine-tune with InfoNCE (MultipleNegativesRanking-style loss), and evaluate on a held-out golden set. A few thousand to tens of thousands of pairs often give clear recall gains. Remember that changing the embedder means re-embedding the whole corpus; version your embeddings and keep the model ID with the index.

MTEB and its caveats

The Massive Text Embedding Benchmark scores models across task families (retrieval, reranking, clustering, classification, STS, and more) Muennighoff+ 2022; MMTEB extended it to hundreds of tasks and many languages Enevoldsen+ 2025. Caveats an interviewer likes to hear:

• The average score mixes tasks; for RAG look at the retrieval column (nDCG@10 on BEIR-style sets Thakur+ 2021).
• Contamination and overfitting: public test sets leak into training data, so leaderboard gaps of a point or two are often noise relative to your domain.
• Leaderboards rarely reflect your constraints: latency, model size, max input length, licence, cost per million tokens, languages, and whether you can self-host.
• The only number that matters is recall@k on your golden set. Shortlist 3–5 models from the leaderboard, then evaluate.

Sparse retrieval: TF-IDF, BM25, SPLADE

Sparse retrieval represents text as a vector over the vocabulary, almost all zeros, and serves it from an inverted index: for every term, a postings list of (doc id, term frequency). A query touches only the postings lists of its terms, so search over billions of documents is fast, and algorithms like WAND and block-max WAND skip documents that cannot make the top-k.

TF-IDF

Weight a term by how often it appears in the document (term frequency) times how rare it is in the corpus: \(\text{idf}(t) = \log \frac{N}{n_t}\), where \(N\) is the number of documents and \(n_t\) the number containing \(t\). Rare terms ("pgvector", "ERR_4012") are discriminative; common ones ("the", "system") are not. Raw TF grows without bound and long documents get an unfair advantage, which BM25 fixes.

BM25

BM25 (Okapi BM25) scores a document \(D\) for query \(Q = \{q_1,\dots,q_n\}\) as Wikipedia:

with \(\text{IDF}(q_i) = \ln\!\left(\frac{N - n(q_i) + 0.5}{n(q_i)+0.5} + 1\right)\) in the Lucene variant. What the parameters do Elastic blog:

• \(k_1\) (term-frequency saturation, typically 1.2–2.0, Lucene default 1.2): the TF term approaches \(k_1+1\) asymptotically. The 1st occurrence of a term matters a lot, the 10th barely. \(k_1=0\) means binary presence; large \(k_1\) behaves more like raw TF.
• \(b\) (length normalisation, 0–1, default 0.75): how much to penalise documents longer than average. \(b=0\) ignores length; \(b=1\) fully normalises. Lower \(b\) for corpora where long docs really are more relevant.

BM25 is a strong zero-shot baseline. On the BEIR benchmark, many early dense retrievers trained on MS MARCO underperformed BM25 out of domain Thakur+ 2021, which is a big reason hybrid search became standard. BM25 shines on exact identifiers, error codes, names, acronyms and rare jargon; it fails on paraphrase and synonyms ("car" vs "automobile") and needs language-specific analysis (tokenisation, stemming, stop words).

Learned sparse retrieval: SPLADE

SPLADE uses a masked-language-model head to predict, for each input token, a weight over the whole vocabulary, then aggregates (log-saturated, max-pooled) into one sparse vector per text, with a sparsity regulariser (FLOPS loss) so most weights are zero Formal+ 2021 Formal+ 2021b. Two effects: term weighting (learned importance instead of TF-IDF) and expansion (a document about "automobiles" also gets weight on "car"). The output still lives in an inverted index, so you keep exact-match behaviour and interpretability while gaining some semantic matching. Costs: a transformer pass per document and query, and expanded documents have longer postings than BM25, so query latency is higher.

Late interaction: ColBERT and ColPali

Between bi-encoders (one vector per text, no interaction) and cross-encoders (full joint attention, no precomputation) sits late interaction. ColBERT encodes the query and document independently into one vector per token, then scores with MaxSim Khattab+ 2020:

Each query token finds its best-matching document token, and those best matches are summed. Document token vectors are precomputed, so it is much cheaper than a cross-encoder, yet token-level matching captures fine-grained relevance and generalises out of domain better than single-vector models. The price is storage: one vector per token instead of per chunk, tens to hundreds of times more vectors. ColBERTv2 attacks this with residual compression (centroid id + quantised residual per token) and denoised supervision, shrinking the index several-fold Santhanam+ 2021. In practice ColBERT-style models are used either as a first-stage retriever (with a specialised engine such as PLAID) or as a reranker over candidates from a cheaper retriever.

ColPali applies the same idea to documents as images: a vision-language model (PaliGemma) embeds a page screenshot into many patch-level vectors, and text queries are matched with MaxSim against those patches Faysse+ 2024. It skips OCR and layout parsing entirely and handles charts, tables and slides naturally; the paper also introduced the ViDoRe benchmark for visually rich retrieval. Trade-offs: on the order of ~1,000 vectors per page, a VLM forward pass per page at indexing, and the generator then needs to read page images (a multimodal LLM), not extracted text.

Approximate nearest-neighbour indexes

Why approximate: the cost of exact kNN

Exact (flat) search computes the distance from the query to every vector: \(O(ND)\) per query. With \(N = 10^7\) and \(D = 768\) that is about \(7.7\times 10^9\) multiply-adds and, more importantly, reading about 31 GB of float32 vectors from memory. At ~100 GB/s of memory bandwidth that's roughly 0.3 s per query on one machine, memory-bandwidth-bound. Flat search is fine (and gives perfect recall) up to roughly a few hundred thousand to a million vectors, on a GPU, or after a selective filter has shrunk the candidate set. Beyond that you trade a little recall for orders-of-magnitude speed with an ANN index. ANN quality is measured as recall@k against exact search: the fraction of the true top-k the index returns.

HNSW: Hierarchical Navigable Small World graphs

HNSW builds a proximity graph where each vector links to nearby vectors, arranged in layers like a skip list Malkov+ 2016:

• Each inserted node draws a maximum layer \(\ell = \lfloor -\ln(U)\cdot m_L \rfloor\) with \(U\sim\text{Uniform}(0,1)\) and \(m_L = 1/\ln M\). So layer 0 has every node, layer 1 roughly \(1/M\) of them, and so on: few nodes and long-range links at the top, all nodes and short links at the bottom.
• Search: start at the entry point on the top layer, greedily move to the neighbour closest to the query until no improvement, drop down a layer, repeat. On layer 0 run a best-first beam search keeping a candidate list of size efSearch, and return the best k. Upper layers do coarse "zoom-in", layer 0 does the fine search. Hop count grows roughly logarithmically with \(N\).
• Insertion: search for the new node's neighbours with a beam of efConstruction, connect to up to \(M\) of them (\(2M\) on layer 0) using a diversity heuristic that prefers neighbours in different directions over a cluster of near-duplicates, and prune neighbours' lists that overflow.

Memory: full vectors plus the graph. Layer-0 links cost about \(2M \times 4\) bytes per node (32-bit ids), so 128 bytes at M=16, plus a small amount for upper layers. For 768-dim float32 vectors (3,072 bytes) the graph adds only a few percent, but the whole thing should live in RAM, because traversal is random access. That is HNSW's main cost: 100M × 768-dim float32 is about 300 GB before overhead. Mitigations: int8/binary/PQ-compressed vectors inside the graph with rescoring, or a disk-based index.

Deletions and updates: removing a node can disconnect the graph, so most implementations use tombstones (mark deleted, skip at query time, still traverse through) and repair or rebuild later. Heavy churn degrades recall and wastes memory until compaction or rebuild. Build is also expensive: each insert is a search, so building 100M vectors takes hours of many-core CPU. Plan index rebuilds as a first-class operation.

IVF: inverted file with a coarse quantizer

IVF clusters the vectors with k-means into nlist centroids (the coarse quantizer) and stores each vector in the list of its nearest centroid. At query time, find the nprobe nearest centroids and scan only those lists. With \(N\) vectors and \(nlist\) lists, a query scans about \(N \cdot nprobe/nlist\) vectors plus \(nlist\) centroid comparisons.

• nlist: Faiss guidance suggests on the order of \(4\sqrt{N}\) to \(16\sqrt{N}\) clusters Faiss wiki; pgvector suggests rows/1000 up to 1M rows and \(\sqrt{\text{rows}}\) above that, with probes starting at \(\sqrt{\text{lists}}\) pgvector README.
• nprobe: the recall/latency dial. Recall drops for queries near cluster boundaries, since their true neighbours sit in adjacent lists.
• Training: k-means needs a representative sample (Faiss suggests 30–256 training vectors per centroid). If the data distribution drifts (new topics), lists become unbalanced and recall degrades, so retrain periodically.
• IVF uses less memory than HNSW and builds faster, but has a worse recall/latency curve at the same memory unless combined with compression or an HNSW over the centroids.

Product quantization (PQ)

PQ compresses each vector into a few bytes. Split the \(D\)-dim vector into \(m\) subvectors of \(D/m\) dims; for each subspace, learn a codebook of 256 centroids with k-means; store each subvector as the 1-byte id of its nearest centroid. A 768-dim float32 vector (3,072 bytes) with \(m=96\) becomes 96 bytes, 32× smaller. The effective codebook is \(256^m\) points, a huge implicit vocabulary from small tables.

Asymmetric distance computation (ADC): the query is kept exact. Before scanning, precompute a table of distances from each query subvector to all 256 centroids in its subspace (\(m \times 256\) entries). The distance to any database code is then \(m\) table lookups and adds, \(\hat d(q,x)^2 = \sum_{j=1}^{m} T_j[c_j(x)]\), with no decompression. That fits in cache and vectorises well (Faiss's "fast scan" variants use 4-bit codes and SIMD registers) Douze+ 2024. PQ distances are approximate, so systems re-rank the top candidates with full-precision vectors if available.

IVF-PQ

Combine both: IVF picks which lists to scan; within a list each vector is PQ-encoded, usually on the residual (vector minus its centroid), which has a smaller range and so quantises more accurately. This is the classic billion-scale recipe: a billion vectors at 64 bytes each is about 64 GB, which fits on one large machine or GPU Johnson+ 2017.

ScaNN

Google's ScaNN observed that for maximum-inner-product search, not all quantization error is equal: error parallel to the datapoint changes inner products with high-scoring queries much more than orthogonal error. Its anisotropic vector quantization loss weights the parallel component more heavily, improving recall at the same code size Guo+ 2020. ScaNN pairs this with partitioning (tree/IVF-style) and exact rescoring, and it ranked near the top of ann-benchmarks at publication. It is available as an open-source library and inside Google Cloud products (AlloyDB, Vertex AI Vector Search).

DiskANN and Vamana: SSD-scale

When vectors don't fit in RAM, HNSW's random access to full vectors is ruinous on disk. DiskANN builds a flat (single-layer) graph with the Vamana algorithm, whose pruning rule with a parameter \(\alpha > 1\) deliberately keeps some long-range edges so the graph has a small diameter, meaning few hops and therefore few SSD reads Subramanya+ 2019. Layout: full-precision vectors and adjacency lists live together on SSD (one read fetches both), while PQ-compressed vectors live in RAM to guide the beam search. Each hop issues a batch of parallel SSD reads; final candidates are re-ranked with full vectors read from disk. The paper reports serving a billion points on a single machine with 64 GB RAM at 95%+ 1-recall@1 and a few milliseconds latency. FreshDiskANN adds streaming inserts and deletes Singh+ 2021. DiskANN-style indexes now appear in several databases (for example Milvus, Azure services, and pgvector-compatible extensions).

Filtered search: the hard part

Real queries carry filters: tenant = X, date > 2025, doc_type = contract, user can read. Combining filters with ANN is where many production systems struggle.

• Post-filtering: run ANN for top-k′, then drop rows that fail the filter. Simple, but if the filter passes only 1% of rows, a top-100 ANN returns about one valid result. pgvector's docs give this exact example: with a filter matching 10% of rows and the default ef_search of 40, only about 4 rows match on average; version 0.8.0 added iterative index scans that keep scanning until enough results pass pgvector README.
• Pre-filtering: compute the allowed id set first (from a B-tree or bitmap), then search only among those. Exact for brute force over small sets, but inside a graph index it breaks navigation: if most nodes are disallowed, the allowed ones may be disconnected from the entry point.
• Filter-aware traversal: traverse through disallowed nodes but only return allowed ones, and/or add extra edges so filtered subgraphs stay connected. Qdrant's filterable HNSW and ACORN (predicate-agnostic, denser graph with two-hop expansion) are examples Qdrant Patel+ 2024.
• Query planning by selectivity: very selective filter (a few thousand rows) → brute-force over the filtered set; permissive filter → ANN with filter-aware traversal. Good engines estimate selectivity and switch automatically.
• Partitioning: for high-cardinality, always-present filters (tenant id), build separate indexes or namespaces per value. Best isolation and performance, more operational overhead for many small tenants.

Recall–latency–memory trade-offs

All latency and recall figures above are rough orders of magnitude; they depend heavily on dimension, hardware, k and parameter settings.

Benchmark on your own data with ann-benchmarks-style sweeps: plot recall@10 vs QPS for each parameter setting and pick the knee ann-benchmarks.

Vector databases and how to choose

A "vector database" is an ANN index plus everything a database needs: persistence, updates and deletes, metadata filtering, replication, sharding, backups, access control and usually keyword search. The options split into extensions of existing databases, search engines that added vectors, and purpose-built systems.

Choosing criteria

• Scale and growth: under ~10M vectors almost anything works, and staying in your existing Postgres or Elasticsearch is often the right call. Hundreds of millions to billions pushes toward distributed or disk/object-storage-based systems.
• Filtering: how selective and how dynamic are your filters? Test filtered recall, not just unfiltered QPS.
• Hybrid support: native BM25 + vector with fusion in one query vs running two systems and fusing in the app.
• Multi-tenancy: thousands of tenants? Need per-tenant indexes or namespaces, cheap idle tenants, tenant-level deletion.
• Update pattern: streaming upserts and deletes vs nightly batch; consistency (read-your-writes) requirements.
• Ops and cost: managed vs self-hosted, RAM vs SSD vs object storage cost model, backups, observability, team familiarity.
• Data governance: residency, encryption, VPC/BYOC deployment, compliance certifications.

Hybrid search and score fusion

Dense and sparse retrievers fail differently: dense handles paraphrase and semantics, sparse handles exact terms and rare tokens. Running both and merging their results is a cheap, reliable gain. The problem is that their scores are on incomparable scales (BM25 is unbounded and query-dependent; cosine lies in [−1, 1] and is usually compressed into a narrow band).

Reciprocal Rank Fusion

RRF ignores scores and uses only ranks Cormack+ 2009:

where \(R\) is the set of result lists, \(\text{rank}_r(d)\) is \(d\)'s 1-based rank in list \(r\) (documents absent from a list contribute nothing), and \(k=60\) is the constant from the original paper. The constant damps the dominance of the very top ranks: with \(k=60\), rank 1 scores 1/61 ≈ 0.0164 and rank 10 scores 1/70 ≈ 0.0143, so a document ranked well in both lists beats one ranked first in only one. Example: doc A is #1 in BM25 and #8 in dense: 1/61 + 1/68 ≈ 0.0311. Doc B is #3 and #2: 1/63 + 1/62 ≈ 0.0320, so B wins. RRF needs no tuning or score calibration, which is why it is the default in many engines.

Weighted score fusion

Alternatively, normalise each list's scores (min-max within the result set, or z-score) and combine linearly: \(s(d) = \alpha\,\tilde s_{\text{dense}}(d) + (1-\alpha)\,\tilde s_{\text{sparse}}(d)\). It keeps magnitude information (a huge BM25 win counts for more than a narrow one) and can beat RRF when \(\alpha\) is tuned on labelled data, but min-max normalisation is sensitive to outliers and to how many results you fetched, and the best \(\alpha\) varies by query type. A practical pattern: RRF by default, tuned weights once you have a golden set, possibly a per-query \(\alpha\) predicted by a router (keyword-looking queries lean sparse).

Reranking

Retrieval is optimised for recall at low cost; reranking is optimised for precision on a small candidate set. The canonical reranker is a cross-encoder: concatenate "[CLS] query [SEP] passage" and run a transformer that outputs a relevance score. Because every query token attends to every passage token, it models relevance much better than a dot product between independent vectors. BERT cross-encoders on MS MARCO were the first big demonstration Nogueira+ 2019.

The cost is one forward pass per (query, candidate) pair at query time, with nothing precomputable. Back-of-envelope: a ~100–500M-parameter cross-encoder scoring 100 passages of ~256 tokens on a GPU takes on the order of tens to a couple of hundred milliseconds batched; on CPU it can be far slower. So the candidate count (50–200) is a direct latency dial.

RankGPT showed that instructing GPT-4 to output a permutation of passages (listwise, with sliding windows) was competitive with or better than supervised rerankers, and that the ability could be distilled into small models Sun+ 2023. Commercial rerank APIs and open rerankers (BGE-reranker, Qwen3-Reranker, mxbai, Jina) are cross-encoder- or small-LLM-based.

Latency budgets. A typical interactive budget might allot ~10–50 ms to query embedding and ANN, ~50–200 ms to reranking, and the rest to LLM time-to-first-token. Techniques: rerank fewer candidates, use a smaller or distilled reranker, truncate passages, cache rerank results for repeated queries, and run rerank on GPU. The reranker score also gives you a threshold: if the best candidate scores below a cutoff, abstain or trigger a fallback (query rewrite, web search) rather than generating from junk.

Query-side techniques

User queries are short, ambiguous, conversational ("what about for contractors?") and phrased unlike the documents that answer them. Query understanding transforms the query before retrieval, usually with a fast LLM call.

HyDE generates a hypothetical document with an instruction-following LLM and uses an unsupervised contrastive encoder to embed it; the encoder's dense bottleneck filters out incorrect details, and it gave strong zero-shot retrieval without relevance labels Gao+ 2022. Query2doc expands queries with LLM pseudo-documents and improved BM25 notably Wang+ 2023. Step-back prompting has the model abstract to higher-level concepts before reasoning Zheng+ 2023.

Advanced RAG patterns

GraphRAG: entity and community graphs

Plain vector RAG answers "local" questions well (a fact lives in a few chunks) and "global" questions badly: "What are the main themes across these 5,000 interviews?" has no single chunk that answers it. Microsoft's GraphRAG builds, at index time, an LLM-extracted knowledge graph of entities and relationships from all chunks, partitions it into hierarchical communities with the Leiden algorithm, and has an LLM pre-write a summary per community Edge+ 2024.

• Global search: map-reduce over community summaries. Each summary produces a partial answer with a helpfulness score; the best partial answers are reduced into a final one. Good for sensemaking and theme questions over a whole corpus; the paper reports gains in comprehensiveness and diversity over vector RAG on such questions.
• Local search: find entities related to the query, then fan out to their neighbours, relationships, community summaries and source chunks. Good for entity-centric questions ("What has Supplier X been involved in?").

Costs: indexing is LLM-heavy (entity extraction over every chunk plus summaries), so it is expensive and slow to refresh, and extraction quality bounds everything downstream. Lighter variants like LightRAG combine graph structure with vector retrieval and incremental updates Guo+ 2024. Use graph approaches when the questions are relational or global; for most fact-lookup workloads hybrid + rerank is cheaper and as good.

RAPTOR: hierarchical summaries

RAPTOR embeds chunks, clusters them (soft clustering with Gaussian mixtures over dimension-reduced embeddings), summarises each cluster with an LLM, then recursively clusters and summarises the summaries, building a tree from leaves (raw chunks) to root (corpus-level summary) Sarthi+ 2024. At query time, searching a "collapsed tree" (all levels in one index) lets a question match a detailed leaf or a high-level summary, whichever fits. Good for long documents (books, long reports) and questions needing synthesis across sections; costs are index-time LLM summarisation and refresh complexity.

Agentic RAG: retrieval as a tool

Instead of one fixed retrieve-then-generate pass, give the LLM a search tool (or several: vector search, keyword search, SQL, web) and let it decide when and what to search, read results, and search again, in a ReAct-style loop that interleaves reasoning and actions Yao+ 2022. This handles multi-hop questions naturally (look up X, use the result to formulate the next query), can reformulate after a bad result, and can stop when it has enough evidence. Newer work trains models with reinforcement learning to interleave reasoning and search calls Jin+ 2025, and "deep research" style products run long agentic retrieval loops.

Trade-offs: higher and more variable latency and cost (several LLM turns), harder evaluation (trajectories, not single calls), and failure modes such as looping, over-searching, or stopping early. Guardrails: step limits, budgets, deduplication of already-seen results, and logging every tool call for debugging.

Corrective and self-reflective RAG

• Self-RAG trains the generator to emit special reflection tokens that decide whether to retrieve, judge whether each passage is relevant, whether the generated segment is supported by it, and whether the response is useful, enabling adaptive retrieval and critique at inference time Asai+ 2023.
• Corrective RAG (CRAG) adds a lightweight retrieval evaluator that grades retrieved documents. Confident-correct results are refined (split into strips, irrelevant strips filtered); incorrect ones trigger web search; ambiguous cases combine both Yan+ 2024.

The production-friendly takeaway is the pattern, not necessarily the trained models: grade the retrieval before generating (reranker score threshold or an LLM relevance check), and branch: answer, re-query, escalate to another source, or abstain. Google researchers formalised a related idea as "sufficient context", whether the retrieved context is enough to answer at all, and found models often answer confidently when it isn't Joren+ 2024.

Permissions, ACLs and multi-tenancy

The rule: a user must never see, in an answer or a citation, content they could not open in the source system. The LLM cannot be trusted to "ignore" documents in its context, and prompt injection can make it reveal them, so enforcement must happen in retrieval, before anything reaches the prompt.

• Capture ACLs at ingest: store allowed principals (users, groups, roles) per document on each chunk. Expand groups at query time from the identity provider (more flexible, as group memberships change often) or at index time (faster queries, but stale when groups change).
• Filter in the query: allowed_principals ∩ user_principals ≠ ∅ as a metadata filter on both dense and sparse retrieval. That makes ACLs a filtered-ANN problem, so test recall for users with narrow access.
• Propagate changes fast: permission revocations should reach the index within minutes, which means ACL-only updates without re-embedding. Some systems add a late check: re-verify access against the source system for the final top-k before generation (defence in depth, small latency cost).
• Multi-tenancy isolation levels: (1) shared index with a tenant-id filter: cheapest, but a filter bug leaks data and noisy neighbours share resources; (2) namespace/partition per tenant: strong logical isolation, per-tenant deletion and indexing, good when tenants are many and mostly small; (3) separate index, cluster or account per tenant: strongest isolation and compliance (residency, encryption keys), highest cost. Many products mix them: dedicated for large enterprise tenants, namespaces for the long tail.
• Everything derived inherits ACLs: contextual summaries, RAPTOR or GraphRAG community summaries, caches, and semantic caches can leak across permission boundaries if they mix documents with different ACLs. Build derived artefacts per permission scope or tag them with the intersection of their sources' ACLs.
• Deletion: right-to-erasure means removing the chunk, its vectors (tombstones must be compacted), derived summaries, caches and eval logs.

Freshness and incremental indexing

Indexes go stale in three ways: new or changed documents, deleted documents, and changed permissions. Design the ingest path as a change stream.

• Change detection: webhooks or change feeds from sources where available, otherwise periodic crawls comparing modified timestamps or content hashes. Postgres sources can use logical replication or CDC.
• Idempotent upserts: deterministic chunk ids (doc id + chunk index or content hash), so reprocessing a document replaces its chunks. When a document changes, delete all its old chunks before or atomically with inserting the new ones; otherwise stale chunks linger, especially when the chunk count changes.
• Skip unchanged work: hash each chunk's text; re-embed only chunks whose hash changed. Embedding is often the dominant ingest cost, contextual retrieval even more so.
• Index mechanics: HNSW handles inserts online but deletes as tombstones; IVF appends easily but drifts as data changes (retrain centroids); many engines use an LSM-like design with small fresh segments merged into big optimised ones, so recent data is searchable within seconds.
• Versioning: keep doc_version, indexed_at and embedding_model on every chunk. Model upgrades become blue-green re-indexes; time-sensitive queries can filter or boost by recency.
• Monitor lag: source-change-to-searchable latency, per-connector error rates, and a canary document updated on a schedule that synthetic queries check.

Evaluating RAG

Evaluate the two halves separately, then end to end. If you only measure final answers, you cannot tell whether a regression came from retrieval or generation.

Retrieval metrics

Given a query with a set of relevant chunks (or documents) and a ranked list of retrieved ones:

• Recall@k = (relevant items in the top k) / (total relevant items). The most important RAG retrieval metric: if the evidence isn't in the top k you pass to the LLM, the answer can't be grounded. "Hit rate@k" is the binary version (at least one relevant in top k).
• Precision@k = (relevant in the top k) / k. Matters because irrelevant context costs tokens and can distract the generator.
• MRR (mean reciprocal rank) \(= \frac{1}{|Q|}\sum_{i=1}^{|Q|} \frac{1}{\text{rank}_i}\), where \(\text{rank}_i\) is the position of the first relevant result. Rewards putting one good answer near the top; ignores the rest.
• nDCG@k handles graded relevance (0 = irrelevant, 1 = partial, 2 = perfect) and position discounting Wikipedia:

where IDCG is the DCG of the ideal ordering, so nDCG lies in [0, 1]. (A linear-gain variant uses \(\text{rel}_i\) instead of \(2^{\text{rel}_i}-1\).) Example with binary relevance, relevant items at ranks 1 and 3 of top 3, two relevant total: DCG = 1/1 + 1/2 = 1.5; IDCG = 1 + 1/log₂3 ≈ 1.631; nDCG ≈ 0.92. nDCG@10 is the standard BEIR/MTEB retrieval metric.

Generation metrics

RAGAS popularised reference-free versions of several of these (faithfulness, answer relevance, context relevance) computed with LLM judges Es+ 2023; ARES trains lightweight judges on synthetic data and uses a small human-labelled set to put confidence intervals on the scores Saad-Falcon+ 2023. LLM judges have known biases (verbosity, position, self-preference) and must be calibrated against human labels on a sample before you trust them.

Building a golden set

1. Collect real questions: from logs, support tickets, or interviews with users. Synthetic LLM-generated questions (from sampled chunks) fill coverage gaps, but they tend to be easier and to reuse the chunk's wording, which flatters lexical and dense retrieval alike.
2. Label: for each question, the relevant chunk or document ids (for retrieval metrics) and a reference answer (for generation metrics). Have domain experts review; label at document level if chunk ids churn with re-chunking.
3. Stratify: fact lookup, multi-hop, comparison, numeric/table, time-sensitive, unanswerable, permission-restricted, and adversarial (prompt injection in documents). Report metrics per stratum; averages hide regressions.
4. Size: a hundred or two well-chosen questions catches large regressions; a few hundred to a thousand gives tighter comparisons. Keep a frozen test split you don't tune on.
5. Run in CI: every change to parsing, chunking, embedder, index parameters, prompt or model runs the suite; track metrics over time. Add every production failure as a new test case.

Failure modes and debugging

The first debugging question for any bad answer is attribution: was the right evidence in the context window?

Log every request with the rewritten query, filters, candidate ids and scores from each retriever, the reranked list, the final prompt, and the answer. Then attribution is a lookup, not guesswork. A useful offline test: give the generator the gold chunk directly ("oracle context"). If it still fails, the problem is generation; if it succeeds, the problem is retrieval.

Interview question bank