RAG (Retrieval-Augmented Generation)

Retrieval-Augmented Generation (RAG) combines a retrieval system (built on embeddings and vector search) with a language model — retrieving relevant, up-to-date information at query time and giving the model that context before it generates its answer.

The Core Problem RAG Solves

A language model's knowledge is fixed at training time — it can't know about events after its training cutoff, doesn't have access to private or proprietary documents it was never trained on, and can sometimes generate plausible-sounding but factually incorrect content (hallucination, see LLM Fundamentals). RAG addresses all three: retrieving relevant, current, verifiable documents and providing them as context lets the model ground its answer in actual retrieved evidence, rather than relying purely on its parametric (trained-in) knowledge.

The RAG Pipeline

1. Index: embed a document collection and store the vectors in a vector database (offline, done once, updated as documents change).
2. Retrieve: embed the user's query, search the vector database for the most relevant document chunks.
3. Augment: insert the retrieved chunks into the prompt as context.
4. Generate: the language model produces an answer, grounded in the retrieved context.

Code — A Minimal RAG Pipeline

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

embed_model = SentenceTransformer('all-MiniLM-L6-v2')

# 1. Index: embed and store documents
documents = ["Doc 1 text...", "Doc 2 text...", "Doc 3 text..."]
doc_embeddings = embed_model.encode(documents)
index = faiss.IndexFlatL2(doc_embeddings.shape[1])
index.add(np.array(doc_embeddings).astype('float32'))

# 2. Retrieve: find the most relevant documents for a query
query = "What is the return policy?"
query_embedding = embed_model.encode([query]).astype('float32')
distances, indices = index.search(query_embedding, k=3)
retrieved_docs = [documents[i] for i in indices[0]]

# 3 & 4. Augment and generate
context = "\n".join(retrieved_docs)
prompt = f"Context:\n{context}\n\nQuestion: {query}\nAnswer based only on the context above:"
answer = llm.generate(prompt)

Why RAG Reduces Hallucination

Explicitly instructing the model to answer based on the provided retrieved context — rather than its general trained-in knowledge — grounds its output in specific, verifiable source material. This doesn't eliminate hallucination entirely, but meaningfully reduces it for questions the retrieved documents actually address, and importantly, allows citing the specific source documents behind an answer, adding a layer of verifiability that a purely parametric answer lacks.

Chunking — A Critical, Often-Overlooked Detail

Documents are typically split into smaller chunks before embedding and indexing, since embedding an entire long document as one vector loses fine-grained detail. Chunk size involves a real tradeoff: chunks too small lose surrounding context; chunks too large dilute the specific relevant information within irrelevant surrounding text, both of which can degrade retrieval quality.

Common Mistakes

• Not instructing the model explicitly to rely on the provided context — without this, the model may still lean on its own possibly outdated or incorrect parametric knowledge rather than the retrieved documents.
• Using a chunking strategy poorly matched to the document type and query patterns — this directly affects retrieval quality, often more than the choice of embedding model itself.

Interview Relevance

Q: "Why does RAG help reduce hallucination in language model outputs, and what does it not fully solve?" RAG grounds the model's generation in specific, retrieved source documents provided as context, rather than relying solely on the model's fixed, trained-in parametric knowledge — this meaningfully reduces hallucination for questions the retrieved documents actually address well, and enables citing verifiable sources. It doesn't fully solve hallucination, however: the model can still misinterpret or misrepresent the retrieved context, and if retrieval itself fails to find genuinely relevant documents, the model may fall back on ungrounded generation or produce a confidently wrong answer despite having "context" to work from.

Practice Question

Why might splitting documents into very large chunks before indexing actually hurt RAG retrieval quality, compared to well-sized chunks?

Related DL Notes

• Vector Databases
• Embeddings
• LLM Fundamentals