Retrieval-augmented generation

Why this matters

An LLM knows only what was in its training data, up to its cutoff, and it will answer confidently either way. Your organisation's policies, product manuals and last week's tickets are not in there. Retrieval-augmented generation (RAG) fixes this without retraining: at question time, find the most relevant passages from your own documents and put them in the prompt, so the model answers from evidence and can cite it. RAG is the most common LLM pattern in industry, and most of its quality comes from ordinary data and search engineering, not from the model.

The core idea: retrieve, then generate

A RAG system has two phases.

Indexing (offline, once per document update):

1. Load documents (PDFs, web pages, wiki exports, tickets) and extract clean text.
2. Chunk them into passages of a few hundred tokens, with a small overlap so sentences are not cut in half.
3. Embed each chunk with an embedding model (the text lesson) and store the vectors plus metadata (source, title, date, permissions) in a vector index.

Answering (online, per question):

4. Embed the question with the same model.
5. Retrieve the top-k most similar chunks (cosine similarity), often 3 to 10.
6. Generate: build a prompt with instructions, the retrieved chunks and the question, and send it to the LLM.

question --> embed --> vector search --> top-k chunks --+
                                                         +--> prompt --> LLM --> answer + citations
instructions -------------------------------------------+

A typical prompt template:

Answer the question using ONLY the sources below. Cite sources as [1], [2].
If the sources do not contain the answer, say "I don't know".

[1] (returns-policy.md) Items can be returned within 30 days of delivery...
[2] (gift-cards.md) Gift cards are final sale and cannot be returned...

Question: Can I return a gift card?

Worked example: sizing the pieces

A help centre has 400 articles, average 1,200 words each. With chunks of about 300 words and 50 words of overlap, each article yields about 5 chunks, so the index holds roughly 2,000 vectors. With a 384-dimension embedding model that is about 3 MB of 32-bit floats: a NumPy array or a small library (FAISS, Chroma, or the pgvector extension for PostgreSQL) is plenty. You only need a dedicated vector database at millions of chunks or with heavy concurrent traffic.

Design decisions that matter

• Chunk size: small chunks retrieve precisely but lose context; large chunks carry context but dilute the match. Start around 200–500 tokens and test. Split on headings and paragraphs rather than fixed character counts.
• Metadata: keep the source, section title and date with each chunk. Use it to filter (only current policies, only this customer's region) and to cite.
• Hybrid search: embeddings capture meaning but can miss exact terms such as product codes or error numbers. Combining them with keyword search (BM25) usually helps.
• Re-ranking: retrieve 20–50 candidates cheaply, then re-order them with a cross-encoder model and keep the best 5.
• Access control: filter by the user's permissions before retrieval. A RAG system must never show someone a document they could not open themselves.
• Freshness: re-index when documents change, and remove deleted ones.

Evaluate retrieval separately from generation

When an answer is wrong, find out which half failed:

• Retrieval: was the right chunk in the top-k? Measure recall@k on a set of questions where you know which source holds the answer.
• Generation: given the right chunks, did the model answer correctly, cite correctly, and refuse when the answer was missing (faithfulness)?

If recall@5 is 60%, no prompt engineering will fix the answers. Improve chunking, search or metadata first. The next lesson builds the evaluation set.

Risks

• Hallucination still happens. Instruct the model to use only the sources and to say "I don't know", then test that it does.
• Prompt injection: a retrieved document can contain instructions ("ignore the above and..."). Treat retrieved text as data, keep system instructions separate, and limit what the app can do with the output.
• Privacy: sending internal documents to a hosted LLM shares them with the provider. Check policy, or use an open-weight model locally.

Common mistakes

• Different embedding models for the index and the query.
• No overlap, so the key sentence is split across two chunks.
• Judging the system on five hand-picked questions.
• Ignoring tables and PDFs: bad text extraction is the most common silent failure. Read some chunks yourself.

Quick recap

• RAG = embed and index your documents, retrieve the top-k chunks for each question, and generate an answer grounded in them, with citations.
• Chunking, metadata, hybrid search and access control matter more than model choice.
• Measure retrieval (recall@k) and generation (correctness, faithfulness) separately.

Try it

Open the Do tab. You will design a RAG pipeline for a small document set of your choice and test its retrieval in a free notebook, with no paid API needed.