Retrieval-augmented generation (RAG)

Intermediate

Introduction

A language model knows only what was in its training data, up to its knowledge cutoff, and nothing about your company's handbook, your course notes or yesterday's news. Asked anyway, it may produce a confident, fluent, wrong answer: a hallucination. Retrieval-augmented generation (RAG) fixes this for many tasks by handing the model the relevant facts at the moment it answers. It is the most common way businesses build assistants on their own documents.

How it works

RAG has three steps:

  1. Retrieve: search a collection of document chunks for the ones most relevant to the question, with keyword search, semantic search or both (previous lesson).
  2. Augment: build a prompt that contains the instructions, the retrieved chunks numbered as sources, and the question.
  3. Generate: the model writes the answer from the prompt, citing the sources it used.

The instructions matter. A typical grounded prompt says: answer only from the sources; cite them by number; if the sources do not contain the answer, say so instead of guessing. Numbered citations let readers check every claim.

RAG fails in recognizable ways:

  • Retrieval misses: the right chunk is not in the top results (wrong chunk size, synonyms, poor ranking). The model cannot use what it never sees.
  • Too much context: many weakly related chunks crowd the context window and distract the model.
  • Ignoring the sources: the model answers from memory anyway; strong instructions and checks reduce this.
  • Outdated or conflicting documents: the answer is only as good as the collection.

Because of this, RAG systems are evaluated in two parts: does retrieval return the right chunks for real questions, and is each answer supported by its cited sources?

Example

chunks = [
    "The library opens at 9 am and closes at 6 pm on weekdays.",
    "On Saturdays the library opens at 10 am. It is closed on Sundays.",
    "Books can be borrowed for three weeks and renewed online.",
]
question = "When does the library open on Saturdays?"


def words(text):
    return {w.strip(".,?!").lower() for w in text.split()}


# Retrieve: score each chunk by the question words it contains
scores = [len(words(question) & words(c)) for c in chunks]
ranked = sorted(range(len(chunks)), key=lambda i: -scores[i])
top = ranked[:2]
print("scores:", scores, "-> using chunks", [i + 1 for i in top])

# Augment: put the retrieved text into the prompt
prompt = "Answer using only the sources below. Cite them as [1], [2]. If they do not contain the answer, say you do not know.\n"
for n, i in enumerate(top, 1):
    prompt += f"[{n}] {chunks[i]}\n"
prompt += f"Question: {question}"
print(prompt)
Open in compiler

Line by line

  1. Three chunks stand in for an indexed collection of documents.
  2. words lower-cases a text and strips punctuation, so "library." matches "library".
  3. Each chunk is scored by how many question words it contains, and the top two are kept. A real system would use semantic or hybrid search here.
  4. The prompt starts with grounding instructions, then lists the retrieved chunks as numbered sources.
  5. The finished prompt is what would be sent to the language model; the Saturday hours are in source [1], so a good model answers "10 am [1]".

Expected result

scores: [3, 4, 0] -> using chunks [2, 1]
Answer using only the sources below. Cite them as [1], [2]. If they do not contain the answer, say you do not know.
[1] On Saturdays the library opens at 10 am. It is closed on Sundays.
[2] The library opens at 9 am and closes at 6 pm on weekdays.
Question: When does the library open on Saturdays?

In the real world

Customer-support bots answer from help-centre articles, internal assistants answer from company wikis and policies, and search engines show AI summaries with links to the pages they used: all RAG. Companies prefer it to retraining a model because updating a document collection is instant and cheap, private data stays in their own systems, and citations make answers auditable.

Common mistakes

  • Not telling the model to admit when the sources lack the answer, which invites guessing.
  • Retrieving too many chunks, which buries the relevant one.
  • Never testing retrieval on its own, so failures are blamed on the model.
  • Dropping source numbers, which makes answers impossible to check.

Best practices

  • Test retrieval with real questions whose answers you know, before tuning prompts.
  • Keep chunks small and focused, and store their sources and dates.
  • Ask for citations and spot-check that each cited source really supports the claim.

Summary

Retrieval-augmented generation answers questions in three steps: retrieve relevant chunks, augment the prompt with them as numbered sources plus grounding instructions, and generate an answer that cites them. It gives models current and private knowledge without retraining and makes answers checkable, but its answers can only be as good as what retrieval finds.

Quick check

1. What problem does RAG mainly solve?
2. What should a grounded prompt tell the model to do when the sources lack the answer?
3. The answer is wrong because the right chunk was never retrieved. Which step failed?
4. Why number the sources in the prompt?

Key takeaways

  • RAG retrieves relevant chunks, adds them to the prompt and asks the model to answer only from them.
  • It gives models up-to-date and private knowledge without retraining, and makes answers checkable.
  • Retrieval quality limits answer quality: if the right chunk is not retrieved, the model cannot use it.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.