JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Why Retrieval-Augmented Generation?

A model only knows what it memorised during training. RAG lets it look things up first.

The closed-book problem

A plain language model answers from memory. Everything it can say was baked into its weights during training, and nothing newer ever gets in. That fixed horizon has a name — the knowledge cutoff — and it means the model has never heard of yesterday's news, your company's internal wiki, or the contract you signed this morning. Ask anyway and it will often answer confidently regardless, inventing plausible-sounding details. That confident fabrication is hallucination.

A closed-book model generates each next token only from what its weights memorised during training.

An autoregressive loop feeding each generated token back in as input for the next one.

You can't fix this by making the model bigger. The missing facts simply weren't in the training data — and even if they were, you can't retrain a giant model every time a document changes. We need a way to feed the right information into the model at question time, fresh and specific to the task.

The open-book idea

Retrieval-Augmented Generation (RAG) turns a closed-book exam into an open book. Before the model writes a word, a search step pulls the most relevant passages from a trusted collection of documents and pastes them into the prompt. The model then answers using those passages instead of guessing from memory. Anchoring the answer in supplied evidence is called grounding.

RAG turns a closed-book exam into an open book: retrieve the most relevant passages, then generate from them.

Pipeline: the query goes to a retriever over a document store, top passages are pasted into the prompt, then the model generates the answer.

Why this works so well: the model is brilliant at reading and rephrasing text it is given, even if it never saw that text during training. RAG plays to that strength. It separates knowing (stored in an external, easily-updated collection) from reasoning and wording (the model's job). Update a document, and the next answer reflects it — no retraining required.

The three steps

Almost every RAG system, however fancy, is built from the same three moves:

  1. Retrieve — take the user's question, search a collection of documents, and return the few passages most likely to contain the answer. The collection usually lives in a vector database that supports fast semantic search.
  2. Augment — paste those passages into the prompt alongside the question and an instruction like answer using only the context below.
  3. Generate — let the model write the answer, now grounded in the retrieved evidence rather than its fuzzy memory.

When to reach for RAG

RAG shines when the knowledge is large, changes often, or is private — manuals, policies, support tickets, research libraries, a codebase. A tempting alternative is to dump the whole corpus into the prompt and rely on a long context window, but that gets slow and expensive fast, and quality sags when the window is stuffed with mostly-irrelevant text. Retrieval keeps the prompt small and on-topic.

The other alternative, fine-tuning, bakes knowledge back into the weights — great for teaching style or format, but a poor fit for facts that must stay current or be cited. A good rule of thumb: fine-tune for behaviour, retrieve for knowledge. We'll spend the rest of this track building each piece — the knowledge base, the retriever, the grounded prompt, and the evaluation that tells you it actually works.