What is RAG (retrieval-augmented generation)?
RAG, for retrieval-augmented generation, is a method that retrieves the relevant passages of your own documents at the moment a question is asked, then hands them to the language model so that it answers from those passages rather than from its training memory alone.
Definition
RAG — retrieval-augmented generation — is a method that retrieves the relevant passages of existing documents at the moment a question is asked, then hands them to the language model so that it composes its answer from those passages.
The important point fits in one sentence: your documents are not learned by the model. They are consulted on demand, for the duration of one answer, the way a lawyer consults a file before replying rather than relying on memory.
The problem it solves
A language model only knows what it saw during training. It knows nothing of your contracts, your internal procedures, your rates or your client history. Asked anyway, it produces a likely-sounding answer — at best generic, at worst wrong but credible.
RAG corrects this by grounding the answer in real sources, which also makes it possible to cite the original document. That traceability is often worth more than the answer itself: it makes verification possible.
The two stages of RAG
1. Indexing, once
- Collect the documents: PDFs, word processor files, internal pages, archived emails.
- Extract the text, using OCR if the documents are scanned.
- Split into fragments of a reasonable size. Too fine a split loses context; too coarse a one drowns the useful information.
- Compute embeddings: each fragment is converted into a numerical representation that situates its meaning.
- Store in a vector database, which can quickly retrieve the fragments closest to a question.
2. Retrieval, at every question
- The question is converted into an embedding by the same process.
- The database returns the closest fragments.
- Those fragments are placed in the model's context, together with the instruction to answer from them.
- The model writes, and can indicate which document each element came from.
One detail changes a great deal: search by closeness of meaning finds a passage about "early termination" even when the question asks about "ending the contract before term". That is what separates RAG from keyword search.
A law and notary practice accumulates deeds, correspondence and rulings over several decades. Finding how a comparable case was handled means knowing it exists, and remembering which file it was in.
With a RAG index, a question in plain language — "have we already dealt with a contested right of way after a land division?" — brings back the relevant passages and the file references. The lawyer keeps the analysis; what is gained is no longer depending on memory to know where to look.
RAG or fine-tuning?
The two approaches are often opposed wrongly: they answer different questions.
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model knows at answer time | How the model answers |
| Updating | Immediate: reindex the document | Requires new training |
| Traceability | The source can be cited | No identifiable source |
| Initial cost | Moderate | High |
| Good use | Company-specific knowledge that evolves | Tone, format, constant domain vocabulary |
For the vast majority of business needs — answering from internal documents — RAG is the right answer. Fine-tuning becomes relevant when the problem is not knowledge but form.
Limits and risks
- Quality depends entirely on retrieval. If the right passages do not surface, the model answers beside the point with the same confidence as if it were right.
- Outdated documents. An index holding both the old and the new version of a procedure will produce contradictory answers. Document governance comes before RAG.
- Tables and complex layouts resist splitting: a table cut in half loses its meaning.
- Access rights. An index that ignores who may see what can hand an employee content that was not meant for them. Permissions have to be carried through into retrieval.
- Global questions. "How many contracts expire this quarter?" is a database question, not a RAG one. Similarity search does not count.
When RAG is the right fit
Yes when the useful information already exists as text, is scattered, changes over time, and the questions asked are varied and freely worded.
No when the information already sits in a structured database — a query is more reliable and instant — or when the corpus is small enough to be read directly.
Frequently asked questions
Does RAG train the model on my documents?
No. Your documents do not modify the model: they are retrieved and placed in the context of the question, for the duration of one answer.
Does RAG eliminate wrong answers?
It reduces them by anchoring the answer in real sources, and it makes it possible to cite the original document. It does not eliminate them: a badly retrieved passage gives a badly grounded answer.
How many documents does it take to be useful?
RAG becomes interesting as soon as the volume exceeds what a person can read on demand, or as soon as the information is scattered across several sources.