A model knows a great deal in general and nothing whatsoever about you. It has never seen your documents, your policies or last Tuesday’s decision. Asked about any of them, it will answer anyway, from general knowledge, with complete confidence.
RAG — retrieval-augmented generation — is the fix, and it is less clever than the acronym suggests. It is the difference between a closed-book exam and an open-book one. Before the question is asked, go and find the pages that are likely to contain the answer. Put them in front of the model. Then ask.
Three steps: find, hand over, ask. That is the whole idea.
In short
- What
- Searching your own material for the passages relevant to a question, putting those passages in front of the model, and asking it to answer from them.
- Why
- It lets a general model answer about your specific, changing world — and show where each answer came from.
- When
- When the answer depends on documents you hold, or on facts that change faster than anyone could retrain a model.
- How
- Spend your effort on the retrieval. Pass a few good passages rather than many mediocre ones, require citations, and allow “this is not in the documents” as an answer.
Why not just train the model on your data
Because documents change and training does not. A policy updated this morning is live in a retrieval system the moment it is indexed; getting it into a model’s weights is a far slower and more expensive undertaking, and has to be repeated every time anything moves.
Because retrieval can show its sources. The passages that produced an answer are right there and can be shown to the person reading it. Something absorbed into weights during training leaves no trail at all, which makes a wrong answer nearly impossible to explain.
And because retrieval can be taken away again. A document that should no longer be used is removed from the index and stops influencing answers immediately. Training has no equivalent of deleting a row.
When you reach for it
When the answer lives in your own material — documentation, policies, listings, past tickets, anything specific to you.
When the facts move. Prices, availability, rules, status: anything where an answer that was right last month is wrong today.
When someone has to be able to check. Support answers, anything involving money or obligations, anything a person will act on — all of them are better with a passage attached than with a confident paragraph and no provenance.
And when not: if the question is general knowledge, retrieval adds cost and latency and nothing else. If the relevant document is short and always the same one, just send it every time. RAG is for when you have too much material to send it all.
The hard part is the finding
Almost everyone building their first RAG system expects the difficulty to be in the generation. It is not. If the right passage reaches the model, a decent answer usually follows. If it does not, no amount of prompting rescues the situation — the model will answer from general knowledge and sound exactly as sure of itself.
So the engineering is in retrieval. Split documents at natural boundaries, so each piece makes sense alone, and keep the document title and heading attached to each piece — a paragraph that says “this is not permitted” is dangerous without the section it belongs to.
Search both ways. Meaning-based search finds the passage that answers a differently-worded question; keyword search finds the exact product name, error code or street that meaning-based search glides past. Running both and combining them beats either alone, reliably enough that it is worth doing by default.
Then narrow. Take more candidates than you need, re-rank them against the actual question, and pass only the best few. A handful of strong passages produces better answers than twenty mediocre ones, which mostly bury the good one.
How to keep it honest
Tell the model to answer from the passages and to say plainly when they do not contain the answer. A system that is allowed to say “I cannot find this in the documents” is far more useful than one that must always produce something.
Ask it to cite. Requiring each claim to point at the passage it came from makes wrong answers visible instead of plausible, and it gives the reader somewhere to go.
Keep the index fresh. The quietest failure in RAG is an index that stopped updating: every answer stays fluent, well-cited and slowly becomes wrong.
Measure retrieval on its own. Write down real questions and the passages that ought to be found, and check how often they are. It is the single most useful number in the system, and it is invisible if you only ever look at final answers.
Where it stops
RAG gets the right material in front of the model. It does not make the model read carefully, check its own answer, or stop when it is unsure — and it cannot do anything beyond answering. Looking something up in a live system, making a booking, changing a record: those need tools, and tools need a setup that decides what is allowed.
Retrieval is how a model learns about your world. It is not how it acts in it.
