Retrieval Augmented Generation

A language model knows only what was in its training data, frozen at the moment training stopped. Ask it about your internal handbook, last week's incident report, or a customer's order history and it has two options: admit ignorance, or invent something plausible. Retrieval Augmented Generation (RAG) removes the dilemma by fetching the relevant text first and handing it to the model as part of the prompt.

What the acronym actually names

• Retrieval — a search step that finds passages likely to contain the answer. Usually semantic search over embeddings, often blended with old-fashioned keyword search.
• Augmented — those passages are inserted into the prompt, typically above the user's question, with an instruction to answer from them.
• Generation — the model writes the answer. Its job shifts from *recall* to *reading comprehension*, which it is far better at.

Nothing about the model changes. No training, no weights touched, no GPUs rented. RAG is a system built *around* a model, which is exactly why it became the default way to ship LLM features on private data: you can add a document at 10am and the assistant knows about it at 10:01.

Where it goes wrong

RAG is not a hallucination cure — it is a hallucination *reducer* whose ceiling is set by the retrieval step. If search returns the wrong three paragraphs, the model will confidently answer from the wrong three paragraphs. In practice, the overwhelming majority of bad RAG answers are retrieval failures wearing a generation costume, which is why the rest of this star spends most of its time on the search half.

• The answer exists in your corpus but search never surfaced it (a recall failure).
• Search surfaced it, but buried it beneath nine irrelevant passages the model trusted more.
• The answer is spread across five documents and no single passage contains it.
• The corpus contains two contradictory documents and nothing marks which is current.