Definition
Retrieval-augmented generation (RAG) is a technique in which an AI system retrieves relevant passages from documents or a search index before it answers and hands them to a large language model along with the question. The model writes the answer from that material and doesn’t learn anything new in the process.
Key takeaways
- RAG means an AI system retrieves relevant passages before it answers and hands them to a language model. The model learns nothing new.
- Google also calls RAG grounding and uses it to retrieve pages from its search index. In a company, the source is a collection you maintain.
- RAG can reduce made-up information, but it doesn’t prevent it. Check important answers against the source they cite.
- According to Anthropic, up to about 500 pages of material fit in a single prompt. For small collections, RAG is often unnecessary.
- Germany’s data protection authorities require a legal basis for personal data in RAG systems. Their guidance dates from October 2025.
How does RAG work?
RAG has two phases. Ahead of time, the system splits documents into short chunks of text and stores them in searchable form. Then, for each question, it retrieves the most relevant chunks and gives them to the language model along with the question. The model writes the answer from them.
Germany’s federal and state data protection authorities, which work together as the German Data Protection Conference (DSK), describe the process in their guidance on RAG from October 2025. Vendor documentation describes it in much the same way:
- Split: The system divides the documents into short passages called chunks. They’re usually no more than a few hundred tokens long (Anthropic, Sep 19, 2024). Tokens are the words or word fragments a language model works with. For fixed-size chunks, Microsoft recommends starting with 512 tokens, or about 2,000 characters, with a 25% overlap (Microsoft Learn, chunking documents, Jul 2, 2026).
- Convert and store: An embedding model turns each chunk into a list of numbers, called a vector, that represents its meaning. The vectors go into a vector database.
- Retrieve: When someone asks a question, the same model converts it into a vector. A search component called a retriever then fetches the chunks with the most similar vectors. A keyword search such as BM25 is often added as well, because it finds exact matches like an error code (Anthropic, Sep 19, 2024).
- Answer: The most relevant chunks are added to the question, and the model writes the answer. Depending on the system, it also shows the sources it used.
The language model itself doesn’t change. The expanded prompt is built from scratch for every question (German Data Protection Conference, October 2025). So if you correct a document and store it again, the next answer can change. Our guide to generative engine optimization (GEO) compares how AI assistants choose their sources on the web.
Where does RAG come from?
The term comes from a 2020 paper by Patrick Lewis and colleagues. They used it for language models that combine what they learned in training with a searchable index of text. In German sources you’ll also come across the word Fundierung, which Google’s German documentation uses as another name for RAG.
The paper connected a language model to a text index built from Wikipedia. The authors report that their models produced more factual text than a comparison model without retrieval (Lewis et al., 2020).
Germany’s data protection authorities simply write RAG. Google’s German documentation adds Fundierung, the German word for grounding (Google Search Central, in German, as of Jul 15, 2026).
Does RAG prevent wrong answers?
No. RAG can reduce made-up information, but it can’t rule out wrong answers. Retrieval can miss the right chunk, the model can disregard what it retrieved, and the documents themselves can be out of date. So check important answers against the source they cite.
The German Data Protection Conference says RAG can reduce hallucinations, meaning made-up information that sounds plausible. How reliable the answers are depends heavily on how good, current and complete the documents are (German Data Protection Conference, October 2025).
A 2024 study from Stanford and Yale tested AI legal research tools whose vendors had promoted RAG as a way to avoid hallucinations. The tools did better than GPT-4 alone but still hallucinated 17% to 33% of the time (Magesh et al., May 30, 2024).
Retrieval itself also misses things. In an Anthropic test, 5.7% of the relevant documents weren’t among the top 20 chunks in the baseline setup. With added context for each chunk, keyword search and reranking, that fell to 1.9% (Anthropic, Sep 19, 2024). Those figures come from the vendor’s own tests on datasets it chose.
What does RAG mean for your website?
When AI systems search the web, your website can be one of the sources they retrieve from. On Google, a page has to be indexed and eligible to be shown with a snippet. Google doesn’t need your content chopped into small pieces. Clear sections that open with the answer help your readers and, in our view, retrieval too.
Google describes RAG as a technique behind its AI features. Its core Search ranking systems retrieve relevant, up-to-date pages from the index, and its systems then review the information on those pages (Google Search Central, as of Jul 10, 2026). A page can only appear there if it’s indexed and eligible to be shown in Search with a snippet. Our article AI crawlers and robots.txt explains which AI providers’ crawlers need access. To check whether your text is in the HTML without JavaScript, follow the step Are your services visible in the HTML? in our ChatGPT how-to.
Google says there’s no need to break content into tiny pieces, because its systems can understand multiple topics on a page (Google Search Central, as of Jul 10, 2026). Other systems split text themselves before retrieval. Microsoft names headings in HTML or Markdown as one way to split text at section boundaries (Microsoft Learn, chunking documents, Jul 2, 2026). The German Data Protection Conference advises that each chunk should hold one complete unit of meaning where possible.
In our view, that means two things for your content: spell out who or what each sentence is about, and say what every number refers to. Anthropic illustrates the problem with a chunk that says only that revenue grew 3%. With no company or time period, the chunk on its own isn’t much use (Anthropic, Sep 19, 2024). If a section opens with the answer, that answer still holds when only that section gets passed along.
When is RAG worth it for a small business?
RAG is worth it when recurring questions depend on documents that are too long for a single prompt or that change often. Small collections often fit into a prompt in full. In that case, it’s enough to give the documents directly to an AI assistant, and RAG would only add work.
RAG suits material such as handbooks, work instructions and proposal templates, where someone looks up the answer by hand today. Germany’s data protection authorities name internal company chatbots with access to current business data as a typical example (German Data Protection Conference, press release, Oct 17, 2025).
Anthropic offers a rule of thumb for “small”: under 200,000 tokens, or about 500 pages, the whole collection can go into the prompt (Anthropic, Sep 19, 2024). The limit depends on the language model. If your documents contain personal data, see the section What rules apply to RAG in Germany and the EU?
The numbers in the “Source” column refer to the list of sources at the end of this article.
| Your situation | Our assessment | Source |
|---|---|---|
| Up to about 500 pages that rarely change | You usually don’t need RAG. Give your AI assistant the documents directly. | 2 |
| More material than fits in one prompt | RAG is the usual approach. | 2 |
| Content such as price lists or instructions changes often | RAG fits, because documents are easy to replace. | 1 |
| Answers must be traceable to a source | RAG fits if the system shows the sources it used. | 1 |
| Documents are out of date, duplicated or contradictory | Clean them up first. RAG passes on wrong content. | 1 |
| The answer is in a database (inventory, unpaid invoices) | A targeted query of that database is a better fit. | our own assessment |
| Not everyone is allowed to see everything | Use RAG only with defined roles and access rights. | 1 |
Only add documents from sources you trust. The security organization OWASP describes a resume with white text on a white background that gives instructions to the screening system (OWASP, LLM08:2025).
How do RAG, grounding, query fan-out and fine-tuning differ?
RAG is the technique: retrieve, pass along, answer. Grounding means tying an answer to sources, and in search engines it often refers to that same retrieval from the web index. Query fan-out generates the searches for that retrieval. Fine-tuning changes the language model itself.
The numbers in the “Source” column refer to the list of sources at the end of this article.
| Term | What happens | Changes the model? | Source |
|---|---|---|---|
| RAG (retrieval-augmented generation) | The system retrieves chunks of text from a collection or an index and gives them to the model along with the question. | no | 1, 2 |
| Grounding | Answers are connected to web content or to your own content. Google equates the term with RAG. Microsoft calls RAG a pattern for grounding responses in your own content. | no | 6, 7, 8 |
| Query fan-out | The model generates several related searches so that retrieval finds additional results. | no | 6 |
| Fine-tuning | The model is retrained on additional data, and its parameters change. | yes | 1 |
Our article What is AI grounding? explains what search systems do with your website and what you can control. What is query fan-out? shows how one question turns into several searches.
What rules apply to RAG in Germany and the EU?
If your documents contain personal data, using RAG requires a legal basis, according to guidance from Germany’s data protection authorities. The authorities see advantages for correcting and deleting data, because the data sits in the documents and the database. RAG doesn’t solve the problems of a model that was trained unlawfully.
The guidance measures RAG against the principles of the General Data Protection Regulation (GDPR) and takes into account Opinion 28/2024 of the European Data Protection Board. These points matter most for businesses (German Data Protection Conference, October 2025):
- Correcting and deleting: Information in documents and the vector database can be updated and deleted. In the language model itself, that remains difficult.
- Access: Roles and separate data storage control who can retrieve which chunks. Our explainer on multi-tenant SaaS architecture shows how this works in software that serves several customers.
- Where it runs: RAG can make smaller models practical, including ones you run on your own premises. Personal data then doesn’t have to go to the online operators of large language models.
- Web search: If you connect external sources, you have to check that their use is lawful and that the data is accurate. According to the DSK, a connected web search regularly fails to meet these requirements.
- The model: A model that was trained unlawfully stays unlawfully trained, even inside a RAG system. RAG can reduce some of the risks for the people affected, but that can only be assessed case by case.
This is a technical assessment. If in doubt, a lawyer can tell you what applies in your case.
From our work: what we calculated on one of our own pages
The top three results gave a language model only 20% to 51% of the 2,984 words in the main text of our article llms.txt: what it does and what it doesn’t. That comes from a sample calculation we ran on November 9, 2026.
We split the text three ways. For seven customer questions we chose ourselves, we used the keyword search BM25 to find the best-matching chunks. Each question has one section that answers it. We counted how often that section’s answer passage appeared in full in the results.
| Chunking | All in result 1 | All in results 1–3 | Words in 1–3 |
|---|---|---|---|
| by section, with heading (8 chunks) | 5 of 7 | 7 of 7 | 756–1,534 |
| fixed chunks of 200 words, no overlap (15 chunks) | 5 of 7 | 6 of 7 | 584–600 |
| fixed chunks of 200 words, 30-word overlap (18 chunks) | 5 of 7 | 5 of 7 | 600 |
Measured · Python 3.12.10, our own script, BM25 (k1 1.2; b 0.75) without stemming
Three things stand out:
- The model gets an excerpt. The rest of the page never reaches it.
- Where the text is cut matters. Whole sections put the right answer in the top three for all seven questions. Fixed chunks did for six without overlap and for five with a 30-word overlap.
- Keyword search on its own only matches identical words. For the question “Is an llms.txt file worth it for my business website?”, the section that answers it came third. It asks whether you should create the file and never uses the word “worth.” That’s why many systems also use embeddings.
We follow a fixed writing rule: every question section in our Insights articles opens with a passage that answers the heading on its own. In the article on llms.txt, that passage runs 39 to 53 words. So if a section is retrieved as a whole, the answer sits in its first few lines.
The calculation is one example, on one page, with questions we picked ourselves. It doesn’t use embeddings. Sections and fixed chunks tied on the first result, and sections did better in the top three. The figures apply only to this English text: On the German version of the article, fixed chunks with overlap came out ahead on the first result. We don’t know how Google, Bing or ChatGPT split our pages. The calculation doesn’t show any effect on mentions in AI answers.
To check this on your own site:
- Copy one section of your most important page into a blank document.
- Read only the first two sentences. Do they answer the heading?
- Check whether your company name or what you offer appears in the section itself.