Definition
AI grounding is the step in which an AI system bases its answer on sources it retrieves while answering. These are usually current web pages from a search index. The model draws on those pages and can cite them as links. Grounding lowers the risk of made-up details, but errors can still occur.
Key takeaways
- Grounding means an AI system retrieves sources while answering and relies on them. They’re usually current web pages from a search index.
- RAG is one method for achieving grounding. Google Search treats the two terms as the same. Google Cloud lists RAG as one of several paths.
- From February 1, 2027, Microsoft will honor a per-page grounding preference. Without one, grounding is allowed. Bing search is unaffected.
- If you rely on NoArchive or NoCache, act by March 1, 2027. After that, Microsoft stops reading both for training and grounding.
- A source in an AI answer doesn’t prove the answer. OpenAI itself warns that search results and citations can be incorrect.
What does grounding mean in AI?
Grounding means tying an AI answer to content the system retrieves while answering. Google Cloud calls it the ability to connect model output to verifiable sources. Microsoft describes it as using content as input for AI-generated responses. According to Microsoft, indexing and ranking in traditional search don’t count as grounding.
Google and Microsoft both use the word grounding in their documentation, and Microsoft has named its new preference after it. Not every provider does. In their help pages, OpenAI, Anthropic and Perplexity write about web search or crawlers instead. Crawlers are programs that fetch web pages. All three describe the same process: the system draws on third-party pages while answering and, at best, cites them as sources.
Our guide to GEO compares how ChatGPT, Perplexity, Gemini and Google choose their sources.
How does grounding work?
The model first decides whether a web search would improve the answer. If it would, the model generates one or more search queries, processes the results and writes the answer, often with links to its sources. If a question doesn’t need current information, the model usually answers from its training.
Google describes the process for its Gemini API in five steps (Google AI Studio (external site)):
- An application sends the question to the model.
- The model checks whether a Google search can improve the answer.
- If so, it writes one or more search queries itself and runs them.
- It processes the results and writes the answer.
- The API returns the answer with citations that link parts of the text to their sources.
Whether the system searches at all depends on the question. According to Anthropic, Claude searches when a request depends on current or changing information and answers without searching when the request draws on stable knowledge (Anthropic, web search tool (external site)). A single question can lead to several searches. This technique is called query fan-out.
For a page to qualify as a source, the system has to be able to retrieve it. At Google, the page must be indexed and eligible to appear in Search with a snippet (Google Search Central, as of Dec 10, 2025 (external site)).
How is grounding different from RAG?
Grounding is the step of tying an answer to verifiable sources. RAG (retrieval-augmented generation) is a specific method for doing it: the system retrieves relevant content from a data store before answering and uses it in the response. For Search, Google treats the two terms as the same. Google Cloud lists RAG as one of several paths.
A 2020 research paper describes RAG as connecting a pre-trained language model to a retrievable index of Wikipedia (Lewis et al., May 22, 2020 (external site)). For AI features in Google Search, Google calls RAG “also known as grounding” (Google Search Central, as of Jul 10, 2026 (external site)). Google Cloud’s documentation, by contrast, lists RAG alongside other paths such as Google Search and Google Maps (Google Cloud, as of Nov 7, 2026 (external site)).
For site owners, the distinction matters little: both words mean that a system retrieves your content while answering. Building your own AI application that answers from your documents is a separate topic, and that’s where Google Cloud explicitly points to RAG. Our article What is RAG? explains how the technique works and when it’s worth it for your own documents.
| Term | What it means | When it happens | What you can control | Sources |
|---|---|---|---|---|
| Grounding | An AI answer is tied to retrieved sources. | when the answer is generated | varies by system | 1, 2 |
| RAG (retrieval-augmented generation) | A method: the system retrieves content from a data store before answering. | when the answer is generated | up to the provider; no dedicated setting for site owners | 1, 5 |
| Query fan-out | The system generates several searches for parts of a question. | before pages are retrieved | no known dedicated setting | 3 |
| Training | Content changes a model’s learned parameters. | in advance, when the model is built | robots.txt; at Microsoft, also per page | 2, 9 |
| Indexing and ranking | Pages are stored and sorted in traditional search. | before the search | robots.txt and noindex; grounding preferences don’t change them | 2, 7 |
How can you control grounding in the major AI systems?
There’s no single rule. Google offers a Search Console control and snippet rules. Microsoft is introducing its own per-page preference. OpenAI, Anthropic and Perplexity handle web search through crawler names in robots.txt. At some of these companies, fetches made on behalf of individual users follow different rules.
In the robots.txt file, you specify which crawlers may fetch which parts of your website.
| System | How you control it | What blocking does | Sources |
|---|---|---|---|
| Google Search: AI Overviews, AI Mode | The “Search generative AI” control in Search Console, per property; the default is to include your site | Links and content no longer appear there or serve as input; Google says this isn’t a signal for other parts of Search | 6, 7 |
| Gemini app, Grounding with Google Search on Vertex AI | Google-Extended in robots.txt, a single token for both training and grounding | No grounding in these products; Google Search isn’t affected | 8 |
| Microsoft: Bing, Copilot, Microsoft 365 | Starting Feb 1, 2027: ai-grounding per page, in a meta tag or the X-Robots-Tag header; with no preference, Allow applies | disallow: no grounding; disallowextended: Microsoft’s own products only; Microsoft warns of significantly reduced visibility in AI answers; Bing search isn’t affected | 2 |
| ChatGPT search | OAI-SearchBot in robots.txt; ChatGPT-User for fetches on behalf of users | A site that blocks OAI-SearchBot isn’t shown in search answers, though it can still appear as a navigational link; robots.txt rules may not apply to ChatGPT-User | 9, 10 |
| Claude | Claude-SearchBot and Claude-User in robots.txt | Content isn’t indexed for search or isn’t retrieved; Anthropic says visibility may drop | 11 |
| Perplexity | PerplexityBot in robots.txt; Perplexity-User for fetches on behalf of users | Perplexity recommends allowing PerplexityBot so a site can appear in results; it says Perplexity-User generally ignores robots.txt | 13 |
The numbers in the “Sources” column refer to the list of sources at the end of this article. At Google, robots.txt rules for Googlebot and snippet rules such as nosnippet or max-snippet also apply, and they cover all of Google Search (Google Search Central, as of Dec 10, 2025 (external site)).
Training and grounding are separate questions. Blocking OpenAI’s training crawler GPTBot doesn’t block OAI-SearchBot. Our article on AI crawlers lists each one under Which AI crawlers are there, and who runs them?
What is Microsoft changing on February 1, 2027?
Starting February 1, 2027, Microsoft will honor a separate per-page grounding preference, set in a meta tag or the X-Robots-Tag HTTP header. With no preference, Allow applies. After March 1, 2027, Microsoft will no longer read NoArchive and NoCache as preferences for training or grounding. Eligibility for traditional Bing search doesn’t change.
Microsoft announced the new preferences on October 30, 2026 (Microsoft Bing, Oct 30, 2026 (external site)). A meta tag sits in the head of an HTML page. An HTTP header is a piece of information the server sends with each response. The grounding preference has three values:
- Allow: Content can inform AI answers. This value applies if you set nothing.
- DisallowExtended: Content can inform answers only in Microsoft’s own products, such as Copilot and Microsoft 365. Services Microsoft provides to third parties are excluded.
- Disallow: Content no longer informs AI answers.
<meta name="Bingbot" content="ai-grounding:disallow">
X-Robots-Tag: Bingbot: ai-grounding:disallowMicrosoft points out three limits:
- The preference applies to a single page. There’s no site-wide rule in robots.txt.
- The preferences govern how content is used. Crawling is still governed by robots.txt, so Allow doesn’t override a block on Bingbot.
- A preference takes effect only once Microsoft recrawls the page. IndexNow reports changed pages and makes a recrawl more likely, but it doesn’t guarantee a date.
According to Microsoft, opting out significantly reduces your visibility in AI answers. And none of the settings guarantees a mention, traffic or payment.
For sites that use NoArchive or NoCache, Microsoft has laid out a transition:
| Date | What happens |
|---|---|
| Dec 15, 2026 | Bing Webmaster Tools starts accepting temporary grounding preferences, only from sites that use NoArchive or NoCache on that date. |
| Feb 1, 2027 | Microsoft starts honoring the new preferences as it crawls pages. |
| Mar 1, 2027 | Deadline if you want to keep excluding content from grounding: publish the new preference on your pages and record a temporary preference in Bing Webmaster Tools. After that, Microsoft no longer reads NoArchive and NoCache as preferences for training or grounding. |
| Sep 30, 2027 | Temporary preferences expire on this date at the latest, or sooner if Microsoft recrawls the page and finds the new preference. |
If you haven’t recorded a temporary preference by March 1, 2027, pages with NoArchive or NoCache default to Allow for grounding until Microsoft recrawls them and finds the new preference.
Does grounding make AI answers reliable?
Grounding lowers the risk of made-up details, but answers with sources can still be wrong. OpenAI says search results and citations can be incomplete, outdated or incorrect. When public service media reviewed answers to news questions, they found serious sourcing problems in 31% of them.
Google Cloud is careful about the benefit: it says grounding reduces the chances of inventing content (Google Cloud, as of Nov 7, 2026). OpenAI advises opening cited sources to check that they support the answer (OpenAI, web search help (external site)). A citation shows which page the system worked from. It doesn’t show whether the answer reflects that page correctly.
In a study led by the BBC, 22 public service media organizations in 18 countries reviewed more than 3,000 answers from ChatGPT, Copilot, Gemini and Perplexity. They found at least one significant issue in 45% of the answers and serious sourcing problems in 31%.
Source: BBC (external site), · more than 3,000 answers to news questions, 18 countries
You can’t apply these rates to answers about businesses. So test a sample of answers about your own offering and open the sources they cite. Bing Webmaster Tools shows a sample of the phrases Microsoft’s AI used to retrieve your pages, under the name grounding queries. The section What do the Google and Bing reports show? explains how to read them.
What does grounding mean for a small business?
Our recommendation for most small businesses: allow grounding and keep your pages reachable and current. AI systems read your page while answering. If it contains outdated information, that information can end up in the answer. Blocking only makes sense if you don’t want specific content to appear in AI answers.
- Stay reachable: Check whether your robots.txt, your CDN (the delivery network in front of your server) or your firewall locks out crawlers. The section What happens when your CDN or firewall blocks AI crawlers? explains how.
- Be readable in the HTML: If your services and prices only appear after JavaScript runs, many crawlers can’t see them. Our how-to covers this check under Are your services visible in the HTML, without JavaScript?
- Keep it current: Review your services, prices and contact people regularly. Grounding works with whatever the system finds when it retrieves the page.
- Check for NoArchive and NoCache: Look in your page source and HTTP headers, even if you never set them yourself. The commands are under From our work.
- Block selectively: Selling or licensing your content might be a reason. Starting February 1, 2027, Microsoft lets you block grounding page by page.
What’s specific to Germany and the EU?
Google’s Search Console control has been available to all websites worldwide since August 31, 2026. Microsoft’s announcement does not exclude any country. If your company uses Copilot with web search, Microsoft says neither its Data Protection Addendum (DPA) nor the EU Data Boundary applies to the search queries Copilot generates.
The EU Data Boundary is Microsoft’s commitment to store and process customer data from its enterprise services within a defined geographic area (Microsoft, as of Jul 22, 2026 (external site)). That commitment does not cover search queries sent to Bing. Microsoft says it acts as a data controller for those queries. The user’s IP address is sent along with them. Microsoft first strips out identifying information based on the user’s Microsoft account (Entra ID). Admins can turn off web search for Copilot (Microsoft, as of Nov 3, 2026 (external site)). Ask your data protection advisor whether your company can use web search in Copilot this way.
According to Microsoft, the preferences do not establish any licensing or payment arrangements. For what a rights reservation means for training under German law, see What does a rights reservation mean for AI training? Our article on AI Overviews covers when Google’s AI features launched in Germany.
This is a technical assessment. If in doubt, a lawyer can tell you what applies in your case.
From our work: what grounding preferences does datafalke.de set?
For this article, we fetched all 98 URLs in our XML sitemap on November 9, 2026, and analyzed the responses.
| Check | Result |
|---|---|
| HTTP status | 98 of 98 URLs return 200 |
| X-Robots-Tag HTTP header | not set on any URL |
| Robots meta tag | the same on all 98 URLs: index, follow, with no noarchive or nocache |
| ai-grounding preference | not set on any URL |
| robots.txt | all crawlers allowed; no line for __usage_ai-training |
As a result, starting February 1, 2027, Microsoft’s default of Allow will apply to our pages for both training and grounding. The transition steps for NoArchive and NoCache don’t affect us. We deliberately allow grounding because our content is public and we want AI answers to mention us.
To check one of your own pages, run these commands in a terminal. Replace the sample URL with your own:
curl -sIL https://example.com/ | grep -i "x-robots-tag"
curl -sL https://example.com/ | grep -i -o -E '<meta[^>]+(noarchive|nocache|ai-grounding)[^>]*>'If neither command prints anything, the page has no such preference. Each command checks only that one URL.
We can’t yet show whether Microsoft or other systems actually use our pages in their answers. We’ll publish our own figures only with our measurement log.