AI crawlers and robots.txt: what to allow

Enter the user agent after User-agent: in robots.txt; capitalization doesn’t matter. The “Source” column gives the number in the list of sources at the end of this article. Our guide to generative engine optimization (GEO) explains how these crawlers relate to AI answers.

The details come from each operator’s documentation:

• OpenAI (external site) and Anthropic (external site)
• Google on its common crawlers (external site) and its user-triggered fetchers (external site)
• Microsoft Bing (external site), Perplexity (external site) and Apple (external site)
• Meta (external site), Amazon (external site) and Mistral AI (external site)
• DuckDuckGo (external site) and Common Crawl (external site)

The list only includes crawlers whose operators publicly describe their user agent and purpose. Crawlers for ads and link previews, such as OAI-AdsBot, are left out. Each row shows the date we last checked it against the documentation. Changes are recorded in the change log at the end of this article.

The list is also available as a CSV file with the same user agents as the table. Only __usage_ai-training is left out, because this control token doesn’t belong to any single operator or crawler.

Whether your content goes into training depends on the user agent you block in robots.txt. The crawler names in your server logs tell you little about that, for two reasons:

• Control tokens don’t fetch anything. Google-Extended and Applebot-Extended have no crawler of their own. They only determine how Google and Apple may use the content that Googlebot and Applebot fetch anyway.
• One fetch can serve several purposes. If a site allows both OAI-SearchBot and GPTBot, OpenAI may use the results of a single crawl for search and training. Data collected by Applebot can go into Apple’s training unless Applebot-Extended is blocked.

With CCBot, Common Crawl runs an open web archive that many providers reuse. Of 47 language models from 2019 to 2023 that the Mozilla Foundation reviewed, at least 30 used filtered versions of it for training.

Source: Mozilla Foundation (external site), Feb 6, 2024 · 47 language models, published 2019 to October 2023

• Search crawlers. A page that Googlebot isn’t allowed to read appears in Google Search without a description at best. It isn’t eligible for AI Overviews or AI Mode, because both require a page that may be shown with a snippet. Bing explicitly advises against blocking Bingbot. Bing and Copilot rely on the same crawling.
• User-initiated fetches. Anthropic and Mistral AI apply robots.txt to Claude-User and MistralAI-User. At OpenAI, Perplexity, Google, Meta and Amazon, the file doesn’t reliably apply to user-initiated fetches, because a person triggers them. So you can’t rely on a block to stop them. Where it does work, it can cost you visibility, according to Anthropic.
• Training. According to OpenAI and Apple, blocking GPTBot or Applebot-Extended doesn’t change whether your pages appear in their search. Google-Extended is the exception: if you block it, your content is also left out of the sources the Gemini app relies on for its answers. Google Search isn’t affected.
• What a block doesn’t undo. According to Anthropic, blocking ClaudeBot applies to future content. With any provider, expect that content collected earlier may remain in training data.

Search engines have separate controls for their AI answers. Since August 31, 2026, the Search generative AI control in Google Search Console has let you exclude your website from AI Overviews and AI Mode. Your site stays in the rest of Google Search. Our how-to Opting out of AI Overviews covers when that is worth it and what it costs. From February 1, 2027, Microsoft will honor separate preferences for training and for grounding, its term for using content in AI answers. After March 1, 2027, Microsoft will no longer read NOARCHIVE and NOCACHE as such preferences. Our article What is AI grounding? explains how to set the new preference. Our article on AI Overviews and AI Mode explains how Google picks the sources for its AI answers.

Training is a trade-off. If you earn money from the content itself, such as courses, expert articles or data, there’s a strong case for blocking training. In our assessment, if your content promotes a service you deliver on site or in projects, you have little to lose from training.

Few studies look at what a block actually does. For 24 large English-language news sites, weekly visits fell by about 7% in the six weeks after a block, compared with sites without one. They had blocked user agents such as GPTBot, CCBot and Google-Extended. The authors consider it plausible that AI answers mentioned these sites less often afterward. After about 20 weeks, the effect was no longer statistically significant.

Source: Zhao, Berman (preprint) (external site), Apr 15, 2026 · 30 news sites, most blocks in 2023

So far, few websites in Germany, Austria and Switzerland block training. In a sample of 24,550 hosts ending in .de, .at and .ch, only 2.27% blocked the training crawler GPTBot in August 2026. Among 28 large news and publishing sites, the figure was 61%, and among 16 large online stores, 6%. Of the 17 publishers that blocked GPTBot, only four also blocked OAI-SearchBot. The analysis doesn’t capture blocks in firewalls or CDNs.

Source: Ahrefs (in German) (external site), Sep 18, 2026 · vendor study · Common Crawl, Aug 2026; small groups of publishers and stores

Example: allow search, block training. This file blocks the training crawlers and control tokens from the table and lets all other bots in. Delete any groups you don’t want, and use your own domain.

# Block training; search and user-initiated fetches stay allowed
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: MistralAI-Training
Disallow: /

# shared control token from several providers, draft since Oct 30, 2026
User-agent: __usage_ai-training
Disallow: /

# also removes your content from the Gemini app's sources
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Meta-ExternalAgent and Amazonbot aren’t in the example, because Meta and Amazon also use them for purposes other than training. Add them if you want to rule out those purposes too.

Common mistakes:

• A crawler’s own group replaces the general rules. A crawler with a group of its own no longer follows the rules under User-agent: *. If the * group contains Disallow: /internal/, add that line to every named group as well.
• The block from the build phase stays in place. User-agent: * with Disallow: / blocks all crawlers, including Googlebot and Bingbot.
• Your robots.txt returns a server error. With a 5xx error, RFC 9309 says crawlers must assume that everything is disallowed. If the file is missing with a 404 status, they may fetch everything.
• Applebot inherits the rules for Googlebot. If your file doesn’t name Applebot, Apple follows the rules for Googlebot. A block on Googlebot then also hits Apple’s search.

• Anyone can read the file. A line such as Disallow: /internal/ makes the path public. The RFC 9309 standard (external site) itself points this out in its security considerations.
• User agents can be faked. A request can claim to be GPTBot without coming from OpenAI. You can tell real crawlers by their IP address.
• Only a firewall or CDN can technically block crawlers. A CDN is the network that delivers your pages to visitors. Anthropic points out that blocking IP addresses isn’t a reliable way to opt out of its crawling. The crawler can then no longer read your robots.txt.
• The file doesn’t keep pages out of Google Search. A blocked URL can still appear in the results without a description. A noindex rule helps here, but Google only sees it if it’s allowed to fetch the page. Our article on websites not showing on Google goes into more detail.

For a reverse DNS lookup, you look up the host name for the IP address in your server log. Then you check that this name points back to the same IP address. For Google, the name must end in googlebot.com, google.com or googleusercontent.com.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

On Windows, nslookup does the same. How to count and verify fetches as part of your analysis is covered in our how-to on tracking AI traffic.

A newer method is Web Bot Auth: the provider signs its requests, and your server or CDN checks the signature. Google is testing it for Google-Agent. Web Bot Auth is still a draft at the IETF, and Google advises you to keep verifying by IP address.

A CDN is the delivery network between your server and your visitors. Check these places:

• Cloudflare’s default. Since July 1, 2026, Cloudflare has let you manage AI crawlers in three groups: search, training and agents acting on a person’s behalf. Cloudflare’s agent group includes user-initiated fetches such as ChatGPT-User. For new domains, Cloudflare’s documentation lists “Allow” as the default for all three groups (as of November 2026). Since September 15, 2026, Cloudflare has suggested stricter settings during setup if you say your pages earn money from ads. In that case, search stays allowed, training is disallowed and agents are blocked on pages with ads. On July 1, 2025, Cloudflare started asking anyone adding a new domain whether to allow AI crawlers.
• Cloudflare’s settings. The settings are in the dashboard under Security > Settings, in the “AI Crawler and Agent access policies” section, and are available on all plans. Set the Search and Agent controls to “Allow” if you want to appear in AI answers. “Disallow” is available for Training only. If you choose “Disallow” with Bot Preference Sync turned on, Cloudflare publishes a no-training preference in robots.txt and blocks training-only crawlers. Googlebot, Bingbot and Applebot can keep fetching your pages for search. Since September 15, 2026, “Block” and “Block on pages with ads” under Training also apply to those three crawlers, so both settings affect search as well.
• Existing domains on Cloudflare. On September 15, 2026, Cloudflare announced that it would carry existing settings over to the new controls. Earlier training blocks become “Disallow.” If only the older “Block AI Bots” switch was on, Cloudflare says the result is search allowed, training disallowed and agents blocked on pages with ads. So check what your account shows under Search, Agent and Training.
• A modified robots.txt. Cloudflare puts its own rules in front of your robots.txt if Bot Preference Sync is turned on and you block at least one group or disallow training. If all three groups are set to “Allow,” Cloudflare serves your robots.txt unchanged. Cloudflare says Bot Preference Sync is turned on from the start for new domains. So check the file a browser actually receives from your domain.
• Firewall allowlists. OpenAI recommends allowing requests from its published IP ranges. Perplexity describes such allowlists for the Cloudflare and AWS firewalls.

Does Cloudflare block AI crawlers? It depends on your settings. For new domains, Cloudflare’s documentation lists “Allow” as the default for search, agents and training. Since September 15, 2026, Cloudflare has suggested stricter settings if you say during setup that your pages earn money from ads: training disallowed and agents blocked on pages with ads. Cloudflare says you can change these settings during setup or at any time afterward.

A comparison shows whether your CDN filters by user agent. Request your home page once normally and once with a crawler’s user agent. Then compare the first line of both responses. Our how-to on JavaScript and AI crawlers shows whether a request fails at the firewall, in robots.txt or because of JavaScript.

curl -sI https://www.example.com/
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.example.com/

If the second response shows a different status code, such as 403 instead of 200, your CDN turns away requests with that user agent. The test won’t reveal blocks by IP address, because it comes from your own network. Only your CDN or server logs show whether real crawlers get through. Our how-to on getting recommended by ChatGPT covers the other steps that help AI assistants find your website.

• What providers commit to. In the General-Purpose AI Code of Practice (external site), signatories commit to using crawlers that read and follow robots.txt as specified in RFC 9309. This applies when they collect data for text and data mining and for training their models. According to the European Commission, the signatories include Amazon, Anthropic, Google, Microsoft, Mistral AI and OpenAI.
• What’s new. On October 30, 2026, OpenAI, Microsoft and Anthropic presented a shared control token against AI training, which Google also refers to. In robots.txt, it reads User-agent: __usage_ai-training with Disallow: /. Anthropic says it began honoring the token that day. Microsoft will honor it from February 1, 2027, and Google plans to support it in early 2027. OpenAI plans to follow in the coming months (external site).
• What to keep in mind. The token is based on an Internet-Draft submitted to the IETF (external site); it isn’t a standard yet. A block under User-agent: * doesn’t count for this token. The user agents of individual crawlers still apply.

Whether a reservation is effective in a particular case is a legal question. This section is a technical overview and is not legal advice. If in doubt, consult a lawyer.