In short
For ChatGPT and other AI assistants to find and recommend your business, AI crawlers must be allowed to read your website, and your services should be spelled out in the HTML. Consistent company facts also matter, as do mentions on other websites, which show the strongest measured link to AI visibility. No one can guarantee a recommendation.
Key takeaways
- Access comes first: if robots.txt, your host or your firewall blocks the search crawler, your site may not appear in the answers.
- Services, location, price range and contact details belong in the delivered HTML, because many AI crawlers don’t run JavaScript.
- Mentions on other sites show the strongest measured link to AI visibility. That’s a correlation.
- AI answers fluctuate a lot. Single questions tell you little; the same questions, repeated, show a direction.
Can AI crawlers access your website?
AI crawlers can only read your website reliably if robots.txt, your host and your firewall let them through. AI crawlers are the programs that providers such as OpenAI, Anthropic and Perplexity use to fetch websites. If robots.txt, your host or your firewall blocks a provider’s search crawler, your website may not appear in that provider’s answers.
Work through the steps in order. Each one ends with a “done when” test, so you can use this how-to as an AI search optimization checklist.
Why this comes first: OpenAI says sites that block its search crawler, OAI-SearchBot, won’t be shown in ChatGPT search answers, though they can still appear as navigational links (OpenAI’s crawler overview (external site)). OpenAI and Perplexity treat fetches that someone triggers in a chat differently: robots.txt doesn’t always apply to those. For Google’s AI Overviews and AI Mode, a page must be indexed and eligible to be shown with a snippet, according to Google (external site). Your site also has to be included in generative AI features (external site) in Search Console, which is the default. If your website is missing from Google, our how-to Why isn’t my website showing up on Google? covers the causes. Our guide to generative engine optimization (GEO) explains how each system picks its sources.
How to do it:
- Open
https://example.com/robots.txtin your browser, with your own domain in the address. This text file tells crawlers what they may fetch. Look forDisallow: /and for the namesOAI-SearchBot,Claude-SearchBot,PerplexityBot,GooglebotandBingbot. - Ask your host or service provider whether your firewall or CDN (the network that delivers your pages) blocks or throttles these crawlers. You can’t see that in robots.txt. Since September 15, 2026, Cloudflare has proposed one of two presets for AI crawlers (external site) for new websites. Search is allowed in both. For sites that carry advertising, training is disallowed and AI agents are blocked on pages with ads. Cloudflare counts fetches made on a person’s behalf (external site) as AI agents, for example when a chat assistant opens your page to answer a question. For all other sites, training and AI agents are allowed as well. Perplexity (external site) also points out that you may need to allow its bots in your firewall.
- Decide separately about search crawlers and training crawlers. According to OpenAI (external site) and Anthropic (external site), OAI-SearchBot and Claude-SearchBot are for search, while GPTBot and ClaudeBot collect content for training. Besides training, Google-Extended (external site) also controls whether the Gemini app may use your content for its answers. It doesn’t affect Google Search. Our article AI crawlers and robots.txt lists every user agent with its purpose and what blocking it does.
Example: The robots.txt on datafalke.de allows all crawlers, including those for training:
User-agent: *
Allow: /
Sitemap: https://datafalke.de/sitemap.xmlTo keep your content out of training while leaving search open, add these lines. At OpenAI, a change takes effect in search after about 24 hours.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Common mistake: A block on all crawlers from the build phase stays in place after launch.
Done when robots.txt doesn’t block any search crawler, your host has confirmed that the firewall and CDN let these crawlers through, and the AI setting in Search Console is set to include your site.
Are your services visible in the HTML, without JavaScript?
Your services should be in the HTML the server delivers, because many AI crawlers don’t run JavaScript. A 2024 measurement by Vercel found this for the crawlers from OpenAI and Anthropic: they download scripts but don’t execute them. Anything that only appears in the browser is invisible to them.
Source: Vercel (external site), · vendor study
Why: HTML is the page code the server sends first. Googlebot also renders pages with JavaScript, so the gap often goes unnoticed in Google Search. Bingbot can do this too, but in a server log analysis (external site) from September 2026, it did so for only about one in 800 pages. In the same analysis, ChatGPT-User fetched not one script in 37,045 page requests. The analysis covers a single news site and comes from a vendor of log tools. We haven’t found a newer large-scale measurement. A searchVIU overview (external site) reports one isolated case from an October 2025 test: a price inserted by JavaScript showed up in a Perplexity answer. One page proves little, so don’t count on AI crawlers running your JavaScript. Since September 2026, there have been conflicting reports on OAI-SearchBot and GPTBot. Our how-to on JavaScript and AI crawlers puts them in context and gives commands for Windows, macOS and Linux with expected output for several pages.
Source: Forge AI Bot Log (external site), · vendor study · one news site, 30 days
How to do it:
- Open your most important service page and view its page source. In most browsers on Windows, that’s
Ctrl+U. In Chrome on a Mac, it’sCommand+Option+U. - Search the page source with
Ctrl+F(Command+Fon a Mac) for the name of your service, your location, your price range and your contact details. Our how-to Service pages for AI search lists the details a service page needs. - If anything is missing, ask your service provider to deliver the page as static HTML. That means prebuilt pages your browser can display without running any code.
Example: Try it on this page. The text you’re reading is all in the page source. That’s also how we deliver the websites we build.
Common mistake: Key details appear only in tabs, accordions, images or PDFs. Microsoft warns (external site) that AI systems may not render hidden content. Our guide to publishing prices explains how to put a price into the HTML as a full sentence, not just on a pricing card or in a calculator.
Done when your service, location, price range and contact details are in the page source of every important page.
Are your company facts clear and consistent everywhere?
Your company facts are clear when your name, address, services and point of contact read the same on your website, on profiles and in directories. Microsoft recommends clear, consistent names because, it says, that makes citations in AI answers more accurate. New brands especially need this, because there’s still little about them online.
Why: For an experiment, Ahrefs invented a brand and published three made-up stories about it online. Almost every AI product in the test repeated the false information. Only two ChatGPT models pushed back, citing the official FAQ in 84% of their answers. The experiment shows only that a clear official page can help counter false information, and not with every system. It doesn’t test consistent company facts. For new brands, we still think clear facts of your own are the first step.
Source: Ahrefs (external site), · vendor study · one made-up brand, eight AI products, 56 questions
Source: Microsoft Bing (external site), · undated guidelines, date of access, item 16
How to do it:
- Write one standard description with your name, location, service and legal entity.
- Use it word for word on your home page, your about page, your legal notice (Impressum) and every profile.
- Keep a list of all your profiles and listings, and compare spelling, address and web address.
- Have structured data of the type Organization (external site) added to your home page. It helps Google tell your company apart from others. According to Google’s guide (external site), there’s no special markup for AI search. What structured data can and can’t do is covered in our article Structured data.
Example: Our standard description reads: “Datafalke – websites, SaaS and business software from Bottrop, Germany. Operated by Mohammad Alboush – Alboush IT & Webdesign.” Our templates read it from one central file. Where it appears as plain text, as it does here, we compare it against that file before publishing. You’ll find the full details in our legal notice.
Common mistake: One place has the short name, another the full company name, and an old directory still shows your previous address. Why AI systems mix up companies, and what helps, is explained in our article Entity SEO.
Done when your list is complete and your name, address and standard description match word for word everywhere.
Do your pages answer your customers’ questions up front?
Your pages answer customers’ questions up front when the first sentence under each heading contains the answer. Microsoft describes how Copilot breaks content into smaller pieces and can often lift question-and-answer pairs word for word. Google doesn’t require a specific format, but it does recommend firsthand experience that goes beyond common knowledge.
This is the heart of optimizing your website for AI search: your pages answer what customers actually ask.
How to do it:
- Collect 10 questions customers ask you by email, on the phone or in a first meeting. Price, timeline, service area and process usually come up.
- Turn each question into a heading. Write the answer first, then the reasoning and evidence. For a single service page, our how-to Service pages for AI search gives you a blueprint in eight fields.
- Add what only you know: your own figures, processes and typical cases from your business. Our article What is E-E-A-T? explains what Google means by experience.
Example: This how-to is built that way: the headings are questions, and the first sentence under each is the answer. The same goes for our guide to website costs.
Common mistake: You create a separate page for every wording of a question. Google treats this as spam (external site) if the main purpose is to influence rankings or AI answers. Google also says you don’t have to break your content into small chunks. Our article What is query fan-out? explains why one page can cover several sub-questions.
Done when each of the 10 questions has a place where the first sentence answers it on its own.
Where do other websites mention you?
Mentions on other websites show the strongest measured link to AI visibility. Across 75,000 brands, Ahrefs found correlations of 0.66–0.71 for web mentions and about 0.74 for YouTube mentions. That’s a correlation and doesn’t prove causation. So look for places with a real reason to write about you.
Source: Ahrefs, · vendor study · 75,000 brands, ChatGPT, AI Mode and AI Overviews
Ahrefs only studied domains with a Domain Rating above 40 and a keyword with at least 800 monthly searches. The study doesn’t show whether the link holds for small businesses.
Why: A University of Toronto study (external site) found that AI search clearly favors editorial third-party sources over brands’ own sites.
How to do it:
- Ask three AI assistants for providers of your service, and note which pages they cite as sources.
- List places where you have a real reason to appear: your chamber of commerce or trade association, partners and suppliers, trade media, regional press, talks, interviews and videos, including on YouTube, plus your profile on professional networks such as LinkedIn.
- Give them something worth citing, such as a figure from your business, a how-to or an expert article.
Example: If you give a talk at your chamber of commerce, ask for a short report on its website, with your name and a link. That’s a real reason, and the mention sits on someone else’s site.
Common mistake: Mentions are bought or staged. Google says (external site) that seeking inauthentic mentions “isn’t as helpful as it might seem.”
Done when your list names at least 10 suitable places and you have a specific reason for the first three.
Which directories and review sites are you missing from?
The directories and review sites that matter are the ones AI answers in your industry and region use as sources today. To find out which those are, run your own queries with questions your customers would ask. You should be listed on those sites with current, consistent facts.
How to do it:
- Put five questions like “Who offers [service] in [place]?” to ChatGPT, Perplexity and Microsoft Copilot. Note every directory they name.
- Create your Google Business Profile, a Bing Places listing and profiles on the sites on your list.
- Use your standard description and the same address there.
- Ask real customers for reviews.
Example: If an answer justifies its recommendation with Google reviews and cites an industry directory, your Google Business Profile and that directory go on your list.
Common mistake: A profile gets created and forgotten. Outdated details stay there and can end up in answers.
Done when every site from your queries has a current profile with the same facts as your website.
How can you tell whether it’s working?
You can tell whether your AI visibility work is paying off by asking a fixed set of customer questions regularly and repeatedly, and by checking the reports from Google and Bing. Single answers tell you little because they fluctuate a lot. According to Profound, the median page’s citation share halves within 11 days of its peak.
Source: Profound (external site), · vendor study · 883,000 pages
-
Google Search Console: Since August 31, 2026, the generative AI performance report (external site) has shown impressions from AI Overviews and AI Mode. If you don’t see it, your site usually hasn’t had enough impressions yet.
-
Bing Webmaster Tools: AI Performance (external site) shows citations in Microsoft Copilot and in AI-generated summaries in Bing, plus a sample of the phrases your content was retrieved for.
-
Model changes: When ChatGPT switched to GPT-5.5 on May 23, 2026, citations in German-language answers shifted by 47% within 48 hours, according to SISTRIX. SISTRIX itself calls this a correlation. Note dates like this next to your numbers.
Source: SISTRIX,
-
New inquiries: Ask new contacts how they found you, and write down the answers. It’s the simplest method, and it also shows what no report captures.
Asking each question three times is a compromise. For statements about a single question, a study with German-language questions (external site) calls for at least seven runs per question per day, and eight for cited sources. So three runs only show a direction across all questions. Our methodology describes how we plan to measure AI visibility. Our how-to How to measure AI search visibility walks you through measuring your own, step by step.
Source: Schulte, Bleeker, Kaufmann (preprint) (external site), · queried from servers in Switzerland
Done when 10 questions sit in a table, each asked at least three times in the same wording and logged with a date, and Search Console and Bing Webmaster Tools are set up.
What’s different for companies in Germany and the EU?
The same steps apply in Germany, and the market has its own figures. AI Overviews have been available in Germany since March 2025 and AI Mode since October 2025. Half of internet users aged 16 and over at least sometimes use AI chat tools instead of traditional internet search.
Source: Bitkom (in German) (external site), · 1,156 respondents aged 16+, including 1,030 internet users
Source: Google (external site), · AI Overviews
Source: Google (in German), · AI Mode
-
Bing on desktop: In October 2026, Bing held a 19.74% desktop market share in Germany. According to Microsoft, Web IQ (external site), which taps the Bing index, also powers Copilot and ChatGPT. So add your website to Bing Webmaster Tools. Our how-to Bing and IndexNow shows how to set it up and check your submissions.
Source: StatCounter (external site), · monthly figure, October 2026
-
Language: German-language expert sources count, and so do English sources in your field. Until April 2026, ChatGPT’s stable core of sources was 68% English even for German questions, according to SISTRIX (external site). In the days after the switch to GPT-5.5 in May 2026, ChatGPT cited originally German sources more often (external site) for German-language queries. The Toronto study (external site) also describes GPT models using different sources for each language. If you sell into Germany, write for that market in German, and check which English sources in your field get cited.
Source: SISTRIX (external site), , updated · stable core, December 2025 to April 2026
Done when your website is registered in Bing Webmaster Tools.
From our work: how do you analyze the sources in your industry?
Take 10 questions that ask for a provider, list every source cited in the answers in a table and give each source a type. The most common type shows you where to appear first. We’ll use the same categories for our own measurement. We’ll only publish our own numbers together with the measurement log; our methodology describes how we’ll go about it.
| Source type | How to recognize it | What to do about it |
|---|---|---|
| Directories and comparison sites | list many providers by place or service | create or check your listing, see step 6 |
| Job boards | job ads instead of provider pages | may mean suitable provider pages are missing: make your own service page clearer, see step 4 |
| Marketplaces | freelancers offering services on a platform | check whether a profile there fits what you offer |
| Agencies and freelancers | service pages and guides from individual providers | answer the same question in more detail on your site, see step 4 |
| Software, tool and hosting vendors | guides published by the vendors themselves | add your own experience and figures, see step 4 |
| Trade media | editorial articles | look for a reason to write an expert article or give an interview, see step 5 |
| Review platforms | star ratings and written reviews | ask real customers for reviews, see step 6 |
Ten questions in one system only show a moment in time. Ask each question several times and in several systems before you act on the results. For our own measurement, we plan to use several systems and three runs per question.