What is llms.txt, and does it actually work?

The idea behind it: web pages are built for people. Navigation, ads and JavaScript make it hard for large language models (LLMs) to pull out the main text. And a whole website rarely fits into a model’s context window, the amount of text it can process at once. So a short overview in a fixed location is meant to take AI agents straight to the content that matters (proposal on llmstxt.org (external site)). AI agents are programs that fetch web pages and complete tasks on their own, on a person’s behalf.

Version 2 adopts what has become common practice (changes in v2 (external site)):

• An llms.txt file can also sit in a subdirectory and then covers the pages below it.
• Individual pages can get a Markdown version at the same URL with a .md extension. Markdown is a simple text format in which, for example, # marks a heading.
• Agents find these files through link relations: rel="alternate" type="text/markdown" points to the Markdown version of a page, and rel="describedby" to the llms.txt file that covers it.

For comparison, robots.txt has been an IETF standard since September 2022 (RFC 9309 (external site)). Separately, the IETF is working on a way to express preferences about how AI may use content, in its AI Preferences working group (external site). These preferences are to go into robots.txt and into HTTP headers, the information a server sends along with every response. That work, too, is still a draft, as of September 14, 2026 (external site). Our guide to generative engine optimization (GEO) explains how llms.txt fits into work on AI visibility.

The structure, as set out in the proposal on llmstxt.org (external site):

1. # Name as the main heading, the only required part.
2. > Summary as a blockquote with the information needed to understand the rest of the file.
3. Free text without headings, for example a note on who your services are for.
4. Sections with ## Heading, each followed by links in the pattern - [Name](URL): Note.

By convention, a section called ## Optional collects links that an agent can skip when space is tight. As of version 2, it no longer has any special technical meaning.

# Example GmbH

> Example GmbH from Anytown plans and installs solar power systems for businesses.

We only work with business customers in North Rhine-Westphalia, Germany.

## Services
- [Solar for businesses](https://example.com/solar/): Process, timeline and funding
- [Maintenance](https://example.com/maintenance/): What our maintenance contracts cover

## Company
- [About us](https://example.com/about/): Team and location
- [Contact](https://example.com/contact/): Inquiry form and address

## Optional
- [Guides](https://example.com/guides/): Articles on battery storage and grid connection

According to the proposal, the links should point to content that language models can read easily, such as Markdown versions. If your website doesn’t have any, link to pages whose full text is in the HTML. Our how-to explains why that matters in the step Are your services visible in the HTML?

Some websites, especially documentation sites, also offer an llms-full.txt file. It puts the site’s entire text into a single file. According to the documentation platform Mintlify, the format was developed in collaboration with Anthropic (Mintlify, updated Mar 14, 2026 (external site)). The proposal itself doesn’t mention it.

Google says explicitly that this also applies to the AI features in Search: AI Overviews and AI Mode. On June 15, 2026, Google added that it’s fine to maintain the file for other services. Doing so will “neither harm nor help” your visibility in Google Search (Google Search Central (external site)). So if Googlebot fetches your llms.txt, that doesn’t mean anything. Google crawls many kinds of files without treating them specially.

The numbers in the “Sources” column refer to the source list at the end of this article. So the AI providers do use the file themselves: as publishers of documentation for developers and their coding agents, which are AI tools that help with programming. The proposal also names this as the most common use. Lighthouse’s reasoning for its audit is that without the file, agents may spend more time crawling a site to understand it (Chrome for Developers (external site)). The category is experimental and, unlike the other Lighthouse categories, doesn’t give a score from 0 to 100. Instead, the report shows how many of the agentic readiness checks a site passes (Lighthouse scoring (external site)).

To find out who actually fetches the file, Ahrefs analyzed the server logs of 137,210 domains that use the company’s analytics tool. Of these, 28% had a valid llms.txt file, and 97% of those files weren’t fetched a single time in May 2026. For the remaining 3%, bots made 96% of the requests. AI bots of all kinds accounted for 19.5% of the requests, and the AI providers’ search crawlers for 1.1%. Where there was no file, not a single AI bot asked for one.

Source: Ahrefs (external site), Jun 15, 2026 · vendor study · 137,210 domains, server logs from May 2026

Ahrefs points out the limits itself: its customers are more tech-savvy than average, and a fetch doesn’t prove that a system actually uses the file.

SE Ranking checked nearly 300,000 domains to see whether having an llms.txt file is linked to how often a domain is cited in AI answers. Of these domains, 10.13% had one. Neither the statistical analysis nor a machine learning model showed an effect. Without the llms.txt variable, the model even predicted citations more accurately.

Source: SE Ranking (external site), Oct 15, 2026 · vendor study · nearly 300,000 domains

Both analyses come from SEO software vendors. They measure correlations and fetches, not causes.

It’s a different story for agents that read documentation on purpose. According to Ahrefs, AI tools fetch an llms.txt file when a link, an index or an instruction points to it. They don’t go looking for one on their own. For software vendors, the file can therefore point agents to tutorials and API references. For what matters more for visibility in AI answers as things stand, see the section What’s proven about GEO, and what isn’t? in our guide.

Crawlers are programs that fetch websites automatically, for example to build a search index or to train AI models.

An llms.txt file replaces neither of the other two files, and it doesn’t override a block in robots.txt. You don’t need to list llms.txt in robots.txt or in your sitemap; the proposal doesn’t call for that. So the label “AI robots.txt” that you’ll see online is misleading.

In the EU’s General-Purpose AI Code of Practice for the AI Act, signatories commit to using crawlers that follow robots.txt as specified in RFC 9309 (Code of Practice, Copyright chapter, Measure 1.3 (external site)). Signatories include OpenAI, Anthropic, Google and Microsoft (European Commission, as of Oct 7, 2026 (external site)). The code doesn’t mention llms.txt. The shared control token against AI training that OpenAI announced on October 30, 2026, also goes in robots.txt (OpenAI (external site)).

For the lines you need and what’s new since October 2026, see the section What does a rights reservation mean for AI training? in our article on AI crawlers and robots.txt. The first step of our ChatGPT how-to also shows the lines for individual training crawlers.

This is a technical assessment. If in doubt, a lawyer can tell you what applies in your case.

How to create an llms.txt file:

1. Choose your pages: Include the pages that explain what you offer, such as services, process, price range and contact. Keep the file small so that a model can read it in one go.
2. Create the file: Write the file as UTF-8 text following the template above, and put it in your root directory. Then open its URL in your browser. It must show the text file, not an error page that the server delivers with status 200 (“OK”). An llms.txt validator can also check the structure, but it won’t tell you whether anyone reads the file.
3. Use tools, but check the result: Several SEO plugins for WordPress and some website builders generate the file at the click of a button or automatically (list in the proposal (external site)). Read through the result anyway, because a tool doesn’t know which pages matter to your customers.
4. Treat it like code: Make every change traceable, link only to your own content and don’t write instructions for agents into the file. Language models process text from websites and files as input, and manipulated content can change their behavior. Experts call this indirect prompt injection (OWASP (external site)). Ahrefs has already found a research crawler called prompt-injection-survey among the fetchers.
5. Keep it up to date: When your services or URLs change, update the file too. It’s easiest if your system generates it from the same data as your pages.
6. Test it: Give an AI assistant only your llms.txt and ask it questions about what you offer. The proposal itself recommends this test.

If you want agents to find the file, point to it with rel="describedby" in your HTML head or in the HTTP Link header. If what you’re after is mentions in ChatGPT and other AI assistants, other steps matter more. Our how-to How to get your business recommended by ChatGPT takes you through them in order.