In short
robots.txt is the long-standing standard (RFC 9309, 2022) that tells crawlers which paths they may fetch, and AI companies publish user agents for it, like GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and the Google-Extended token. llms.txt is a 2024 proposal for a Markdown file pointing AI tools to a site's most useful pages; it is not a standard. Decide separately whether to allow AI training and AI search, and keep every discovery file aligned.
robots.txt: the access rules.
robots.txt sits at the root of a domain and tells crawlers which paths they may fetch. It was formalised as RFC 9309 in 2022.
- Rules are grouped by user agent, with Allow and Disallow lines for paths.
- Reputable crawlers follow it, but it is a request, not access control; protect private pages with sign-in.
- Blocking a URL in robots.txt does not reliably remove it from search; a noindex on a crawlable page does.
- It is also the usual place to point crawlers to your sitemap.
The AI user agents worth knowing.
AI companies publish their crawler names, and several split training from search.
- OpenAI: GPTBot gathers training data, OAI-SearchBot finds pages for ChatGPT search, and ChatGPT-User fetches pages a person asks about.
- Anthropic: ClaudeBot is its crawler, with separate agents for search and for fetches a person requests.
- Google: Google-Extended is a token that governs use of your content for Gemini models; Google Search, including AI Overviews, uses Googlebot.
- Perplexity: PerplexityBot indexes pages for its answers. Check each company's documentation, because names and behaviour change.
Decide training and search separately.
Blocking every AI agent can remove you from the AI answers your customers already use.
- To appear in AI search, allow the search agents such as OAI-SearchBot and PerplexityBot.
- To keep content out of model training, disallow training agents such as GPTBot and Google-Extended.
- Keep Googlebot allowed if you want to stay in Google Search and AI Overviews.
- Review the list a few times a year as new agents appear.
llms.txt: a proposed guide, not a rule.
llms.txt was proposed in September 2024 as a Markdown file at /llms.txt that summarises a site and links its most useful pages for language models.
- It is a community proposal, not a standard, and major search engines have not said they use it for ranking.
- It is cheap to publish and handy for documentation sites and AI tools that do read it.
- Keep it short: a title, a one-paragraph summary, and grouped links with one-line descriptions.
- Some sites also publish llms-full.txt with the full page text for tools that want it.
Keep every discovery file in step.
Machines trust a site more when its files agree with each other and with the pages.
- Every URL in the sitemap, llms.txt and feeds should be canonical and load without redirects or errors.
- Sign-in pages, admin tools and APIs stay out of all of them.
- Update the files when you publish, move or remove a page.
- Anslation publishes a sitemap, an RSS feed, llms.txt and AI-readable JSON files for its public pages, and keeps account areas out of them.
Questions
Questions this article answers.
What is the difference between llms.txt and robots.txt?
robots.txt is a standard that tells crawlers which paths they may fetch. llms.txt is a proposed Markdown file that summarises a site and points AI tools to its most useful pages. robots.txt sets access; llms.txt only offers guidance.
Does blocking GPTBot remove my site from ChatGPT search?
Not by itself. OpenAI uses GPTBot for training data and OAI-SearchBot for ChatGPT search, and each is controlled separately in robots.txt. Check OpenAI's current crawler documentation before changing your rules.
Does Google-Extended affect my Google rankings?
No. Google says Google-Extended only governs whether content may be used for its Gemini models, and does not affect inclusion or ranking in Google Search.
Do I need an llms.txt file?
It is optional. It costs little and may help AI tools that read it, but it is not a standard and no major search engine has said it uses it for ranking. Clear, crawlable pages matter more.
Put it to work
The solution this is about, and how to start with it.
Marketing
Positioning, landing pages, funnels, campaigns, analytics, and demand systems.