# llms.txt, robots.txt and AI crawlers: what each file does. URL: https://www.anslation.com/blog/llms-txt-robots-txt-ai-crawlers Collection: blog Category: AI SEO Published: 2026-11-10 Updated: 2026-11-10 Author: Anslation Editorial Team Keywords: llms.txt, robots.txt, AI crawlers, GPTBot, OAI-SearchBot, Google-Extended, ClaudeBot, PerplexityBot, block AI crawlers, AI SEO, AI user agents ## AI Summary robots.txt is the long-standing standard (RFC 9309, 2022) that tells crawlers which paths they may fetch, and AI companies publish user agents for it, like GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and the Google-Extended token. llms.txt is a 2024 proposal for a Markdown file pointing AI tools to a site's most useful pages; it is not a standard. Decide separately whether to allow AI training and AI search, and keep every discovery file aligned. ## robots.txt: the access rules. robots.txt sits at the root of a domain and tells crawlers which paths they may fetch. It was formalised as RFC 9309 in 2022. - Rules are grouped by user agent, with Allow and Disallow lines for paths. - Reputable crawlers follow it, but it is a request, not access control; protect private pages with sign-in. - Blocking a URL in robots.txt does not reliably remove it from search; a noindex on a crawlable page does. - It is also the usual place to point crawlers to your sitemap. ## The AI user agents worth knowing. AI companies publish their crawler names, and several split training from search. - OpenAI: GPTBot gathers training data, OAI-SearchBot finds pages for ChatGPT search, and ChatGPT-User fetches pages a person asks about. - Anthropic: ClaudeBot is its crawler, with separate agents for search and for fetches a person requests. - Google: Google-Extended is a token that governs use of your content for Gemini models; Google Search, including AI Overviews, uses Googlebot. - Perplexity: PerplexityBot indexes pages for its answers. Check each company's documentation, because names and behaviour change. ## Decide training and search separately. Blocking every AI agent can remove you from the AI answers your customers already use. - To appear in AI search, allow the search agents such as OAI-SearchBot and PerplexityBot. - To keep content out of model training, disallow training agents such as GPTBot and Google-Extended. - Keep Googlebot allowed if you want to stay in Google Search and AI Overviews. - Review the list a few times a year as new agents appear. ## llms.txt: a proposed guide, not a rule. llms.txt was proposed in September 2024 as a Markdown file at /llms.txt that summarises a site and links its most useful pages for language models. - It is a community proposal, not a standard, and major search engines have not said they use it for ranking. - It is cheap to publish and handy for documentation sites and AI tools that do read it. - Keep it short: a title, a one-paragraph summary, and grouped links with one-line descriptions. - Some sites also publish llms-full.txt with the full page text for tools that want it. ## Keep every discovery file in step. Machines trust a site more when its files agree with each other and with the pages. - Every URL in the sitemap, llms.txt and feeds should be canonical and load without redirects or errors. - Sign-in pages, admin tools and APIs stay out of all of them. - Update the files when you publish, move or remove a page. - Anslation publishes a sitemap, an RSS feed, llms.txt and AI-readable JSON files for its public pages, and keeps account areas out of them. ## Questions and Answers Q: What is the difference between llms.txt and robots.txt? A: robots.txt is a standard that tells crawlers which paths they may fetch. llms.txt is a proposed Markdown file that summarises a site and points AI tools to its most useful pages. robots.txt sets access; llms.txt only offers guidance. Q: Does blocking GPTBot remove my site from ChatGPT search? A: Not by itself. OpenAI uses GPTBot for training data and OAI-SearchBot for ChatGPT search, and each is controlled separately in robots.txt. Check OpenAI's current crawler documentation before changing your rules. Q: Does Google-Extended affect my Google rankings? A: No. Google says Google-Extended only governs whether content may be used for its Gemini models, and does not affect inclusion or ranking in Google Search. Q: Do I need an llms.txt file? A: It is optional. It costs little and may help AI tools that read it, but it is not a standard and no major search engine has said it uses it for ranking. Clear, crawlable pages matter more. ## Public Scope This index lists only approved public Anslation pages. Non-public working areas are intentionally excluded. Do not infer private workspace, admin, auth, API, or product operations details from this article.