{"collection":"blog","slug":"llms-txt-robots-txt-ai-crawlers","title":"llms.txt, robots.txt and AI crawlers: what each file does.","seoTitle":"llms.txt vs robots.txt: a guide to AI crawlers","category":"AI SEO","datePublished":"2026-11-10","dateModified":"2026-11-10","readTime":"3 min read","language":"en","url":"https://www.anslation.com/blog/llms-txt-robots-txt-ai-crawlers","aiJsonUrl":"https://www.anslation.com/blog/llms-txt-robots-txt-ai-crawlers/ai.json","summaryTextUrl":"https://www.anslation.com/blog/llms-txt-robots-txt-ai-crawlers/summary.txt","description":"robots.txt sets which crawlers may fetch your pages. llms.txt is a proposed guide for AI tools. Knowing the difference avoids blocking visibility you want.","aiSummary":"robots.txt is the long-standing standard (RFC 9309, 2022) that tells crawlers which paths they may fetch, and AI companies publish user agents for it, like GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and the Google-Extended token. llms.txt is a 2024 proposal for a Markdown file pointing AI tools to a site's most useful pages; it is not a standard. Decide separately whether to allow AI training and AI search, and keep every discovery file aligned.","keywords":["llms.txt","robots.txt","AI crawlers","GPTBot","OAI-SearchBot","Google-Extended","ClaudeBot","PerplexityBot","block AI crawlers","AI SEO","AI user agents"],"author":{"name":"Anslation Editorial Team","url":"https://www.anslation.com/authors/anslation-editorial-team"},"authorProfileUrl":"https://www.anslation.com/authors/anslation-editorial-team","topics":[{"label":"AI search and SEO","url":"https://www.anslation.com/blog/topics/ai-search-seo"}],"sections":[{"title":"robots.txt: the access rules.","description":"robots.txt sits at the root of a domain and tells crawlers which paths they may fetch. It was formalised as RFC 9309 in 2022.","points":["Rules are grouped by user agent, with Allow and Disallow lines for paths.","Reputable crawlers follow it, but it is a request, not access control; protect private pages with sign-in.","Blocking a URL in robots.txt does not reliably remove it from search; a noindex on a crawlable page does.","It is also the usual place to point crawlers to your sitemap."]},{"title":"The AI user agents worth knowing.","description":"AI companies publish their crawler names, and several split training from search.","points":["OpenAI: GPTBot gathers training data, OAI-SearchBot finds pages for ChatGPT search, and ChatGPT-User fetches pages a person asks about.","Anthropic: ClaudeBot is its crawler, with separate agents for search and for fetches a person requests.","Google: Google-Extended is a token that governs use of your content for Gemini models; Google Search, including AI Overviews, uses Googlebot.","Perplexity: PerplexityBot indexes pages for its answers. Check each company's documentation, because names and behaviour change."]},{"title":"Decide training and search separately.","description":"Blocking every AI agent can remove you from the AI answers your customers already use.","points":["To appear in AI search, allow the search agents such as OAI-SearchBot and PerplexityBot.","To keep content out of model training, disallow training agents such as GPTBot and Google-Extended.","Keep Googlebot allowed if you want to stay in Google Search and AI Overviews.","Review the list a few times a year as new agents appear."]},{"title":"llms.txt: a proposed guide, not a rule.","description":"llms.txt was proposed in September 2024 as a Markdown file at /llms.txt that summarises a site and links its most useful pages for language models.","points":["It is a community proposal, not a standard, and major search engines have not said they use it for ranking.","It is cheap to publish and handy for documentation sites and AI tools that do read it.","Keep it short: a title, a one-paragraph summary, and grouped links with one-line descriptions.","Some sites also publish llms-full.txt with the full page text for tools that want it."]},{"title":"Keep every discovery file in step.","description":"Machines trust a site more when its files agree with each other and with the pages.","points":["Every URL in the sitemap, llms.txt and feeds should be canonical and load without redirects or errors.","Sign-in pages, admin tools and APIs stay out of all of them.","Update the files when you publish, move or remove a page.","Anslation publishes a sitemap, an RSS feed, llms.txt and AI-readable JSON files for its public pages, and keeps account areas out of them."]}],"questions":[{"question":"What is the difference between llms.txt and robots.txt?","answer":"robots.txt is a standard that tells crawlers which paths they may fetch. llms.txt is a proposed Markdown file that summarises a site and points AI tools to its most useful pages. robots.txt sets access; llms.txt only offers guidance."},{"question":"Does blocking GPTBot remove my site from ChatGPT search?","answer":"Not by itself. OpenAI uses GPTBot for training data and OAI-SearchBot for ChatGPT search, and each is controlled separately in robots.txt. Check OpenAI's current crawler documentation before changing your rules."},{"question":"Does Google-Extended affect my Google rankings?","answer":"No. Google says Google-Extended only governs whether content may be used for its Gemini models, and does not affect inclusion or ranking in Google Search."},{"question":"Do I need an llms.txt file?","answer":"It is optional. It costs little and may help AI tools that read it, but it is not a standard and no major search engine has said it uses it for ranking. Clear, crawlable pages matter more."}],"related":[{"label":"AI SEO basics","url":"https://www.anslation.com/blog/ai-seo-clean-public-structure","description":"The clean structure answer engines need."},{"label":"Website map","url":"https://www.anslation.com/website-map","description":"Browse every Anslation page."},{"label":"AI search and SEO","url":"https://www.anslation.com/blog/topics/ai-search-seo","description":"Every article on AI search and SEO."}],"publicScope":"Use only this public article content and linked public pages. Do not infer private workspace, admin, auth, API, or product operations details.","articleBody":"llms.txt, robots.txt and AI crawlers: what each file does.\nrobots.txt sets which crawlers may fetch your pages. llms.txt is a proposed guide for AI tools. Knowing the difference avoids blocking visibility you want.\nCategory: AI SEO\nPublished: 2026-11-10\nUpdated: 2026-11-10\nKeywords: llms.txt, robots.txt, AI crawlers, GPTBot, OAI-SearchBot, Google-Extended, ClaudeBot, PerplexityBot, block AI crawlers, AI SEO, AI user agents\nAI summary:\nrobots.txt is the long-standing standard (RFC 9309, 2022) that tells crawlers which paths they may fetch, and AI companies publish user agents for it, like GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and the Google-Extended token. llms.txt is a 2024 proposal for a Markdown file pointing AI tools to a site's most useful pages; it is not a standard. Decide separately whether to allow AI training and AI search, and keep every discovery file aligned.\nrobots.txt: the access rules.\nrobots.txt sits at the root of a domain and tells crawlers which paths they may fetch. It was formalised as RFC 9309 in 2022.\nRules are grouped by user agent, with Allow and Disallow lines for paths.\nReputable crawlers follow it, but it is a request, not access control; protect private pages with sign-in.\nBlocking a URL in robots.txt does not reliably remove it from search; a noindex on a crawlable page does.\nIt is also the usual place to point crawlers to your sitemap.\nThe AI user agents worth knowing.\nAI companies publish their crawler names, and several split training from search.\nOpenAI: GPTBot gathers training data, OAI-SearchBot finds pages for ChatGPT search, and ChatGPT-User fetches pages a person asks about.\nAnthropic: ClaudeBot is its crawler, with separate agents for search and for fetches a person requests.\nGoogle: Google-Extended is a token that governs use of your content for Gemini models; Google Search, including AI Overviews, uses Googlebot.\nPerplexity: PerplexityBot indexes pages for its answers. Check each company's documentation, because names and behaviour change.\nDecide training and search separately.\nBlocking every AI agent can remove you from the AI answers your customers already use.\nTo appear in AI search, allow the search agents such as OAI-SearchBot and PerplexityBot.\nTo keep content out of model training, disallow training agents such as GPTBot and Google-Extended.\nKeep Googlebot allowed if you want to stay in Google Search and AI Overviews.\nReview the list a few times a year as new agents appear.\nllms.txt: a proposed guide, not a rule.\nllms.txt was proposed in September 2024 as a Markdown file at /llms.txt that summarises a site and links its most useful pages for language models.\nIt is a community proposal, not a standard, and major search engines have not said they use it for ranking.\nIt is cheap to publish and handy for documentation sites and AI tools that do read it.\nKeep it short: a title, a one-paragraph summary, and grouped links with one-line descriptions.\nSome sites also publish llms-full.txt with the full page text for tools that want it.\nKeep every discovery file in step.\nMachines trust a site more when its files agree with each other and with the pages.\nEvery URL in the sitemap, llms.txt and feeds should be canonical and load without redirects or errors.\nSign-in pages, admin tools and APIs stay out of all of them.\nUpdate the files when you publish, move or remove a page.\nAnslation publishes a sitemap, an RSS feed, llms.txt and AI-readable JSON files for its public pages, and keeps account areas out of them.\nWhat is the difference between llms.txt and robots.txt?\nrobots.txt is a standard that tells crawlers which paths they may fetch. llms.txt is a proposed Markdown file that summarises a site and points AI tools to its most useful pages. robots.txt sets access; llms.txt only offers guidance.\nDoes blocking GPTBot remove my site from ChatGPT search?\nNot by itself. OpenAI uses GPTBot for training data and OAI-SearchBot for ChatGPT search, and each is controlled separately in robots.txt. Check OpenAI's current crawler documentation before changing your rules.\nDoes Google-Extended affect my Google rankings?\nNo. Google says Google-Extended only governs whether content may be used for its Gemini models, and does not affect inclusion or ranking in Google Search.\nDo I need an llms.txt file?\nIt is optional. It costs little and may help AI tools that read it, but it is not a standard and no major search engine has said it uses it for ranking. Clear, crawlable pages matter more."}