{"id":2490,"date":"2026-09-18T03:34:20","date_gmt":"2026-09-18T03:34:20","guid":{"rendered":"https:\/\/maxaeo.ai\/blog\/robots-txt-for-llms\/"},"modified":"2026-09-18T03:34:20","modified_gmt":"2026-09-18T03:34:20","slug":"robots-txt-for-llms","status":"publish","type":"post","link":"https:\/\/maxaeo.ai\/blog\/robots-txt-for-llms\/","title":{"rendered":"Robots.txt for LLMs: How to Control AI Crawlers on Your Site"},"content":{"rendered":"

\u4f5c\u8005\uff1amaxaeo.ai\uff5c\u53d1\u5e03\u65e5\u671f\uff1a2025-06-02\uff5c\u66f4\u65b0\u65e5\u671f\uff1a2025-06-02<\/em><\/p>\n

Robots.txt for LLMs is the practice of using your site’s robots.txt file to allow or block the crawler bots operated by AI companies<\/strong> \u2014 such as OpenAI’s GPTBot, Anthropic’s ClaudeBot, and PerplexityBot \u2014 so you control whether your content is used for model training or fetched for AI answers. For brands that want visibility in ChatGPT, Perplexity, and Gemini, the configuration is no longer optional housekeeping: one misplaced Disallow<\/code> rule can make you invisible in AI search results.<\/p>\n

This guide covers the current AI user-agents, copy-paste configuration templates, the most common mistakes we see in real audits, and how to decide what to block.<\/p>\n

What Is robots.txt for LLMs?<\/h2>\n

Robots.txt is a plain-text file at your domain root (e.g., https:\/\/example.com\/robots.txt<\/code>) that tells automated crawlers which paths they may access. It follows the Robots Exclusion Protocol<\/a>, standardized as RFC 9309 in 2022.<\/p>\n

In the LLM era, the same file now governs a second class of visitors: AI crawlers that either train foundation models<\/strong> on your content or retrieve pages in real time<\/strong> to answer user questions. The distinction matters enormously \u2014 blocking a training crawler protects your content from being absorbed into a model, while blocking a retrieval crawler removes you from live AI answers entirely.<\/p>\n

Robots.txt is advisory, not enforceable. Most major AI companies publicly commit to honoring it, but compliance is voluntary \u2014 which is why many sites pair it with CDN-level bot blocking (Cloudflare, for example, offers one-click AI bot blocking) for hard enforcement.<\/p>\n

Which AI User-Agents Should You Know?<\/h2>\n

There are roughly two dozen AI-related user-agents in the wild, but a handful account for the vast majority of crawl traffic. Here is the working list we maintain from analyzing robots.txt behavior and crawl logs across the sites we monitor:<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
User-agent<\/th>\nCompany<\/th>\nPurpose<\/th>\n<\/tr>\n<\/thead>\n
GPTBot<\/td>\nOpenAI<\/td>\nTraining data collection<\/td>\n<\/tr>\n
ChatGPT-User<\/td>\nOpenAI<\/td>\nReal-time browsing for ChatGPT answers<\/td>\n<\/tr>\n
OAI-SearchBot<\/td>\nOpenAI<\/td>\nIndexing for ChatGPT search<\/td>\n<\/tr>\n
ClaudeBot<\/td>\nAnthropic<\/td>\nTraining data collection<\/td>\n<\/tr>\n
Claude-User \/ Claude-SearchBot<\/td>\nAnthropic<\/td>\nReal-time retrieval<\/td>\n<\/tr>\n
PerplexityBot<\/td>\nPerplexity<\/td>\nRetrieval and indexing for answers<\/td>\n<\/tr>\n
Google-Extended<\/td>\nGoogle<\/td>\nControl for Gemini training (separate token)<\/td>\n<\/tr>\n
Meta-ExternalAgent<\/td>\nMeta<\/td>\nAI training<\/td>\n<\/tr>\n
Bytespider<\/td>\nByteDance<\/td>\nAI training<\/td>\n<\/tr>\n
Amazonbot<\/td>\nAmazon<\/td>\nAlexa \/ AI training<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

The critical pattern: OpenAI and Anthropic both separate training bots from answer-retrieval bots<\/strong>. If you block GPTBot<\/code> but allow ChatGPT-User<\/code> and OAI-SearchBot<\/code>, you can opt out of training while staying visible in ChatGPT answers. OpenAI documents these tokens on its GPTBot documentation page<\/a>.<\/p>\n

\"Table<\/figure>\n

Recommended robots.txt Configurations (Copy-Paste)<\/h2>\n

The right rules depend on your goal. Three common scenarios:<\/p>\n

Scenario A: Full visibility in AI answers (recommended for most brands)<\/h3>\n
User-agent: GPTBot\nAllow: \/\n\nUser-agent: ChatGPT-User\nAllow: \/\n\nUser-agent: OAI-SearchBot\nAllow: \/\n\nUser-agent: ClaudeBot\nAllow: \/\n\nUser-agent: Claude-User\nAllow: \/\n\nUser-agent: PerplexityBot\nAllow: \/\n\nUser-agent: Google-Extended\nAllow: \/\n<\/code><\/pre>\n
If your business model depends on being discovered \u2014 SaaS, e-commerce, publishing monetized by ads \u2014 this is usually the correct default.<\/p>\n
Scenario B: Allow AI answers, block training<\/h3>\n
User-agent: GPTBot\nDisallow: \/\n\nUser-agent: ClaudeBot\nDisallow: \/\n\nUser-agent: ChatGPT-User\nAllow: \/\n\nUser-agent: OAI-SearchBot\nAllow: \/\n\nUser-agent: PerplexityBot\nAllow: \/\n<\/code><\/pre>\n
This is the nuanced middle path: your pages can still be fetched and cited in answers, but you withhold them from training corpora.<\/p>\n
Scenario C: Block everything AI<\/h3>\n
User-agent: GPTBot\nDisallow: \/\nUser-agent: ChatGPT-User\nDisallow: \/\nUser-agent: OAI-SearchBot\nDisallow: \/\nUser-agent: ClaudeBot\nDisallow: \/\nUser-agent: Claude-User\nDisallow: \/\nUser-agent: Claude-SearchBot\nDisallow: \/\nUser-agent: PerplexityBot\nDisallow: \/\nUser-agent: Google-Extended\nDisallow: \/\nUser-agent: Meta-ExternalAgent\nDisallow: \/\nUser-agent: Bytespider\nDisallow: \/\nUser-agent: Amazonbot\nDisallow: \/\n<\/code><\/pre>\n
Appropriate for premium paywalled content or sites with strict licensing concerns \u2014 but understand the trade-off covered below.<\/p>\n
The #1 Mistake: Accidentally Blocking AI Retrieval<\/h2>\n
In our audits of brand sites that mysteriously don’t appear in ChatGPT or Perplexity answers, an overly broad robots.txt rule is one of the recurring root causes. The typical failure modes:<\/p>\n
    \n
  1. A legacy Disallow: \/<\/code> on a staging wildcard<\/strong> that got promoted to production.<\/li>\n
  2. CDN default rules<\/strong> (some managed WordPress and security plugins ship AI-blocking presets without telling you).<\/li>\n
  3. Confusing Google-Extended with Googlebot.<\/strong> Google-Extended<\/code> only controls AI training; blocking it does not affect Google Search rankings. But some site owners block it expecting to opt out of AI Overviews \u2014 that doesn’t work, since AI Overviews uses the standard Googlebot crawl.<\/li>\n
  4. Blocking ChatGPT-User<\/code> while allowing GPTBot<\/code><\/strong>, which is exactly backwards for brands that want visibility.<\/li>\n<\/ol>\n
    If you’re already invisible in AI answers, robots.txt is the first thing to check. Our guide on why sites fail to get cited in AI search<\/a> walks through the other six causes, from thin entity signals to unstructured content.<\/p>\n
    robots.txt vs. llms.txt: What’s the Difference?<\/h2>\n
    Robots.txt controls access; llms.txt guides interpretation.<\/strong> They are complements, not substitutes:<\/p>\n