{"id":801,"date":"2026-06-29T03:56:52","date_gmt":"2026-06-29T03:56:52","guid":{"rendered":"https:\/\/maxaeo.ai\/blog\/block-or-allow-ai-crawlers\/"},"modified":"2026-06-29T03:56:52","modified_gmt":"2026-06-29T03:56:52","slug":"block-or-allow-ai-crawlers","status":"publish","type":"post","link":"https:\/\/maxaeo.ai\/blog\/block-or-allow-ai-crawlers\/","title":{"rendered":"Block or Allow AI Crawlers? A Per-Bot Decision Guide"},"content":{"rendered":"

Whether to block or allow AI crawlers<\/strong> is now a real line item in technical SEO, and a single site-wide rule will cost you. GPTBot, ClaudeBot, PerplexityBot, and Google-Extended do different jobs, and blocking each one carries a different price. Some feed model training you may not care about. Others decide whether your brand appears when a buyer asks ChatGPT or Perplexity for a shortlist. This guide replaces blanket advice with a per-bot framework: what each crawler powers, what you lose by blocking it, and how to confirm your robots.txt does what you think it does.<\/p>\n

Most articles hand you a copy-paste block list and call it done. The real decision isn't block everything<\/em> or allow everything<\/em>\u2014it's matching each bot to the visibility you'd lose and the control you'd gain.<\/p>\n

Block or allow AI crawlers: the short answer<\/h2>\n

Allow the bots that put you in AI answers; treat training bots as a separate, lower-stakes decision.<\/strong> Search and retrieval crawlers\u2014OAI-SearchBot, PerplexityBot, Claude-SearchBot\u2014are what get you cited when someone asks an assistant for a recommendation. Block those and you remove yourself from the results.<\/p>\n

Training crawlers\u2014GPTBot, ClaudeBot, Google-Extended\u2014are an optional choice. Block them only for a specific reason: licensing use, genuinely proprietary content, or a brand-safety policy. Whatever you do, don't block the wrong bot by accident\u2014the most common self-inflicted wound here is blocking your own visibility while believing you're protecting your data.<\/p>\n

\"Decision<\/figure>\n

What's actually at stake: visibility versus control<\/h2>\n

Two different decisions hide inside "should I block AI crawlers," and conflating them produces most bad robots.txt files.<\/p>\n

Decision one is about training.<\/strong> Bots like GPTBot and ClaudeBot collect public pages that may<\/em> train future models. Blocking them affects whether your content shapes tomorrow's model weights. It does little to your visibility today\u2014and since your content likely already sits in past training sets and third-party copies, the control you gain is partial at best.<\/p>\n

Decision two is about live answers.<\/strong> Search and retrieval bots index or fetch your pages to answer questions right now<\/em>. Block these and you vanish when ChatGPT runs a search or Perplexity builds a citation list. This is the reversible-but-expensive lever: the cost lands immediately, every time someone asks.<\/p>\n

The four bots, decoded: training, search, and user fetches<\/h2>\n

AI crawlers are automated bots that fetch web pages for AI systems<\/strong>\u2014to train models, build search indexes, or pull live answers for assistants like ChatGPT, Claude, and Perplexity. Before you decide anything, you need the map. Each operator now runs multiple<\/em> bots with distinct user agents and distinct robots.txt behavior\u2014so "block GPTBot" and "block OpenAI" are not the same instruction.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n
User agent<\/th>\nOperator<\/th>\nJob<\/th>\nObeys robots.txt?<\/th>\nWhat blocking costs you<\/th>\n<\/tr>\n<\/thead>\n
GPTBot<\/strong><\/td>\nOpenAI<\/td>\nTrain foundation models<\/td>\nYes<\/td>\nFuture training inclusion<\/td>\n<\/tr>\n
OAI-SearchBot<\/strong><\/td>\nOpenAI<\/td>\nIndex for ChatGPT search<\/td>\nYes<\/td>\nYour spot in ChatGPT search answers<\/td>\n<\/tr>\n
ChatGPT-User<\/strong><\/td>\nOpenAI<\/td>\nLive fetch when a user asks<\/td>\nNo (user-initiated)<\/td>\nLittle\u2014robots.txt may not apply<\/td>\n<\/tr>\n
ClaudeBot<\/strong><\/td>\nAnthropic<\/td>\nTrain Claude models<\/td>\nYes<\/td>\nFuture training inclusion<\/td>\n<\/tr>\n
Claude-SearchBot<\/strong><\/td>\nAnthropic<\/td>\nIndex for Claude search<\/td>\nYes<\/td>\nVisibility in Claude search answers<\/td>\n<\/tr>\n
Claude-User<\/strong><\/td>\nAnthropic<\/td>\nLive fetch for a user query<\/td>\nYes<\/td>\nReal-time answers that cite you<\/td>\n<\/tr>\n
PerplexityBot<\/strong><\/td>\nPerplexity<\/td>\nBuild the answer index<\/td>\nYes (states it does)<\/td>\nYour place in Perplexity citations<\/td>\n<\/tr>\n
Perplexity-User<\/strong><\/td>\nPerplexity<\/td>\nLive user-triggered fetch<\/td>\nMay ignore robots.txt<\/td>\nLittle you can reliably block<\/td>\n<\/tr>\n
Google-Extended<\/strong><\/td>\nGoogle<\/td>\nToken: Gemini training\/grounding<\/td>\nToken only<\/td>\nGemini training use\u2014not<\/strong> Search or AI Overviews<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The pattern that matters: every operator now separates training<\/strong>, search indexing<\/strong>, and user-initiated fetches<\/strong>. That separation is your lever. Use it surgically.<\/p>\n

GPTBot, OAI-SearchBot, and ChatGPT-User: OpenAI's three controls<\/h2>\n

OpenAI runs three crawlers, and they are not interchangeable. GPTBot<\/strong> trains models. OAI-SearchBot<\/strong> indexes the web so ChatGPT can surface pages in search. ChatGPT-User<\/strong> fetches a page live when a person asks ChatGPT to look something up.<\/p>\n

The practical upshot: blocking GPTBot keeps your pages out of training but does nothing<\/strong> to your presence in ChatGPT's search answers\u2014that's a different bot. OpenAI's crawler documentation<\/a> recommends allowing OAI-SearchBot and notes that because ChatGPT-User actions are user-initiated, "robots.txt rules may not apply." So if you want to get recommended by ChatGPT<\/a> while still opting out of training, the move is simple: disallow GPTBot, allow OAI-SearchBot.<\/strong> The single most expensive mistake is blocking OAI-SearchBot by accident\u2014a direct visibility loss with no upside.<\/p>\n

ClaudeBot, Claude-SearchBot, and Claude-User: Anthropic's split<\/h2>\n

Anthropic mirrors the same three-way split, with one notable difference. ClaudeBot<\/strong> trains models; Claude-SearchBot<\/strong> indexes pages to improve Claude's search answers; Claude-User<\/strong> fetches a page when a user's question requires it.<\/p>\n

Anthropic states that all of its bots respect standard robots.txt directives\u2014including Disallow<\/code> and the non-standard Crawl-delay<\/code>\u2014and that you can control each independently. That's the meaningful contrast with OpenAI: where ChatGPT-User may ignore robots.txt, Anthropic's crawler documentation<\/a> says Claude-User honors it. So for Claude, your robots.txt is a more reliable instrument across all three bots. The decision framework is identical: keep Claude-SearchBot and Claude-User open<\/strong> to stay in live answers, and decide ClaudeBot on its own merits.<\/strong><\/p>\n

PerplexityBot and Perplexity-User: when robots.txt isn't enough<\/h2>\n

Perplexity splits its crawling too: PerplexityBot<\/strong> builds the index behind every answer, and Perplexity-User<\/strong> fetches pages in real time when a person asks a current question. Per Perplexity's published crawler documentation, PerplexityBot won't index the full content of a site that disallows it\u2014though it may still retain the domain, headline, and a brief factual summary.<\/p>\n

Here's why Perplexity needs its own section. In August 2025, Cloudflare reported<\/a> that Perplexity used undeclared crawlers\u2014rotating user agents and source networks, sometimes impersonating a Chrome-on-macOS browser\u2014to reach content on sites that had explicitly blocked its declared bots, generating an estimated 3\u20136 million requests a day before Cloudflare delisted it as a Verified Bot. Perplexity disputed the framing, arguing Perplexity-User acts only on live user requests.<\/p>\n

The lesson isn't "Perplexity is the villain." It's structural: a Disallow<\/code> line is a request, not a wall.<\/strong> If blocking genuinely matters for a given bot, robots.txt alone won't prove it worked. (If your problem is the opposite\u2014Perplexity citing rivals instead of you\u2014the fix is content and citations, not crawler config, which we cover in why Perplexity cites competitors instead of you<\/a>.)<\/p>\n

Google-Extended: the most misunderstood toggle<\/h2>\n

Google-Extended is the bot people block for the wrong reason.<\/strong> It isn't a crawler at all\u2014it's a robots.txt control token. Google Search Central's documentation<\/a> is explicit: "Google-Extended doesn't have a separate HTTP request user agent string," and it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."<\/p>\n

So what does it do? It governs whether content Google already crawls may be used to train Gemini and ground generative-AI products. What it does not<\/strong> do is remove you from AI Overviews<\/strong>\u2014those are part of Google Search and draw on the live Search index that Googlebot builds, not on the Google-Extended training pipeline. Blocking Google-Extended to "get out of AI Overviews" is a misfire; it changes nothing there. If you're worried about showing up in Google's AI answers at all, that's a visibility problem, not a crawler one\u2014it starts with being citable, not with blocking a token.<\/p>\n

Beyond the big four: other AI crawlers worth knowing<\/h2>\n

The four operators above dominate the decision, but several more bots show up in real server logs\u2014and a few follow the same training-versus-search logic.<\/p>\n\n\n\n\n\n\n\n\n\n
User agent<\/th>\nOperator<\/th>\nJob<\/th>\nWhat to know<\/th>\n<\/tr>\n<\/thead>\n
CCBot<\/strong><\/td>\nCommon Crawl<\/td>\nBuilds the open crawl dataset many AI models train on<\/td>\nObeys robots.txt; blocking removes you from a dataset reused across many models<\/td>\n<\/tr>\n
Bytespider<\/strong><\/td>\nByteDance (TikTok)<\/td>\nTrains ByteDance's models<\/td>\nReported to crawl aggressively and disregard robots.txt at times\u2014verify with logs<\/td>\n<\/tr>\n
Amazonbot<\/strong><\/td>\nAmazon<\/td>\nFeeds Alexa and Amazon's AI answers<\/td>\nObeys robots.txt<\/td>\n<\/tr>\n
Meta-ExternalAgent<\/strong><\/td>\nMeta<\/td>\nTrains Meta AI and Llama<\/td>\nObeys robots.txt<\/td>\n<\/tr>\n
Applebot-Extended<\/strong><\/td>\nApple<\/td>\nOpt-out token for Apple Intelligence training<\/td>\nToken only\u2014like Google-Extended, it doesn't change Siri or Spotlight indexing<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The same three-question test applies to every one of them: is it search\/retrieval (allow\u2014it feeds answers), a user-triggered fetch (you can't reliably block it anyway), or training (decide on the merits)?<\/p>\n

The per-bot decision framework<\/h2>\n

Now the part the ranking pages skip: a framework instead of a verdict. The goal isn't crawler control for its own sake\u2014it's answer engine optimization<\/a>, making sure that when someone asks an assistant for options, your brand is on the list. Score each bot against one question: does allowing it help me get recommended, or does blocking it protect something I genuinely need to protect?<\/em><\/p>\n\n\n\n\n\n\n\n
Bot group<\/th>\nDefault<\/th>\nBlock only if<\/th>\n<\/tr>\n<\/thead>\n
Search \/ retrieval<\/strong> (OAI-SearchBot, PerplexityBot, Claude-SearchBot)<\/td>\nAllow<\/strong><\/td>\nAlmost never\u2014blocking is a direct AI visibility loss<\/td>\n<\/tr>\n
User fetches<\/strong> (ChatGPT-User, Claude-User, Perplexity-User)<\/td>\nAllow<\/strong><\/td>\nYou can't reliably enforce a block anyway; the signal is symbolic<\/td>\n<\/tr>\n
Training<\/strong> (GPTBot, ClaudeBot, Google-Extended, CCBot)<\/td>\nDecide deliberately<\/strong><\/td>\nYou have licensing use, paywalled or proprietary content, or a legal\/brand-safety mandate<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Then layer your role on top, because the right answer changes with your business model:<\/p>\n