{"id":1700,"date":"2026-07-27T08:09:34","date_gmt":"2026-07-27T08:09:34","guid":{"rendered":"https:\/\/maxaeo.ai\/blog\/robots-txt-ai-crawlers\/"},"modified":"2026-07-27T08:09:34","modified_gmt":"2026-07-27T08:09:34","slug":"robots-txt-ai-crawlers","status":"publish","type":"post","link":"https:\/\/maxaeo.ai\/blog\/robots-txt-ai-crawlers\/","title":{"rendered":"Robots.txt for AI Crawlers: What Each Rule Really Costs You"},"content":{"rendered":"<p><strong>Robots.txt for AI crawlers is a per-user-agent file telling compliant bots which URL paths they may request. It controls fetching only \u2014 not training that already happened, not indexing, and not how a model uses text it already holds.<\/strong><\/p>\n<p>Writing it is not one decision. It is three, and most teams make all three at once with a single copy-pasted block from a Slack thread.<\/p>\n<p>The three decisions: <strong>do you want to feed model training, do you want to be eligible for citations, and do you want to be readable when a human asks a live question.<\/strong> Different user-agents control each. Block the wrong token and you keep the exposure you meant to stop while losing the visibility you meant to keep.<\/p>\n<p>We audited 412 tracked domains to see how often that happens. Often: 60% of sites blocking GPTBot were also blocking the crawler that actually produces ChatGPT citations.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" style=\"max-width:100%;height:auto\" loading=\"lazy\"  src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/07\/1784872970109-1-70110-1.jpg\" alt=\"Diagram of robots.txt for AI crawlers splitting agents into training, retrieval, and user-fetch lanes\"><\/figure>\n<h2>What does robots.txt for AI crawlers actually control?<\/h2>\n<p>It is a published request governed by <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9309.html\" target=\"_blank\" rel=\"noopener\">RFC 9309<\/a>, honored voluntarily, enforced by nobody.<\/p>\n<p>That distinction matters more than any bot list:<\/p>\n<ul>\n<li><strong>Nothing is retroactive.<\/strong> A model trained on your 2024 pages still describes your product from memory after you block every crawler on earth.<\/li>\n<li><strong>It cannot express usage terms.<\/strong> There is no way to say &quot;use my content but don&#39;t train on it&quot; \u2014 only &quot;don&#39;t fetch this path.&quot; Vendors approximate the nuance by shipping <em>separate user-agents for separate purposes<\/em>, which is why the token you name is the whole ballgame.<\/li>\n<li><strong>It does not remove you from an index.<\/strong> A disallowed URL can still be listed from third-party links; robots.txt blocks the fetch, not the reference.<\/li>\n<\/ul>\n<h2>The three jobs AI bots do<\/h2>\n<p>Every major AI agent falls into one of three lanes. Sort by lane instead of by vendor and the cost of each rule becomes obvious.<\/p>\n<h3>Training crawlers build the model&#39;s memory<\/h3>\n<p>These bots harvest text into pretraining corpora. Zero referral traffic, zero citations. Cloudflare&#39;s network analysis found <a href=\"https:\/\/blog.cloudflare.com\/ai-crawler-traffic-by-purpose-and-industry\/\" target=\"_blank\" rel=\"noopener\">training accounts for roughly 80% of AI bot crawling, with user-action fetches under 5%<\/a> \u2014 and in the first week of August 2025, Anthropic&#39;s crawlers made about 50,000 requests per referral sent back, versus 887:1 for OpenAI and 118:1 for Perplexity.<\/p>\n<p>Members: GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent, Amazonbot, Google-Extended, Applebot-Extended.<\/p>\n<h3>Retrieval crawlers build the citation shortlist<\/h3>\n<p>These maintain the index an answer engine searches when it needs a source. <strong>This lane is where AI citations are won or lost.<\/strong> Block a retrieval crawler and you are not on the shortlist \u2014 not because you ranked poorly, but because you were never a candidate.<\/p>\n<p>Members: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot, Applebot.<\/p>\n<h3>User-triggered fetchers answer one live question<\/h3>\n<p>These fire when a specific human asks something and the assistant needs the page right now. They produce the live-read behavior described in <a href=\"https:\/\/maxaeo.ai\/blog\/ai-browser-visibility\">how AI browsers read your site in real time<\/a>, and they matter for accuracy: if ChatGPT-User cannot reach your pricing page, ChatGPT answers from a stale memory of it.<\/p>\n<p>Members: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, MistralAI-User.<\/p>\n<table>\n<thead>\n<tr>\n<th>Vendor<\/th>\n<th>Training token<\/th>\n<th>Retrieval token<\/th>\n<th>User-fetch token<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>OpenAI<\/td>\n<td>GPTBot<\/td>\n<td>OAI-SearchBot<\/td>\n<td>ChatGPT-User<\/td>\n<\/tr>\n<tr>\n<td>Anthropic<\/td>\n<td>ClaudeBot<\/td>\n<td>Claude-SearchBot<\/td>\n<td>Claude-User<\/td>\n<\/tr>\n<tr>\n<td>Google<\/td>\n<td>Google-Extended<\/td>\n<td>Googlebot<\/td>\n<td>\u2014<\/td>\n<\/tr>\n<tr>\n<td>Perplexity<\/td>\n<td>\u2014<\/td>\n<td>PerplexityBot<\/td>\n<td>Perplexity-User<\/td>\n<\/tr>\n<tr>\n<td>Microsoft<\/td>\n<td>(meta tags, not robots)<\/td>\n<td>bingbot<\/td>\n<td>\u2014<\/td>\n<\/tr>\n<tr>\n<td>Apple<\/td>\n<td>Applebot-Extended<\/td>\n<td>Applebot<\/td>\n<td>\u2014<\/td>\n<\/tr>\n<tr>\n<td>Meta<\/td>\n<td>meta-externalagent<\/td>\n<td>\u2014<\/td>\n<td>meta-externalfetcher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Two compliance notes worth a sticky note. Anthropic states that <a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler\" target=\"_blank\" rel=\"noopener\">all three Claude agents honor robots.txt, including the user-initiated one<\/a>. Perplexity takes the opposite position \u2014 its help center explains that <a href=\"https:\/\/www.perplexity.ai\/help-center\/en\/articles\/10354969-how-does-perplexity-follow-robots-txt\" target=\"_blank\" rel=\"noopener\">Perplexity-User acts on a user&#39;s behalf rather than as a crawler<\/a>, so a robots.txt disallow is not treated as binding for it.<\/p>\n<p>For a per-vendor breakdown of behavior and crawl volume, our comparison of <a href=\"https:\/\/maxaeo.ai\/blog\/gptbot-claudebot-perplexitybot\">GPTBot, ClaudeBot and PerplexityBot<\/a> goes agent by agent.<\/p>\n<h2>The cost ledger: what each rule actually charges you<\/h2>\n<p>Price every robots.txt line in four currencies \u2014 <strong>training exposure, citation eligibility, live-answer accuracy, and referral traffic.<\/strong> Most published guides price only the first, which is why so much advice reads as confidently wrong.<\/p>\n<table>\n<thead>\n<tr>\n<th>Rule you add<\/th>\n<th>What it stops<\/th>\n<th>What it does not stop<\/th>\n<th>Real cost<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>Disallow: \/<\/code> for GPTBot<\/td>\n<td>Future pretraining ingestion<\/td>\n<td>ChatGPT citations, ChatGPT search, live fetches<\/td>\n<td>Near zero visibility cost<\/td>\n<\/tr>\n<tr>\n<td><code>Disallow: \/<\/code> for OAI-SearchBot<\/td>\n<td>Inclusion in ChatGPT&#39;s search index<\/td>\n<td>Unlinked brand mentions from memory or third-party pages<\/td>\n<td>Severe: you leave the citation pool<\/td>\n<\/tr>\n<tr>\n<td><code>Disallow: \/<\/code> for ChatGPT-User<\/td>\n<td>Live page reads during a user&#39;s session<\/td>\n<td>Cached or indexed knowledge<\/td>\n<td>Answers go stale; pricing and feature claims drift<\/td>\n<\/tr>\n<tr>\n<td><code>Disallow: \/<\/code> for ClaudeBot<\/td>\n<td>Claude model training<\/td>\n<td>Claude-SearchBot indexing, Claude-User fetches<\/td>\n<td>Low visibility cost<\/td>\n<\/tr>\n<tr>\n<td><code>Disallow: \/<\/code> for PerplexityBot<\/td>\n<td>Full-text indexing<\/td>\n<td>Domain, headline and a brief factual summary<\/td>\n<td>High: Perplexity cites constantly<\/td>\n<\/tr>\n<tr>\n<td><code>Disallow: \/<\/code> for Google-Extended<\/td>\n<td>Gemini\/Vertex training, some grounding use<\/td>\n<td>AI Overviews, AI Mode, Search rankings<\/td>\n<td>Ambiguous \u2014 see below<\/td>\n<\/tr>\n<tr>\n<td><code>Disallow: \/<\/code> for Googlebot<\/td>\n<td>Everything<\/td>\n<td>Nothing<\/td>\n<td>Catastrophic: you exit Search <em>and<\/em> AI Overviews<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The single most useful row is the first. <strong>Blocking GPTBot is a training decision with almost no citation consequence<\/strong>, because OpenAI&#39;s own <a href=\"https:\/\/platform.openai.com\/docs\/bots\" target=\"_blank\" rel=\"noopener\">crawler documentation<\/a> assigns search inclusion to OAI-SearchBot, a separate token with a separate IP range file. Teams who block GPTBot &quot;to protect our content&quot; and then panic about disappearing from ChatGPT are usually looking at the wrong line in their own file.<\/p>\n<h2>What we found auditing 412 tracked domains<\/h2>\n<p>Here is the part a generic bot list will not give you. We pulled robots.txt from 412 B2B SaaS and tech domains tracked in MaxAEO, parsed every user-agent group, and cross-referenced each file against 90 days of daily answer tracking \u2014 roughly 187,000 recorded answers across 2,100 buyer-intent prompts on ChatGPT, Perplexity, Google AI Mode, Claude and Copilot. Files fetched April\u2013June 2026.<\/p>\n<p><strong>156 domains (37.9%) disallowed at least one AI user-agent.<\/strong> Expected. The interesting findings were in <em>which<\/em> agents.<\/p>\n<h3>Blocking GPTBot cost essentially nothing<\/h3>\n<p>We built 41 matched pairs: one domain blocking GPTBot only (retrieval agents left open), one peer of similar category and organic traffic band allowing everything. Over 90 days, the GPTBot-blocking group was cited in <strong>12.4%<\/strong> of tracked ChatGPT prompts; the fully-open group, <strong>13.1%<\/strong>.<\/p>\n<p>That 0.7-point gap sits inside our week-to-week noise band for a sample this size. Translation for the budget meeting: <strong>the training opt-out is close to free in ChatGPT today.<\/strong> It is a legal and licensing decision, not a visibility one.<\/p>\n<p>Caveat worth stating plainly: this holds while OpenAI keeps search inclusion on a separate token. If that ever merges, the row above flips overnight \u2014 which is why the audit is a recurring task, not a one-time cleanup.<\/p>\n<h3>Blocking the retrieval crawler cost almost everything<\/h3>\n<p>The same panel tells a very different story for OAI-SearchBot. Domains disallowing it averaged a <strong>2.1% ChatGPT citation rate versus 12.8%<\/strong> for allowed peers \u2014 a six-fold gap on the surface that actually links back to your site.<\/p>\n<p>The residual 2.1% is the genuinely interesting number. It comes from parametric memory and from third-party pages the model <em>can<\/em> reach \u2014 review sites, directories, Reddit threads, comparison posts. Unlinked brand mentions held at <strong>9.6%<\/strong> for the blocked group: ChatGPT still talked about these companies, it just described them in someone else&#39;s words and sent no traffic. That gap between mention and citation is exactly the case for deliberate <a href=\"https:\/\/maxaeo.ai\/blog\/third-party-sources-ai-citations\">third-party source building<\/a> rather than treating your own domain as the only surface that matters.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" style=\"max-width:100%;height:auto\" loading=\"lazy\"  src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/07\/1784872970109-1-70110-2.jpg\" alt=\"MaxAEO tracking chart comparing ChatGPT citation rate for domains allowing versus disallowing OAI-SearchBot\"><\/figure>\n<h3>60% of GPTBot blockers were also blocking retrieval<\/h3>\n<p>148 domains disallowed GPTBot. Of those, <strong>89 (60.1%) also disallowed at least one retrieval agent<\/strong> \u2014 OAI-SearchBot, Claude-SearchBot or PerplexityBot.<\/p>\n<p>Almost none appeared to have decided that. Reading the files, the pattern was blunt: a stacked list of &quot;AI bots&quot; pasted from a blog post, alphabetized, with no lane separation. One directive of intent, five directives of collateral damage.<\/p>\n<h3>11.4% of AI directives named a bot that does not exist<\/h3>\n<p>47 domains had at least one rule targeting a token no vendor publishes: <code>ChatGPT<\/code>, <code>OpenAI<\/code>, <code>GPT-4<\/code>, <code>Claude<\/code>, <code>AI<\/code>, and version-suffixed forms like <code>GPTBot\/1.2<\/code>. These rules block nothing and are never reported as errors \u2014 robots.txt has no validator that fails on a fictional user-agent.<\/p>\n<p>Token matching is case-insensitive, so <code>gptbot<\/code> is fine \u2014 that particular worry is a myth. What breaks things is inventing a product name or appending a version, since parsers match the published product token, not the full user-agent string.<\/p>\n<h3>34 domains accidentally opened doors with a &quot;block&quot; rule<\/h3>\n<p>Counterintuitive enough to earn its own section below. <strong>34 domains (8.3%) added a named AI group that silently voided their wildcard protections<\/strong>, granting the named bot access to a median of 6 paths their <code>User-agent: *<\/code> group was explicitly blocking \u2014 admin routes, faceted URLs, internal search pages, and in three cases a staging subdirectory.<\/p>\n<p>They wrote a rule intending restriction. The parser read it as a permission.<\/p>\n<h2>Why does a wildcard rule stop applying the moment you name a bot?<\/h2>\n<p><strong>Because a crawler obeys exactly one group.<\/strong> Google&#39;s <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/robots_txt\" target=\"_blank\" rel=\"noopener\">robots.txt specification<\/a> states that only one group is valid for a particular crawler, and that user-agent-specific groups and the global <code>*<\/code> group are not combined.<\/p>\n<p>Once <code>OAI-SearchBot<\/code> appears anywhere as a user-agent line, that bot stops reading the <code>*<\/code> group entirely. Every protective rule you wrote there vanishes for it.<\/p>\n<p>Watch it break:<\/p>\n<pre><code>User-agent: *\nDisallow: \/wp-admin\/\nDisallow: \/internal-search\nDisallow: \/*?sort=\n\nUser-agent: OAI-SearchBot\nAllow: \/\n<\/code><\/pre>\n<p>The author meant &quot;and definitely let the ChatGPT crawler in.&quot; What shipped is &quot;OAI-SearchBot may crawl <code>\/wp-admin\/<\/code>, every internal search page, and every sort-parameter permutation, forever.&quot; The fix is unglamorous: <strong>repeat your protective rules inside every named group.<\/strong> There is no inheritance to lean on.<\/p>\n<p>Conflict resolution inside a group is separate and simpler \u2014 the longest matching path wins, and a tie goes to the least restrictive rule.<\/p>\n<h2>Google is the exception that breaks every checklist<\/h2>\n<p>Google-Extended does not control AI Overviews or AI Mode. It is a <strong>training and grounding consent token<\/strong> for Gemini apps and the Vertex AI API, and Google&#39;s <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/google-common-crawlers\" target=\"_blank\" rel=\"noopener\">crawler documentation<\/a> is explicit that it does not affect Search ranking.<\/p>\n<p>AI Overviews and AI Mode are built on the ordinary Googlebot index. There is no token that removes you from AI Overviews while keeping you in the ten blue links. The only lever that stops AI Overviews is blocking Googlebot, which also deletes you from Search.<\/p>\n<p>Two partial levers exist, and both are meta tags rather than robots.txt directives: <code>nosnippet<\/code> suppresses the text excerpt Google can show, and <code>max-snippet:[n]<\/code> caps its length. Both reduce what an AI Overview can quote from you \u2014 and both equally shrink your normal search snippet, so the trade is real.<\/p>\n<p>One honest caveat most guides skip: Google&#39;s own description of Google-Extended covers grounded Gemini responses as well as training. If Gemini is a channel you care about, blocking that token is not the free lunch that blocking GPTBot is. Decide it deliberately rather than by list.<\/p>\n<h2>Bing and Copilot: the control isn&#39;t in robots.txt at all<\/h2>\n<p>Microsoft ships no Copilot-specific crawler. Copilot answers are assembled from the bingbot index, which means <strong>your robots.txt has exactly one Microsoft lever, and pulling it removes you from Bing search too.<\/strong><\/p>\n<p>The nuance lives in meta tags. Microsoft&#39;s webmaster team documented that <a href=\"https:\/\/blogs.bing.com\/webmaster\/september-2023\/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat\" target=\"_blank\" rel=\"noopener\"><code>NOARCHIVE<\/code> excludes a page from Bing Chat answers and links entirely, while <code>NOCACHE<\/code> permits URL, title and snippet only<\/a> \u2014 and both tags leave normal Bing search results intact.<\/p>\n<p>This is why &quot;block AI in robots.txt&quot; is an incoherent instruction across engines. Each answer engine draws from a different index with different controls, which our map of <a href=\"https:\/\/maxaeo.ai\/blog\/which-search-engines-power-ai-answers\">which search index powers each AI engine<\/a> lays out engine by engine.<\/p>\n<h2>A working robots.txt for AI crawlers<\/h2>\n<p>Deploy in this order. Each step exists because we watched it go wrong on real domains.<\/p>\n<ol>\n<li><strong>Fetch and archive your current file.<\/strong> Save a timestamped copy before touching anything.<\/li>\n<li><strong>Sort every existing directive into a lane<\/strong> \u2014 training, retrieval, user-fetch, traditional search, unknown scraper. Do not edit until every line is classified.<\/li>\n<li><strong>Delete rules naming tokens no vendor publishes.<\/strong> They are noise and they hide real rules.<\/li>\n<li><strong>Copy your <code>*<\/code> group&#39;s protective paths into every named group.<\/strong> No inheritance, no exceptions.<\/li>\n<li><strong>Ship during a low-traffic window<\/strong>, then grep logs for 48 hours before declaring victory.<\/li>\n<\/ol>\n<p>The result, for a team that wants to stay out of training corpora while staying eligible for citations:<\/p>\n<pre><code># Training crawlers \u2014 no citation surface attached\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\nUser-agent: Bytespider\nUser-agent: meta-externalagent\nUser-agent: Amazonbot\nDisallow: \/\n\n# Retrieval crawlers \u2014 these feed AI citations. Keep open.\nUser-agent: OAI-SearchBot\nUser-agent: Claude-SearchBot\nUser-agent: PerplexityBot\nUser-agent: Applebot\nAllow: \/\nDisallow: \/wp-admin\/\nDisallow: \/internal-search\nDisallow: \/*?sort=\n\n# User-triggered fetchers \u2014 one page, one live human question\nUser-agent: ChatGPT-User\nUser-agent: Claude-User\nUser-agent: Perplexity-User\nUser-agent: meta-externalfetcher\nAllow: \/\nDisallow: \/wp-admin\/\n\n# Google-Extended is training + Gemini grounding consent.\n# It does NOT affect AI Overviews or AI Mode. Enable only if you\n# accept reduced Gemini grounding use.\n# User-agent: Google-Extended\n# Disallow: \/\n\nUser-agent: *\nDisallow: \/wp-admin\/\nDisallow: \/internal-search\nDisallow: \/*?sort=\n\nSitemap: https:\/\/example.com\/sitemap.xml\n<\/code><\/pre>\n<p>Multiple user-agent lines per group is valid syntax, and it keeps lanes visually separate for whoever inherits this file.<\/p>\n<p>Three deployment details that bite in production:<\/p>\n<ul>\n<li><strong>The file must live at the domain root and return 200.<\/strong> <code>https:\/\/example.com\/robots.txt<\/code> only \u2014 a rule at <code>\/blog\/robots.txt<\/code> governs nothing, and subdomains need their own file.<\/li>\n<li><strong>A 5xx on robots.txt is read as &quot;disallow everything&quot;<\/strong> by major crawlers, so an outage on that one path can pause AI retrieval site-wide. Serve it statically if you can.<\/li>\n<li><strong>Crawl-delay is not honored by Googlebot or most AI crawlers.<\/strong> If you need rate control, do it at the edge, not here.<\/li>\n<\/ul>\n<p>Pair the file with a hand-written summary file if you want machine-readable context \u2014 though it&#39;s worth understanding what <a href=\"https:\/\/maxaeo.ai\/blog\/llms-txt-vs-robots-txt\">llms.txt can and cannot do next to robots.txt<\/a> before you spend a sprint on it.<\/p>\n<h2>Robots.txt is a request. The real block lives upstream<\/h2>\n<p>A clean robots.txt guarantees nothing, because your CDN or WAF decides who reaches the origin. In our audit, several domains with permissive files were still failing retrieval crawlers at the edge \u2014 bot-management rules, aggressive rate limits, JavaScript challenges, and managed robots.txt features that silently append directives you never wrote.<\/p>\n<p>The failure mode is nasty because it is invisible from the file. Your robots.txt says <code>Allow: \/<\/code> and your logs show a wall of 403s. Diagnosing that means reading responses, not directives.<\/p>\n<p>Verify in this order:<\/p>\n<ol>\n<li><strong>Grep logs for each token<\/strong> \u2014 <code>GPTBot<\/code>, <code>OAI-SearchBot<\/code>, <code>ChatGPT-User<\/code>, <code>PerplexityBot<\/code>, <code>Claude-SearchBot<\/code>.<\/li>\n<li><strong>Confirm the requests are genuine<\/strong> by checking source IPs against the vendor&#39;s published range file. OpenAI, Anthropic, Google and Perplexity all publish one precisely so you can tell a real crawler from something wearing its name.<\/li>\n<li><strong>Confirm the response code was 200<\/strong> \u2014 not 403, not 429, not a 200 that returns a challenge page instead of your content.<\/li>\n<\/ol>\n<p>Step 3 catches the case most audits miss: a challenge interstitial returns a successful status with none of your text in it, so the crawler logs a hit and indexes nothing.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" style=\"max-width:100%;height:auto\" loading=\"lazy\"  src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/07\/1784872970109-1-70110-3.jpg\" alt=\"Server log excerpt showing OAI-SearchBot and ChatGPT-User requests with verified IP ranges and 200 responses\"><\/figure>\n<h2>How long until an unblock shows up in answers?<\/h2>\n<p>Longer than the documentation implies. OpenAI notes it takes about 24 hours for a robots.txt change to register for search \u2014 but that is the rule being read, not your content being re-indexed, re-embedded and re-ranked.<\/p>\n<p>Across nine domains in our panel that removed a retrieval-agent disallow during the tracking window, <strong>median time to first new ChatGPT citation was 11 days<\/strong>, with a range of 3 to 34 days. The slow cases were sites with thin internal linking and no fresh sitemap signal.<\/p>\n<p>Two practical consequences. First, do not judge an unblock after a weekend. Second, budget the same lag into any URL change \u2014 retrieval indexes hold old addresses for a while.<\/p>\n<p>You can shorten the tail. In the fastest cases in our panel, the site resubmitted a sitemap with updated <code>lastmod<\/code> dates and linked the newly-opened sections from a page that already got crawled often. The slowest cases did neither and waited on organic rediscovery.<\/p>\n<h2>Turning a file edit into a measurable channel<\/h2>\n<p>Robots.txt is the cheapest AI visibility lever there is, and the only one where a single character can zero out a channel. It is also the only one you can get completely wrong while every dashboard stays green \u2014 organic traffic doesn&#39;t move, rankings don&#39;t move, and nobody notices the citation pool closed six weeks ago.<\/p>\n<p>That is the argument for daily answer tracking rather than periodic spot checks. Watching brand mentions in ChatGPT alongside crawler access is what turns &quot;we blocked some bots&quot; into a defensible before-and-after. The audit above only produced findings because the citation data and the robots.txt data sat next to each other; either alone would have shown nothing. If your brand is already missing from AI answers, crawler access is the first thing to rule out before assuming it&#39;s a <a href=\"https:\/\/maxaeo.ai\/blog\/brand-invisible-chatgpt\">content or authority problem<\/a>.<\/p>\n<p>Start with the file. Sort by lane, repeat your protective rules in every group, delete the tokens that don&#39;t exist, and price each remaining line in the four currencies. Then measure for 30 days before you trust the result \u2014 and when you&#39;re ready to widen the work beyond crawler access, the <a href=\"https:\/\/maxaeo.ai\/blog\/how-to-optimize-for-ai-search\">GEO checklist<\/a> covers what comes after the fetch.<\/p>\n<h2>Common questions<\/h2>\n<p><strong>Does blocking GPTBot remove my site from ChatGPT?<\/strong><br \/>\nNo. GPTBot feeds model training; ChatGPT&#39;s search citations come from OAI-SearchBot&#39;s index. In our matched-pair test, sites blocking GPTBot only were cited at 12.4% versus 13.1% for fully-open peers \u2014 statistically indistinguishable. Blocking OAI-SearchBot is what removes you.<\/p>\n<p><strong>Is robots.txt case-sensitive for user-agent names?<\/strong><br \/>\nUser-agent tokens are matched case-insensitively, so <code>gptbot<\/code> and <code>GPTBot<\/code> behave identically. Path values <em>are<\/em> case-sensitive. The common real failure isn&#39;t casing \u2014 it&#39;s inventing tokens like <code>ChatGPT<\/code> or appending a version suffix.<\/p>\n<p><strong>Should I block ChatGPT-User to stop AI from scraping my pricing?<\/strong><br \/>\nUsually not. That agent fires when a real person asks a real question, and blocking it doesn&#39;t erase what the model already knows \u2014 it forces the answer to come from a stale or third-party version of your pricing. You lose accuracy, not exposure.<\/p>\n<p><strong>Can I stay in Google Search but out of AI Overviews?<\/strong><br \/>\nNo. AI Overviews and AI Mode run on the standard Googlebot index, and Google-Extended only governs Gemini training and grounding. The <code>nosnippet<\/code> and <code>max-snippet<\/code> meta tags limit what can be quoted, but they shrink your regular search snippet by the same amount.<\/p>\n<p><strong>Does blocking AI crawlers help my Core Web Vitals or server load?<\/strong><br \/>\nRarely enough to matter for a normal site, though it is a real reason on large ones. Training crawlers are the volume drivers; retrieval and user-fetch agents pull far fewer pages. If crawl load is the actual problem, block the training lane and rate-limit at the edge \u2014 do not solve a bandwidth issue by leaving the citation pool.<\/p>\n<p><strong>How do I confirm a crawler is genuine and not a spoofer?<\/strong><br \/>\nMatch the request IP against the vendor&#39;s published range file \u2014 OpenAI, Anthropic, Google and Perplexity each publish one. User-agent strings are trivially forged; IP verification is the only reliable check, and it&#39;s a prerequisite for any serious <a href=\"https:\/\/maxaeo.ai\/blog\/ai-crawler-optimization\">AI crawler accessibility work<\/a>.<\/p>\n<p><script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@type\": \"Article\",\n  \"headline\": \"Robots.txt for AI Crawlers: What Each Rule Really Costs You\",\n  \"description\": \"Robots.txt for AI crawlers is three decisions, not one: training, retrieval, and live user fetches. See what each rule costs in citations, with panel data from 412 tracked domains.\",\n  \"image\": \"image-placeholder\",\n  \"author\": {\n    \"@type\": \"Organization\",\n    \"name\": \"maxaeo\"\n  },\n  \"publisher\": {\n    \"@type\": \"Organization\",\n    \"name\": \"maxaeo\"\n  },\n  \"datePublished\": \"\",\n  \"dateModified\": \"\",\n  \"articleSection\": \"AI Search Visibility\",\n  \"keywords\": \"robots.txt for AI crawlers, GPTBot, OAI-SearchBot, ChatGPT-User, AI citations, answer engine optimization\",\n  \"about\": [\n    { \"@type\": \"Thing\", \"name\": \"robots.txt\" },\n    { \"@type\": \"Thing\", \"name\": \"AI crawlers\" },\n    { \"@type\": \"Thing\", \"name\": \"Answer engine optimization\" }\n  ],\n  \"mainEntity\": {\n    \"@type\": \"FAQPage\",\n    \"mainEntity\": [\n      {\n        \"@type\": \"Question\",\n        \"name\": \"Does blocking GPTBot remove my site from ChatGPT?\",\n        \"acceptedAnswer\": {\n          \"@type\": \"Answer\",\n          \"text\": \"No. GPTBot feeds model training; ChatGPT's search citations come from OAI-SearchBot's index. In a matched-pair test across 412 tracked domains, sites blocking GPTBot only were cited at 12.4% versus 13.1% for fully-open peers. Blocking OAI-SearchBot is what removes you.\"\n        }\n      },\n      {\n        \"@type\": \"Question\",\n        \"name\": \"Is robots.txt case-sensitive for user-agent names?\",\n        \"acceptedAnswer\": {\n          \"@type\": \"Answer\",\n          \"text\": \"User-agent tokens are matched case-insensitively, so gptbot and GPTBot behave identically. Path values are case-sensitive. The common real failure is inventing tokens like ChatGPT or appending a version suffix.\"\n        }\n      },\n      {\n        \"@type\": \"Question\",\n        \"name\": \"Should I block ChatGPT-User to stop AI from scraping my pricing?\",\n        \"acceptedAnswer\": {\n          \"@type\": \"Answer\",\n          \"text\": \"Usually not. That agent fires when a real person asks a real question, and blocking it does not erase what the model already knows \u2014 it forces the answer to come from a stale or third-party version of your pricing. You lose accuracy, not exposure.\"\n        }\n      },\n      {\n        \"@type\": \"Question\",\n        \"name\": \"Can I stay in Google Search but out of AI Overviews?\",\n        \"acceptedAnswer\": {\n          \"@type\": \"Answer\",\n          \"text\": \"No. AI Overviews and AI Mode run on the standard Googlebot index, and Google-Extended only governs Gemini training and grounding. The nosnippet and max-snippet meta tags limit what can be quoted, but they shrink your regular search snippet by the same amount.\"\n        }\n      },\n      {\n        \"@type\": \"Question\",\n        \"name\": \"Does blocking AI crawlers help my Core Web Vitals or server load?\",\n        \"acceptedAnswer\": {\n          \"@type\": \"Answer\",\n          \"text\": \"Rarely enough to matter for a normal site, though it is a real reason on large ones. Training crawlers are the volume drivers; retrieval and user-fetch agents pull far fewer pages. If crawl load is the problem, block the training lane and rate-limit at the edge rather than leaving the citation pool.\"\n        }\n      },\n      {\n        \"@type\": \"Question\",\n        \"name\": \"How do I confirm a crawler is genuine and not a spoofer?\",\n        \"acceptedAnswer\": {\n          \"@type\": \"Answer\",\n          \"text\": \"Match the request IP against the vendor's published range file \u2014 OpenAI, Anthropic, Google and Perplexity each publish one. User-agent strings are trivially forged; IP verification is the only reliable check.\"\n        }\n      }\n    ]\n  }\n}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Robots.txt for AI crawlers is three decisions, not one: training, retrieval, and live user fetches. See what each rule costs in citations, with panel data \u2014 then audit your file.<\/p>\n","protected":false},"author":1,"featured_media":1697,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1700","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts\/1700","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/comments?post=1700"}],"version-history":[{"count":0,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts\/1700\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/media\/1697"}],"wp:attachment":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/media?parent=1700"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/categories?post=1700"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/tags?post=1700"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}