{"id":1153,"date":"2026-07-10T02:45:16","date_gmt":"2026-07-10T02:45:16","guid":{"rendered":"https:\/\/maxaeo.ai\/blog\/block-ai-crawlers\/"},"modified":"2026-07-10T02:45:16","modified_gmt":"2026-07-10T02:45:16","slug":"block-ai-crawlers","status":"publish","type":"post","link":"https:\/\/maxaeo.ai\/blog\/block-ai-crawlers\/","title":{"rendered":"How to Block AI Crawlers Without Losing AI Search Visibility"},"content":{"rendered":"

To block AI crawlers<\/strong> safely, do not start with a blanket Disallow: \/<\/code>. Start by separating training crawlers<\/strong>, AI search crawlers<\/strong>, and user-triggered agents<\/strong>. Then decide which page types should be discoverable, which should be excluded from model training, and which should be protected with real access controls.<\/p>\n

The practical rule for brand websites is simple: allow AI search access to public evidence pages you want cited; block training use where rights, privacy, or strategy require it; protect private content with authentication, not robots.txt.<\/strong><\/p>\n

\"Decision<\/figure>\n

What Does It Mean to Block AI Crawlers?<\/h2>\n

Blocking AI crawlers means restricting automated systems connected to AI products from fetching selected URLs or using fetched content for search, retrieval, summarization, training, or user-initiated agent actions.<\/strong> In practice, site owners combine robots.txt directives, bot verification, WAF rules, rate limits, noindex tags, and authentication.<\/p>\n

For SEO and AI visibility, the key distinction is what the bot is doing<\/strong>:<\/p>\n\n\n\n\n\n\n\n\n\n
AI crawler type<\/th>\nExamples<\/th>\nMain use<\/th>\nWhat blocking can do<\/th>\n<\/tr>\n<\/thead>\n
Training crawler<\/td>\nGPTBot<\/code>, ClaudeBot<\/code>, Google-Extended<\/code><\/td>\nCollect content that may be used for model training or related AI uses<\/td>\nReduces future training use by compliant operators<\/td>\n<\/tr>\n
AI search crawler<\/td>\nOAI-SearchBot<\/code>, Claude-SearchBot<\/code>, PerplexityBot<\/code><\/td>\nIndex or retrieve web content for AI search answers<\/td>\nCan reduce AI citations, answer inclusion, and source accuracy<\/td>\n<\/tr>\n
User-triggered fetcher<\/td>\nChatGPT-User<\/code>, Claude-User<\/code>, Perplexity-User<\/code><\/td>\nFetch a page because a person asked for it<\/td>\nCan affect real-time answer quality, but robots.txt may not apply consistently<\/td>\n<\/tr>\n
Traditional search crawler<\/td>\nGooglebot<\/code>, Bingbot<\/code><\/td>\nBuild search indexes used across search surfaces<\/td>\nBlocking can damage organic search and downstream AI discovery<\/td>\n<\/tr>\n
Unknown scraper or data collector<\/td>\nSpoofed or rotating user agents<\/td>\nPrice scraping, dataset collection, competitive intelligence<\/td>\nRequires WAF, IP verification, bot management, and rate limits<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

This is why the right question is not "Should we block AI?" It is: Which AI use case deserves access to which public pages, and what evidence do we have that the tradeoff is worth it?<\/strong><\/p>\n

The Short Answer: What Most Brand Sites Should Do<\/h2>\n

Most B2B SaaS, ecommerce, and publisher brands should use a selective policy:<\/p>\n

    \n
  1. Allow public search discovery.<\/strong> Keep Googlebot, Bingbot, and AI search crawlers able to access homepage, product, comparison, documentation, pricing, research summaries, and high-value blog pages.<\/li>\n
  2. Block training crawlers where rights or strategy require it.<\/strong> Use robots.txt tokens such as GPTBot<\/code>, ClaudeBot<\/code>, and Google-Extended<\/code> for content you do not license for model training.<\/li>\n
  3. Use authentication for private content.<\/strong> Customer portals, account pages, admin routes, staging sites, support tickets, and private docs should not be publicly reachable.<\/li>\n
  4. Use noindex<\/code> for pages that may be crawled but should not appear in search.<\/strong> Do not combine noindex<\/code> with robots.txt disallow if you need Google to see the noindex<\/code> directive.<\/li>\n
  5. Verify bots before allowing or blocking at the network layer.<\/strong> User-agent strings can be spoofed, so WAF rules should combine user agent, IP ranges, reverse DNS where available, and log review.<\/li>\n
  6. Measure AI visibility before and after.<\/strong> Track brand mentions, citations, source URLs, and answer accuracy in ChatGPT, Perplexity, Claude, Gemini, Copilot, Grok, and Google AI Overviews.<\/li>\n<\/ol>\n

    If your team needs a bot-by-bot companion to this page, use MaxAEO's Block or Allow AI Crawlers decision guide<\/a>.<\/p>\n

    Why Blocking AI Crawlers Is Not One Decision<\/h2>\n

    Robots.txt was standardized as the Robots Exclusion Protocol in RFC 9309<\/a>, but the standard itself says robots rules are not access authorization<\/strong>. Google also states that robots.txt is mainly for controlling crawler access and is not a way to keep a page out of Google Search<\/strong>; for that, Google recommends noindex<\/code> or password protection in its robots.txt guidance<\/a>.<\/p>\n

    That limitation matters more with AI crawlers because "AI access" now includes several different behaviors:<\/p>\n