{"id":3426,"date":"2026-10-29T03:15:03","date_gmt":"2026-10-29T03:15:03","guid":{"rendered":"https:\/\/maxaeo.ai\/blog\/unblock-oai-searchbot\/"},"modified":"2026-10-29T03:15:03","modified_gmt":"2026-10-29T03:15:03","slug":"unblock-oai-searchbot","status":"publish","type":"post","link":"https:\/\/maxaeo.ai\/blog\/unblock-oai-searchbot\/","title":{"rendered":"Unblock OAI-SearchBot AWS WAF Rule"},"content":{"rendered":"<p><em>By maxaeo.ai \uff5c Published 2026-10-29 \uff5c Updated 2026-10-29<\/em><\/p>\n<p>If your site returns <strong>403, 429, or a challenge page to OAI-SearchBot<\/strong>, the crawler may be blocked by AWS WAF, a managed bot rule, rate limiting, or an upstream CDN. To unblock OAI-SearchBot AWS WAF rule behavior safely, allow the crawler only after verifying both its user-agent and its source IP, then place the exception before blocking rules.<\/p>\n<figure class=\"wp-block-image size-large\" style=\"margin:1.5em 0;\"><img decoding=\"async\" src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/10\/backend-8635-1.jpg\" alt=\"AWS WAF rule flow for allowing OAI-SearchBot\" style=\"max-width:100%;height:auto;\"><\/figure>\n<h2>What is OAI-SearchBot, and why does AWS WAF block it?<\/h2>\n<p>OAI-SearchBot is OpenAI\u2019s crawler for discovering websites that may appear in ChatGPT search results. It is separate from GPTBot, which OpenAI describes as a crawler used for training-related purposes, and from ChatGPT-User, which represents user-triggered fetches. OpenAI recommends allowing OAI-SearchBot in <code>robots.txt<\/code> and allowing requests from its published IP ranges. (<a href=\"https:\/\/developers.openai.com\/api\/docs\/bots?orgid=145\" target=\"_blank\" rel=\"noopener\">developers.openai.com<\/a>)<\/p>\n<p>AWS WAF can still block the request even when <code>robots.txt<\/code> permits it. Common causes include:<\/p>\n<ul>\n<li>AWS Managed Rules detecting an unfamiliar automated client<\/li>\n<li>A custom rule blocking non-browser user-agents<\/li>\n<li>AWS WAF Bot Control labeling the request as a search or content-fetching bot<\/li>\n<li>A rate-based rule returning <code>429<\/code><\/li>\n<li>CAPTCHA, challenge, or JavaScript checks that a crawler cannot complete<\/li>\n<li>An allow rule placed below a terminating block rule<\/li>\n<\/ul>\n<p>The key distinction is important: <strong>robots.txt expresses crawl permission, while AWS WAF controls whether the HTTP request reaches your application<\/strong>.<\/p>\n<h2>Before changing AWS WAF, separate the four failure points<\/h2>\n<p>A reliable diagnosis checks the request path in order rather than adding a broad \u201callow OpenAI\u201d exception.<\/p>\n<div style=\"overflow-x:auto;\">\n<table style=\"width:100%;border-collapse:collapse;margin:1.5em 0;font-size:0.95em;\">\n<thead>\n<tr>\n<th style=\"border:1px solid #e3e6ea;padding:8px 12px;background:#f6f8fa;text-align:left;font-weight:600;\">Layer<\/th>\n<th style=\"border:1px solid #e3e6ea;padding:8px 12px;background:#f6f8fa;text-align:left;font-weight:600;\">What to inspect<\/th>\n<th style=\"border:1px solid #e3e6ea;padding:8px 12px;background:#f6f8fa;text-align:left;font-weight:600;\">Typical failure<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">DNS\/CDN<\/td>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">Which service receives the request first<\/td>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">A different WAF blocks traffic<\/td>\n<\/tr>\n<tr>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\"><code>robots.txt<\/code><\/td>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\"><code>User-agent: OAI-SearchBot<\/code> rules<\/td>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">The crawler is disallowed<\/td>\n<\/tr>\n<tr>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">AWS WAF<\/td>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">Rule labels, action, priority, sampled request<\/td>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">403, CAPTCHA, or rate limit<\/td>\n<\/tr>\n<tr>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">Origin\/server<\/td>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">Access logs and response headers<\/td>\n<td style=\"border:1px solid #e3e6ea;padding:8px 12px;vertical-align:top;\">5xx, redirect loop, or auth block<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>A practical four-request test is more useful than checking only the browser. Send the same public URL with a normal browser user-agent, the OAI-SearchBot user-agent, the OAI-SearchBot user-agent from a verified source IP, and the bot user-agent from an unverified IP. Record the HTTP status, response length, redirect chain, and WAF rule ID.<\/p>\n<p>Expected interpretation:<\/p>\n<ul>\n<li>Browser <code>200<\/code>, bot <code>403<\/code>: likely WAF or bot mitigation<\/li>\n<li>Both browser and bot <code>403<\/code>: application, authentication, origin, or CDN issue<\/li>\n<li>Bot <code>429<\/code>: rate-based rule or upstream throttling<\/li>\n<li>Bot <code>200<\/code> with a challenge body: technically allowed by one rule but unusable to a crawler<\/li>\n<\/ul>\n<p>This test also exposes a common security mistake: trusting the user-agent alone. A malicious client can send <code>OAI-SearchBot<\/code> in a header without being operated by OpenAI.<\/p>\n<h2>How to create a safer OAI-SearchBot allow rule in AWS WAF<\/h2>\n<p>The safest AWS WAF design uses a <strong>two-factor match<\/strong>:<\/p>\n<ol>\n<li>The request user-agent contains <code>OAI-SearchBot<\/code>.<\/li>\n<li>The source address belongs to OpenAI\u2019s published OAI-SearchBot ranges.<\/li>\n<\/ol>\n<p>OpenAI publishes crawler IP addresses separately for its bots, including <code>searchbot.json<\/code>. Do not copy a third-party list into a permanent rule; use OpenAI\u2019s official range source and establish a review process when the list changes. (<a href=\"https:\/\/developers.openai.com\/api\/docs\/bots?orgid=145\" target=\"_blank\" rel=\"noopener\">developers.openai.com<\/a>)<\/p>\n<h3>Step 1: Create an IP set<\/h3>\n<p>In AWS WAF:<\/p>\n<ol>\n<li>Open the Web ACL attached to CloudFront, an Application Load Balancer, or API Gateway.<\/li>\n<li>Create an <strong>IP set<\/strong> for IPv4 addresses.<\/li>\n<li>Add the current CIDR ranges published for OAI-SearchBot.<\/li>\n<li>Create a separate IPv6 IP set only if OpenAI publishes applicable IPv6 ranges for your deployment.<\/li>\n<li>Name it clearly, such as <code>openai-oai-searchbot-ipset<\/code>.<\/li>\n<\/ol>\n<p>Avoid combining OAI-SearchBot, GPTBot, and ChatGPT-User into one IP set. Their purposes differ, and a single exception can unintentionally broaden access.<\/p>\n<h3>Step 2: Match the user-agent token<\/h3>\n<p>Create a custom statement that matches the <code>User-Agent<\/code> header for the token:<\/p>\n<pre><code class=\"language-text\">OAI-SearchBot\n<\/code><\/pre>\n<p>Use a case-insensitive contains match or a regex match that targets the token rather than the entire user-agent string. OpenAI notes that the version number and surrounding browser text may change, so matching the full example string is brittle. (<a href=\"https:\/\/developers.openai.com\/api\/docs\/bots?orgid=145\" target=\"_blank\" rel=\"noopener\">developers.openai.com<\/a>)<\/p>\n<p>Conceptually, the rule should be:<\/p>\n<pre><code class=\"language-text\">(User-Agent contains &quot;OAI-SearchBot&quot;)\nAND\n(source IP is in the OAI-SearchBot IP set)\n<\/code><\/pre>\n<p>In AWS WAF JSON-style logic, the structure is equivalent to:<\/p>\n<pre><code class=\"language-json\">{\n  &quot;AndStatement&quot;: {\n    &quot;Statements&quot;: [\n      {\n        &quot;ByteMatchStatement&quot;: {\n          &quot;FieldToMatch&quot;: { &quot;SingleHeader&quot;: { &quot;Name&quot;: &quot;user-agent&quot; } },\n          &quot;PositionalConstraint&quot;: &quot;CONTAINS&quot;,\n          &quot;SearchString&quot;: &quot;OAI-SearchBot&quot;,\n          &quot;TextTransformations&quot;: [\n            { &quot;Priority&quot;: 0, &quot;Type&quot;: &quot;LOWERCASE&quot; }\n          ]\n        }\n      },\n      {\n        &quot;IPSetReferenceStatement&quot;: {\n          &quot;ARN&quot;: &quot;arn:aws:wafv2:REGION:ACCOUNT:regional\/ipset\/openai-oai-searchbot-ipset\/ID&quot;\n        }\n      }\n    ]\n  }\n}\n<\/code><\/pre>\n<p>The ARN, scope, and IP version must match your actual Web ACL. CloudFront Web ACLs use the global scope in <code>us-east-1<\/code>; regional resources use their regional scope.<\/p>\n<h3>Step 3: Put the rule before terminating blocks<\/h3>\n<p>Set the allow rule priority above:<\/p>\n<ul>\n<li>AWS Managed Rules that block automated traffic<\/li>\n<li>Bot Control rules<\/li>\n<li>Custom \u201cblock unknown bots\u201d rules<\/li>\n<li>Rate-based rules that would terminate the request<\/li>\n<li>CAPTCHA or challenge actions<\/li>\n<\/ul>\n<p>AWS WAF evaluates rules in priority order. An allow action can stop later rules from inspecting the request, but it cannot undo a block that has already terminated evaluation.<\/p>\n<p>AWS WAF Bot Control also exposes labels for bot categories such as search engines and content fetchers. If you use Bot Control, inspect the labels in sampled requests before deciding whether to exempt or allow a category. (<a href=\"https:\/\/docs.aws.amazon.com\/pdfs\/waf\/latest\/developerguide\/waf-dg.pdf\" target=\"_blank\" rel=\"noopener\">docs.aws.amazon.com<\/a>)<\/p>\n<h2>Should you allow by user-agent only?<\/h2>\n<p>No, not for a production WAF exception.<\/p>\n<p>A user-agent-only rule is easy to impersonate. It may solve a 403 quickly, but it weakens the purpose of your bot protection and can expose endpoints that should remain private. The IP-plus-user-agent rule is narrower and easier to audit.<\/p>\n<p>You can use a temporary user-agent-only rule in a controlled test environment, but log it, limit its path scope, and remove it after verification. Never apply the exception to login, checkout, admin, upload, or API mutation routes unless there is a separate business requirement.<\/p>\n<figure class=\"wp-block-image size-large\" style=\"margin:1.5em 0;\"><img decoding=\"async\" src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/10\/backend-8635-2.jpg\" alt=\"OAI-SearchBot verification matrix comparing user-agent and source IP\" style=\"max-width:100%;height:auto;\"><\/figure>\n<h2>Validate the fix at the edge, not only at the origin<\/h2>\n<p>After publishing the rule, repeat the four-request test. Check:<\/p>\n<ol>\n<li>AWS WAF sampled requests show the intended rule action.<\/li>\n<li>CloudFront or load balancer logs show the expected status.<\/li>\n<li>The response body is the actual HTML, not a CAPTCHA or JavaScript challenge.<\/li>\n<li>Redirects do not send the crawler to a blocked hostname.<\/li>\n<li><code>robots.txt<\/code> permits OAI-SearchBot for the paths you want discovered.<\/li>\n<\/ol>\n<p>A successful <code>200<\/code> response is necessary but not sufficient. The returned page should contain readable server-rendered content, a canonical URL, and no authentication barrier. If the page depends on client-side JavaScript to reveal its main content, the crawler may receive a technically successful response with little useful text.<\/p>\n<p>For related failure modes, see this guide to <a href=\"https:\/\/maxaeo.ai\/blog\/openai-crawler-403\/\">debugging WAF 403 errors for OpenAI crawlers<\/a> and the broader <a href=\"https:\/\/maxaeo.ai\/blog\/allowlist-ai-search-agents\/\">AWS WAF allowlist guide for AI search agents<\/a>.<\/p>\n<h2>What to do if OAI-SearchBot still receives 403 or 429<\/h2>\n<p>Use this troubleshooting sequence:<\/p>\n<ol>\n<li><strong>Check the actual source IP.<\/strong> Confirm it belongs to OpenAI\u2019s published OAI-SearchBot ranges.<\/li>\n<li><strong>Inspect WAF logs.<\/strong> Identify the terminating rule, not merely the first matching rule.<\/li>\n<li><strong>Review rule priority.<\/strong> Move the narrowly scoped allow rule above the blocker.<\/li>\n<li><strong>Check the CDN layer.<\/strong> CloudFront, a third-party CDN, or an origin firewall may apply a separate policy.<\/li>\n<li><strong>Inspect redirects.<\/strong> A permitted homepage can redirect to a protected regional or language URL.<\/li>\n<li><strong>Review rate limits.<\/strong> Exempt only verified crawler traffic, and avoid broad IP-based exemptions.<\/li>\n<li><strong>Test <code>robots.txt<\/code>.<\/strong> WAF access does not override a disallow directive.<\/li>\n<li><strong>Compare response bodies.<\/strong> A 200 status containing a challenge page is not a successful crawl.<\/li>\n<\/ol>\n<p>Do not assume that allowing OAI-SearchBot will automatically increase citations or recommendations. It removes one technical access barrier; content quality, source authority, relevance, and how AI systems evaluate your pages still matter.<\/p>\n<h2>How to monitor whether the unblock improved AI visibility<\/h2>\n<p>Server logs tell you whether a crawler requested a page. They do not tell you whether your brand is mentioned, recommended, or cited in AI answers.<\/p>\n<p>A separate visibility workflow should track:<\/p>\n<ul>\n<li>Brand mention rate<\/li>\n<li>Average recommendation position<\/li>\n<li>Competitor visibility<\/li>\n<li>Sentiment and factual accuracy<\/li>\n<li>Cited domains and specific source pages<\/li>\n<li>Trends by AI engine and prompt<\/li>\n<\/ul>\n<p>MaxAEO monitors brand visibility across eight AI engines, including ChatGPT, Perplexity, Gemini, Claude, Copilot, Grok, Google AI Mode, and Google AI Overview. Its daily monitoring and citation tracking can help distinguish \u201cthe crawler can access my site\u201d from \u201cAI systems actually use my content.\u201d A <a href=\"https:\/\/maxaeo.ai\/\">free AI visibility diagnostic<\/a> provides a starting audit using your brand and website information.<\/p>\n<h2>Frequently asked questions<\/h2>\n<h3>Does allowing OAI-SearchBot also allow GPTBot?<\/h3>\n<p>No. OpenAI documents these as separate user agents with different purposes. Create separate <code>robots.txt<\/code>, IP, and WAF decisions for each crawler. (<a href=\"https:\/\/developers.openai.com\/api\/docs\/bots?orgid=145\" target=\"_blank\" rel=\"noopener\">developers.openai.com<\/a>)<\/p>\n<h3>Can I allow OAI-SearchBot with only <code>robots.txt<\/code>?<\/h3>\n<p>Only if your WAF and other infrastructure already permit the request. <code>robots.txt<\/code> does not override AWS WAF blocks, rate limits, authentication, or CDN challenges.<\/p>\n<h3>Why does the user-agent match but the rule still block?<\/h3>\n<p>The request may come from an unverified IP, or a higher-priority rule may terminate evaluation first. Check the terminating rule ID and source address in AWS WAF logs.<\/p>\n<h3>Should the rule cover the whole site?<\/h3>\n<p>Prefer a narrow public-content scope where practical. Exclude administrative, transactional, private, and mutation endpoints from crawler exceptions.<\/p>\n<h3>Does a successful crawl guarantee ChatGPT visibility?<\/h3>\n<p>No. It only confirms that the crawler can retrieve the response. AI visibility also depends on whether the content is relevant, trustworthy, accessible, and selected for a particular answer.<\/p>\n<p><script type=\"application\/ld+json\">\n{\"@context\":\"https:\/\/schema.org\",\"@type\":\"Article\",\"author\":{\"@type\":\"Organization\",\"name\":\"maxaeo.ai\"},\"dateModified\":\"2026-10-29\",\"datePublished\":\"2026-10-29\",\"description\":\"Unblock OAI-SearchBot in AWS WAF with a verified user-agent and IP allow rule, safe ordering, test requests, and troubleshooting steps. Apply the fix.\",\"headline\":\"Unblock OAI-SearchBot AWS WAF Rule | maxaeo.ai\",\"image\":\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/10\/art-12933-cover.jpg\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"maxaeo.ai\"}}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Unblock OAI-SearchBot in AWS WAF with a verified user-agent and IP allow rule, safe ordering, test requests, and troubleshooting steps. Apply the fix.<\/p>\n","protected":false},"author":1,"featured_media":3425,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3426","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts\/3426","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/comments?post=3426"}],"version-history":[{"count":0,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts\/3426\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/media\/3425"}],"wp:attachment":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/media?parent=3426"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/categories?post=3426"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/tags?post=3426"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}