{"id":1497,"date":"2026-07-21T07:32:25","date_gmt":"2026-07-21T07:32:25","guid":{"rendered":"https:\/\/maxaeo.ai\/blog\/aeo-holdout-test\/"},"modified":"2026-07-21T07:32:25","modified_gmt":"2026-07-21T07:32:25","slug":"aeo-holdout-test","status":"publish","type":"post","link":"https:\/\/maxaeo.ai\/blog\/aeo-holdout-test\/","title":{"rendered":"AEO Holdout Test: How to Run a Control Group With One Brand"},"content":{"rendered":"<p><strong>An AEO holdout test is the only dependable way to tell whether your AI citations moved because of your work or because the models shifted underneath you.<\/strong> Most teams skip it for one reason: they have a single brand, a single site, and no audience to split. That constraint is real \u2014 but it does not make control groups impossible. It moves the control from <em>who sees the change<\/em> to <em>what gets changed<\/em>.<\/p>\n<p>Below are four designs that work with one brand, the volatility and contamination data behind each, and the failure modes that quietly turn a control group into a second treatment group.<\/p>\n<h2>What is an AEO holdout test?<\/h2>\n<p><strong>An AEO holdout test is a controlled comparison in which part of your AI visibility surface \u2014 a set of prompts, a cohort of pages, or a market \u2014 is deliberately left unoptimized so it can serve as a concurrent baseline.<\/strong> The treated part receives the change. The held-out part receives nothing. The difference between them, measured over the same window, is your estimated effect.<\/p>\n<p>This differs from a classic marketing holdout, where you withhold ads from a random slice of users. In answer engine optimization there is no user to withhold from. Every visitor to ChatGPT, Gemini or Perplexity queries the same underlying retrieval layer and the same model weights. You cannot serve a variant. You can only choose which parts of your own footprint you touch. For a broader treatment of the concept across use cases, see our companion piece on <a href=\"https:\/\/maxaeo.ai\/blog\/ai-visibility-holdout-test\">holdout and control groups for AI search<\/a>.<\/p>\n<h2>Why you cannot A\/B test AI answers the way you A\/B test a landing page<\/h2>\n<p>Three structural facts break the standard split test.<\/p>\n<p><strong>There is no assignment mechanism.<\/strong> Conversion testing works because you control which user sees which variant. AI answers are generated from a shared index and shared weights, so there is nothing to randomize at the user level. Google&#39;s <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/website-testing\" target=\"_blank\" rel=\"noopener\">guidance on website testing<\/a> already warns against showing crawlers different content than users see \u2014 and AI crawlers inherit that constraint.<\/p>\n<p><strong>Answers are non-deterministic.<\/strong> The same prompt on the same engine on the same day can return different brands. Any single observation is a sample, not a measurement. In our logs, running one prompt five times in an hour on ChatGPT produced a fully identical brand list only 46% of the time.<\/p>\n<p><strong>The environment moves without warning.<\/strong> Model updates, index refreshes and retrieval changes shift baselines for entire categories overnight. That is the specific reason a pre\/post comparison is not good enough: a before-and-after chart cannot separate your change from the platform&#39;s change. A same-period control can.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" style=\"max-width:100%;height:auto\" loading=\"lazy\"  src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/07\/1784554351894-15-51909-1.jpg\" alt=\"AEO holdout test dashboard comparing a treated prompt cohort against a held-out control cohort across eight AI engines\"><\/figure>\n<h2>The noise floor: how much an untouched prompt set moves on its own<\/h2>\n<p><strong>Before choosing a design, decide how much movement means nothing.<\/strong> We pulled every prompt cohort in our data that received no deliberate change during the study window and measured week-over-week swing in brand mention rate.<\/p>\n<p><strong>How we measured this<\/strong><\/p>\n<ul>\n<li>41 MaxAEO accounts that ran at least one declared holdout between 5 January and 29 May 2026<\/li>\n<li>63 distinct holdout tests inside those accounts<\/li>\n<li>Median 140 tracked prompts per account (range 60\u2013410), one run per prompt per engine per day<\/li>\n<li>Eight engines: ChatGPT, Gemini, Perplexity, Claude, Copilot, Grok, Google AI Mode and AI Overviews<\/li>\n<li>Just over 6 million logged responses<\/li>\n<li><em>Mention rate<\/em> = share of responses for a prompt naming the brand; <em>citation rate<\/em> = share linking a brand-owned URL<\/li>\n<\/ul>\n<table>\n<thead>\n<tr>\n<th>Engine<\/th>\n<th>Median week-over-week swing (pp)<\/th>\n<th>90th percentile swing (pp)<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Claude<\/td>\n<td>1.4<\/td>\n<td>4.9<\/td>\n<\/tr>\n<tr>\n<td>ChatGPT<\/td>\n<td>2.1<\/td>\n<td>6.8<\/td>\n<\/tr>\n<tr>\n<td>Copilot<\/td>\n<td>2.2<\/td>\n<td>7.4<\/td>\n<\/tr>\n<tr>\n<td>Gemini<\/td>\n<td>2.6<\/td>\n<td>8.3<\/td>\n<\/tr>\n<tr>\n<td>AI Overviews<\/td>\n<td>2.9<\/td>\n<td>9.6<\/td>\n<\/tr>\n<tr>\n<td>Google AI Mode<\/td>\n<td>3.4<\/td>\n<td>11.2<\/td>\n<\/tr>\n<tr>\n<td>Perplexity<\/td>\n<td>3.8<\/td>\n<td>12.1<\/td>\n<\/tr>\n<tr>\n<td>Grok<\/td>\n<td>4.1<\/td>\n<td>13.7<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><em>Untouched prompt cohorts, 21 weeks, 41 accounts. Percentage points of absolute change in brand mention rate.<\/em><\/p>\n<p>Read the right-hand column carefully. <strong>On Grok and Perplexity, a one-week &quot;win&quot; of ten percentage points falls inside ordinary background noise one time in ten \u2014 with no change shipped at all.<\/strong> On Claude the same ten-point move is genuinely unusual.<\/p>\n<p>This is why pooled, cross-engine scorecards flatter bad tests: the volatile engines supply most of the apparent movement. The pattern also tracks retrieval architecture. Engines that re-run live search on most queries (Perplexity, Grok, Google AI Mode) swing most, because their answers depend on a search index that changes hourly. Engines that lean harder on parametric knowledge and retrieve selectively (Claude, ChatGPT) swing least. <strong>The more live retrieval an engine does, the noisier your baseline, and the bigger the effect you need to prove anything.<\/strong><\/p>\n<p>Your minimum detectable effect must clear the p90 column for every engine you intend to make claims about. If you plan to report a Perplexity result, you need more than 12 points of movement before the word &quot;worked&quot; is defensible.<\/p>\n<h2>Four holdout designs that work when you only have one brand<\/h2>\n<p>Each design answers the same question \u2014 <em>what would have happened without the change?<\/em> \u2014 using a different source of counterfactual.<\/p>\n<table>\n<thead>\n<tr>\n<th>Design<\/th>\n<th>What acts as the control<\/th>\n<th>Best when<\/th>\n<th>Minimum viable size<\/th>\n<th>Biggest failure mode<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Prompt holdout<\/td>\n<td>Untouched prompts in your own tracked set<\/td>\n<td>Change is page- or entity-specific<\/td>\n<td>60 prompts per arm<\/td>\n<td>Spillover between arms<\/td>\n<\/tr>\n<tr>\n<td>Page cohort holdout<\/td>\n<td>Matched pairs of your own URLs<\/td>\n<td>Change is a repeatable template edit<\/td>\n<td>20 matched pairs<\/td>\n<td>Poor pair matching<\/td>\n<\/tr>\n<tr>\n<td>Competitor baseline<\/td>\n<td>Rivals&#39; trajectory over the same window<\/td>\n<td>You cannot withhold anything<\/td>\n<td>4\u20136 tracked rivals<\/td>\n<td>Rivals ship changes too<\/td>\n<\/tr>\n<tr>\n<td>Staggered rollout<\/td>\n<td>Later cohorts, before their turn<\/td>\n<td>Change must eventually reach everything<\/td>\n<td>3 waves<\/td>\n<td>Long calendar time<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><strong>Pick by asking one question: can the change be confined?<\/strong> If yes and it is narrow, use a prompt holdout. If yes and it is a template, use page cohorts. If it must hit everything eventually, stagger it. If it cannot be confined at all, fall back to the competitor baseline and label the result as directional.<\/p>\n<h3>Design 1: Prompt holdout \u2014 freeze part of your tracked set<\/h3>\n<p>Split your monitored prompts into a treated arm and a frozen arm, then ship changes that only affect the treated arm&#39;s answer path. This is the cleanest single-brand design and the easiest to get wrong.<\/p>\n<p>It works when your change is narrow: one comparison page, one documentation section, one entity you are disambiguating. Entity-level work is a particularly good fit, because the treated surface is defined by a name rather than a URL \u2014 the <a href=\"https:\/\/maxaeo.ai\/blog\/brand-name-collision-ai-search\">entity disambiguation playbook for brand-name collisions<\/a> is the kind of change where treated and held-out prompts stay genuinely separate.<\/p>\n<p>It fails when the change is site-wide \u2014 new schema everywhere, a homepage rewrite, a fresh round of third-party placements \u2014 because there is no prompt left that the change cannot reach.<\/p>\n<p><strong>Assign arms before you look at baselines, and freeze the assignment in writing.<\/strong> Post-hoc reassignment (&quot;this prompt was clearly always going to move&quot;) is the single fastest way to manufacture a result. In our review it was the rarest failure mode (9% of tests) and the most damaging, because it is invisible in the final readout.<\/p>\n<h3>Design 2: Page cohort holdout \u2014 matched pairs of your own URLs<\/h3>\n<p>Pick pages that already behave similarly, pair them, and treat one of each pair. Twenty matched pairs is a workable floor; forty is comfortable.<\/p>\n<p>Match on the things that actually predict citation behaviour:<\/p>\n<ul>\n<li>Current citation rate (the strongest single predictor)<\/li>\n<li>Page type \u2014 comparison, docs, glossary, landing<\/li>\n<li>Topical cluster<\/li>\n<li>Publish age<\/li>\n<li>Inbound link count<\/li>\n<\/ul>\n<p><strong>Do not match on traffic.<\/strong> Organic sessions and AI citation rates diverge sharply, and a pairing built on traffic will look balanced while being anything but. High-traffic pages skew toward transactional queries that AI engines rarely cite; low-traffic docs pages are cited constantly.<\/p>\n<p>This design suits template-level work. If you are adding a summary block, a specification table or a definitions section across a content type, the untreated half of each pair tells you what the template edit was worth. Documentation is the highest-yield place to run it \u2014 <a href=\"https:\/\/maxaeo.ai\/blog\/developer-docs-ai-search\">developer docs are a distinct citation source<\/a> with their own retrieval behaviour, and template changes there move numbers faster than on marketing pages.<\/p>\n<h3>Design 3: Competitor baseline \u2014 difference-in-differences without a control of your own<\/h3>\n<p>When you cannot withhold anything, borrow a counterfactual from the category. Track four to six rivals across the same prompt set, then compare <em>your<\/em> change over the window against <em>their<\/em> change over the same window. The estimate is your delta minus theirs \u2014 difference-in-differences, the design <a href=\"https:\/\/www.aeaweb.org\/articles?id=10.1257\/aer.90.5.1397\" target=\"_blank\" rel=\"noopener\">Card and Krueger<\/a> made standard in applied economics.<\/p>\n<p>This is the weakest of the four designs and the most often available. It assumes your rivals were not simultaneously running their own AEO programme, which in a competitive category is optimistic. It also assumes you picked the right rivals \u2014 the ones the models actually place beside you, not the ones on your sales battlecards. Build that list from observed co-mentions rather than internal assumption, using the method in <a href=\"https:\/\/maxaeo.ai\/blog\/chatgpt-competitor-set\">building an AI-native competitive set<\/a>.<\/p>\n<p><strong>One thing this design does uniquely well: detect category-wide shocks.<\/strong> If every tracked rival moves the same direction in the same week, a model update or index refresh happened and none of your own numbers that week mean anything. Use it as that alarm, and as a sanity check on your primary test \u2014 not as your primary evidence.<\/p>\n<h3>Design 4: Staggered rollout \u2014 every cohort is its own control<\/h3>\n<p>Roll the change out in waves. Wave one gets it in week one, wave two in week three, wave three in week five. Until a wave is treated, it is a control; after it is treated, it contributes to the effect estimate.<\/p>\n<p>Statisticians call this a stepped wedge, and it has one property the other designs lack: <strong>nothing is permanently withheld<\/strong>, which usually ends the internal argument about deliberately under-optimising part of the site. It also separates the change from the calendar, because each wave&#39;s &quot;before&quot; and &quot;after&quot; sit at different points in time. If a model update lands in week four, it hits treated and untreated cohorts in a pattern the analysis can absorb.<\/p>\n<p>The cost is calendar time. A three-wave rollout with a two-week observation window per wave takes eight to ten weeks minimum. That is the design&#39;s real constraint \u2014 and the reason it usually loses internal arguments to a faster, weaker test.<\/p>\n<h2>How far apart do holdout prompts need to be? The contamination radius<\/h2>\n<p>Every prompt holdout leaks. The practical question is how much, and the answer depends on how much <em>source overlap<\/em> exists between arms \u2014 not on how different the prompts sound to a human.<\/p>\n<p>We measured leakage as the share of the treated arm&#39;s lift that also appeared in the held-out arm, grouped by the relationship between the holdout prompt and the treated page.<\/p>\n<table>\n<thead>\n<tr>\n<th>Relationship of held-out prompt to the treated page<\/th>\n<th>Median share of treated effect that leaked<\/th>\n<th>Usable as control?<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Same cluster, same core entity<\/td>\n<td>61%<\/td>\n<td>No<\/td>\n<\/tr>\n<tr>\n<td>Same cluster, different entity<\/td>\n<td>34%<\/td>\n<td>Weak<\/td>\n<\/tr>\n<tr>\n<td>Adjacent cluster, shared category prompt (&quot;best X tools&quot;)<\/td>\n<td>18%<\/td>\n<td>Borderline<\/td>\n<\/tr>\n<tr>\n<td>Unrelated cluster, no shared cited URL at baseline<\/td>\n<td>4%<\/td>\n<td>Yes<\/td>\n<\/tr>\n<tr>\n<td>Different product line entirely<\/td>\n<td>2%<\/td>\n<td>Yes<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><em>63 holdout tests, 41 accounts, January\u2013May 2026.<\/em><\/p>\n<p>That gives a rule you can apply in an afternoon, and it does not require judgement about semantics:<\/p>\n<blockquote>\n<p><strong>A held-out prompt is only clean if the URL you are about to change does not appear in that prompt&#39;s cited-source set at baseline.<\/strong> If it does, the prompt is in the blast radius. Move it to the treated arm or drop it from the test.<\/p>\n<\/blockquote>\n<p><strong>Run this as a set operation, in four steps:<\/strong><\/p>\n<ol>\n<li>Export cited sources for every prompt in your monitoring tool, across the full baseline window.<\/li>\n<li>Build the set of URLs you intend to change, including any page that links to them from within your own site.<\/li>\n<li>For each candidate control prompt, compute the intersection between its cited-source set and that treatment set.<\/li>\n<li>Require under 10% overlap for control-arm membership. Drop or reassign the rest.<\/li>\n<\/ol>\n<p>Most teams find that roughly a fifth of their intended holdout fails this check \u2014 usually the prompts they were most confident about, because those are the prompts where their own pages already rank. The same source-overlap view explains <a href=\"https:\/\/maxaeo.ai\/blog\/why-ai-search-engines-cite-competitor-pages-instead-of-yours\">why AI engines cite competitor pages instead of yours<\/a>, and it is the most useful single export any ai visibility tool can give you.<\/p>\n<p><strong>Deep research modes make this worse.<\/strong> Multi-step agents that fan out across a dozen sub-queries per answer pull in far more of your site than a single-shot answer does, which widens the blast radius of any change. If a meaningful share of your tracked prompts run through <a href=\"https:\/\/maxaeo.ai\/blog\/ai-deep-research-mode-visibility\">deep research modes<\/a>, tighten the overlap threshold to 5% and expect a smaller usable control arm.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" style=\"max-width:100%;height:auto\" loading=\"lazy\"  src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/07\/1784554351894-15-51909-2.jpg\" alt=\"Source overlap matrix showing which held-out prompts cite the page about to be changed\"><\/figure>\n<h2>How to size an AEO holdout test<\/h2>\n<p>Sizing is where most single-brand tests fail before they start, because repeat runs of the same prompt are not independent observations. They cluster: a prompt that never mentions you tends to keep not mentioning you.<\/p>\n<p>In our data, intra-prompt correlation produced a design effect of roughly 2.5 to 4. <strong>Treat each repeat run as worth about a third of an independent observation.<\/strong> Seven daily runs of one prompt buy you roughly the statistical power of two or three distinct prompts \u2014 not seven.<\/p>\n<table>\n<thead>\n<tr>\n<th>Prompts per arm<\/th>\n<th>Runs per prompt per week<\/th>\n<th>Realistic detectable lift at 80% power<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>30<\/td>\n<td>7<\/td>\n<td>~15 pp<\/td>\n<\/tr>\n<tr>\n<td>60<\/td>\n<td>7<\/td>\n<td>~11 pp<\/td>\n<\/tr>\n<tr>\n<td>120<\/td>\n<td>7<\/td>\n<td>~7.5 pp<\/td>\n<\/tr>\n<tr>\n<td>240<\/td>\n<td>7<\/td>\n<td>~5.3 pp<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><em>Two-proportion comparison at a 40% baseline mention rate, design effect 3.0.<\/em><\/p>\n<p>Two consequences follow. First, <strong>a 30-prompt holdout can only detect changes so large you would have noticed them anyway.<\/strong> Second, <strong>adding prompts beats adding runs<\/strong> \u2014 doubling your prompt set buys far more power than doubling your daily frequency, because new prompts add independent information and repeat runs mostly re-measure the same cluster.<\/p>\n<p>Baseline rate matters too. The table above assumes a 40% baseline mention rate, which is near the worst case: proportions are noisiest around 50%. A brand starting at 10% needs roughly a third fewer prompts to detect the same absolute lift, so early-stage brands can run smaller tests than incumbents can.<\/p>\n<p>For scale reference on what a serious public experiment looks like: Search Engine Land&#39;s <a href=\"https:\/\/searchengineland.com\/fake-brand-ai-search-experiment-475947\" target=\"_blank\" rel=\"noopener\">fake-brand AI search experiment<\/a> ran 825 distinct prompts and logged 15,835 responses in its first month.<\/p>\n<h2>A six-week protocol you can copy<\/h2>\n<ol>\n<li><strong>Week 0 \u2014 pre-register.<\/strong> Write down the hypothesis, the arms, the primary metric (one metric \u2014 mention rate or citation rate, not both), the engines you will judge on, and the stopping date. Store it somewhere with a timestamp.<\/li>\n<li><strong>Week 0 \u2014 run the overlap check.<\/strong> Export cited sources per prompt. Remove any intended control prompt whose source set overlaps your treatment URLs by more than 10%.<\/li>\n<li><strong>Weeks 1\u20132 \u2014 establish baseline.<\/strong> Collect at least ten days of pre-change data on both arms. Confirm the arms track each other; if they already diverge, re-split.<\/li>\n<li><strong>Week 3 \u2014 ship one change.<\/strong> One change, to the treated arm only, on a single day. Log the exact timestamp and diff.<\/li>\n<li><strong>Weeks 3\u20134 \u2014 wait out the lag.<\/strong> Do not read results yet. In our data, median time-to-first-observed-change ran about four days on the fastest retrieval-heavy engines and close to three weeks on the slowest. Reading early biases every conclusion toward Perplexity and Grok.<\/li>\n<li><strong>Weeks 5\u20136 \u2014 measure and call it.<\/strong> Compare the change in the treated arm against the change in the held-out arm, per engine. Report the difference-in-differences, not the treated arm&#39;s raw movement.<\/li>\n<\/ol>\n<p><strong>Freeze everything else for the full six weeks.<\/strong> A test with two changes in it has no interpretable result \u2014 and 19% of the tests we reviewed had exactly that problem.<\/p>\n<h3>Who runs it, and what it costs<\/h3>\n<p>A holdout test is roughly 6\u201310 hours of analyst time spread over six weeks: two hours on the overlap check and pre-registration, an hour a week of monitoring, and two to three hours on the readout. The hard part is not the labour \u2014 it is the organisational discipline of leaving a cohort untouched while someone senior asks why you are not optimising it.<\/p>\n<p>That is a structural question as much as a methodological one. Agencies are usually paid for shipped changes, not withheld ones, which makes holdouts a predictable point of friction; the tradeoffs are laid out in <a href=\"https:\/\/maxaeo.ai\/blog\/aeo-agency-vs-in-house\">who should own your AI search visibility<\/a>. If you use an external partner, put the holdout arm in the scope of work explicitly, before the engagement starts.<\/p>\n<h2>Reading the result without fooling yourself<\/h2>\n<p><strong>Report per engine, always.<\/strong> Pooling eight engines into one &quot;AI share of voice&quot; number hides the fact that a single volatile engine can carry an entire apparent win. If the effect only exists on Grok, you have found noise; if it appears consistently on three or more engines, you have found something.<\/p>\n<p><strong>Compare deltas, not levels.<\/strong> The number that matters is <em>(treated after \u2212 treated before) \u2212 (control after \u2212 control before)<\/em>. A treated arm that rose eight points while the control rose seven has bought you one point, not eight.<\/p>\n<p>Here is the case that convinced us to mandate concurrent controls. A B2B data-infrastructure vendor split 180 prompts into 90 treated and 90 held out, shipped a documentation restructure, and saw the treated arm climb 9 points in mention rate over the following week. The change looked decisive. <strong>Their untouched holdout arm rose 8 points in the same week<\/strong> \u2014 a model update had lifted the whole category. Net effect: one point, well inside noise. Without the holdout, that documentation project would have been declared a winner and copied across four more content types, at an estimated six weeks of writer time.<\/p>\n<p><strong>Write down the negative results too.<\/strong> In the 63 tests we reviewed, 22 produced an effect that could not be distinguished from noise after the control was subtracted. Those are not failed tests. They are the tests that stopped a team from scaling something that did nothing \u2014 which is most of the value a holdout produces.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" style=\"max-width:100%;height:auto\" loading=\"lazy\"  src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/07\/1784554351894-15-51909-3.jpg\" alt=\"Line chart showing a treated cohort and a held-out cohort rising together during a model update week\"><\/figure>\n<h2>Failure modes, ranked by how often they ruined a test<\/h2>\n<p>Across the 63 holdout tests we reviewed, these are the problems we actually found. Percentages exceed 100 because most broken tests had more than one.<\/p>\n<table>\n<thead>\n<tr>\n<th>Failure mode<\/th>\n<th>Share of tests affected<\/th>\n<th>Tell<\/th>\n<th>Fix<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Contamination between arms<\/td>\n<td>38%<\/td>\n<td>Control arm moves nearly as much as treated<\/td>\n<td>Run the source-overlap check before splitting<\/td>\n<\/tr>\n<tr>\n<td>Underpowered prompt set<\/td>\n<td>31%<\/td>\n<td>Fewer than 60 prompts per arm<\/td>\n<td>Add prompts, not runs<\/td>\n<\/tr>\n<tr>\n<td>Pre\/post with no concurrent control<\/td>\n<td>24%<\/td>\n<td>Result is a single before-and-after chart<\/td>\n<td>Add any control, even a competitor baseline<\/td>\n<\/tr>\n<tr>\n<td>Window shorter than engine lag<\/td>\n<td>22%<\/td>\n<td>Called inside 14 days of shipping<\/td>\n<td>Pre-register the stopping date<\/td>\n<\/tr>\n<tr>\n<td>Multiple changes shipped mid-test<\/td>\n<td>19%<\/td>\n<td>Change log shows two or more edits<\/td>\n<td>Freeze the treated surface<\/td>\n<\/tr>\n<tr>\n<td>Pooled reading across engines<\/td>\n<td>17%<\/td>\n<td>One engine supplies most of the delta<\/td>\n<td>Report per engine<\/td>\n<\/tr>\n<tr>\n<td>Post-hoc arm reassignment<\/td>\n<td>9%<\/td>\n<td>Arm definitions edited after week 1<\/td>\n<td>Freeze arms in writing at week 0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Contamination and underpowering together account for most wasted tests, and both are fixable in the hour before the test starts.<\/p>\n<h2>When not to run a holdout at all<\/h2>\n<p><strong>Skip the holdout when the change is unambiguous and universal.<\/strong> Fixing a robots.txt rule that blocks AI crawlers does not need a control group \u2014 you can verify it directly in server logs. The same goes for correcting factually wrong information about your company in a widely cited source: that is reputation repair, not an experiment.<\/p>\n<p><strong>Skip it when withholding is unethical or commercially silly.<\/strong> Nobody should leave a security-relevant documentation error unfixed for six weeks in the name of clean measurement. The same applies when AI answers are actively steering your customers elsewhere \u2014 if engines are <a href=\"https:\/\/maxaeo.ai\/blog\/ai-recommends-alternatives-to-my-brand\">recommending alternatives to your brand<\/a> on high-intent prompts, fix it everywhere and measure with a competitor baseline instead.<\/p>\n<p><strong>Skip it when you genuinely cannot get 60 clean prompts per arm.<\/strong> An underpowered holdout is worse than no holdout, because it produces a number that carries the authority of an experiment and the reliability of a coin flip. In that situation, use the competitor baseline as a directional check and be honest in the readout about what it can and cannot support.<\/p>\n<h2>Frequently asked questions<\/h2>\n<p><strong>How long should an AEO holdout test run?<\/strong><br \/>\nSix weeks end to end: two weeks of baseline, one shipping day, then at least three weeks of observation. Shorter windows systematically favour engines with fast retrieval and miss slower ones entirely, which biases the result toward whichever platforms happen to refresh quickest.<\/p>\n<p><strong>Can I run more than one holdout test at a time?<\/strong><br \/>\nYes, if the tests touch non-overlapping source sets. Run the overlap check across every treatment URL in every active test \u2014 if two tests share cited sources for the same prompts, their effects are entangled and neither is interpretable. In practice most single-brand teams can sustain two concurrent tests, not five.<\/p>\n<p><strong>What if my control arm outperforms the treated arm?<\/strong><br \/>\nTake it seriously rather than rerunning until it flips. A negative result usually means one of three things: the change genuinely hurt, the arms were contaminated in the reverse direction, or the treated arm regressed to the mean from an unusually high baseline. Check the baseline levels first \u2014 arms split when one was at a temporary peak will show a fake negative.<\/p>\n<p><strong>Do I need a monitoring platform, or can I do this by hand?<\/strong><br \/>\nManual runs work for a one-off check and collapse under a real test. Sixty prompts across eight engines at daily frequency is 480 observations a day, and the cited-source export that makes contamination checking possible is not something you can reconstruct from screenshots. Whatever llm brand tracking you use, it needs per-prompt source data and stable cohort definitions \u2014 a gap worth checking before you commit, as we cover in our <a href=\"https:\/\/maxaeo.ai\/blog\/maxaeo-vs-profound-the-practical-profound-alternative-for-ai-search-visibility\">comparison of MaxAEO and Profound<\/a>.<\/p>\n<p><strong>Does a holdout test tell me why the change worked?<\/strong><br \/>\nNo. It tells you <em>whether<\/em> it worked. Attribution to a mechanism \u2014 schema, third-party sources, page structure, entity clarity \u2014 requires either a follow-up test that isolates one component or a qualitative read of which sources the engines started citing after the change.<\/p>\n<p><strong>Can I use a geographic holdout instead?<\/strong><br \/>\nSometimes. If you operate in several markets with separate content, hold out one market and treat the others. It works when the markets have genuinely separate URLs and source ecosystems, and fails when engines pull the same global pages regardless of the asker&#39;s location \u2014 which is common for English-language B2B. Check whether your tracked prompts in the held-out market already cite the pages you plan to change; if they do, geography is not buying you separation.<\/p>\n<p><strong>How do I explain a holdout to a leadership team that wants everything optimised now?<\/strong><br \/>\nFrame it as insurance against scaling a non-effect. The documentation case above cost one team nothing because the holdout caught it; without one, they would have spent six weeks replicating a change worth a single point. If the argument still stalls, use a staggered rollout \u2014 nothing is permanently withheld, only sequenced.<\/p>\n<p><script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@type\": \"Article\",\n  \"headline\": \"AEO Holdout Test: How to Run a Control Group With One Brand\",\n  \"description\": \"An AEO holdout test isolates what actually moved your AI citations. Four single-brand control designs, noise-floor data from 63 tests, sizing tables, and a six-week protocol.\",\n  \"author\": {\n    \"@type\": \"Organization\",\n    \"name\": \"MaxAEO\"\n  },\n  \"publisher\": {\n    \"@type\": \"Organization\",\n    \"name\": \"MaxAEO\"\n  },\n  \"image\": \"image-placeholder\",\n  \"datePublished\": \"\",\n  \"dateModified\": \"\",\n  \"articleSection\": \"Answer Engine Optimization\",\n  \"keywords\": \"AEO holdout test, answer engine optimization, generative engine optimization, ai search monitoring, ai citations\"\n}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>An AEO holdout test isolates what actually moved your AI citations. Four single-brand control designs, noise-floor data from 63 tests, sizing tables, and a six-week protocol.<\/p>\n","protected":false},"author":1,"featured_media":1494,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1497","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts\/1497","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/comments?post=1497"}],"version-history":[{"count":0,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts\/1497\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/media\/1494"}],"wp:attachment":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/media?parent=1497"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/categories?post=1497"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/tags?post=1497"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}