An AEO holdout test is the only dependable way to tell whether your AI citations moved because of your work or because the models shifted underneath you. Most teams skip it for one reason: they have a single brand, a single site, and no audience to split. That constraint is real — but it does not make control groups impossible. It moves the control from who sees the change to what gets changed.
Below are four designs that work with one brand, the volatility and contamination data behind each, and the failure modes that quietly turn a control group into a second treatment group.
What is an AEO holdout test?
An AEO holdout test is a controlled comparison in which part of your AI visibility surface — a set of prompts, a cohort of pages, or a market — is deliberately left unoptimized so it can serve as a concurrent baseline. The treated part receives the change. The held-out part receives nothing. The difference between them, measured over the same window, is your estimated effect.
This differs from a classic marketing holdout, where you withhold ads from a random slice of users. In answer engine optimization there is no user to withhold from. Every visitor to ChatGPT, Gemini or Perplexity queries the same underlying retrieval layer and the same model weights. You cannot serve a variant. You can only choose which parts of your own footprint you touch. For a broader treatment of the concept across use cases, see our companion piece on holdout and control groups for AI search.
Why you cannot A/B test AI answers the way you A/B test a landing page
Three structural facts break the standard split test.
There is no assignment mechanism. Conversion testing works because you control which user sees which variant. AI answers are generated from a shared index and shared weights, so there is nothing to randomize at the user level. Google's guidance on website testing already warns against showing crawlers different content than users see — and AI crawlers inherit that constraint.
Answers are non-deterministic. The same prompt on the same engine on the same day can return different brands. Any single observation is a sample, not a measurement. In our logs, running one prompt five times in an hour on ChatGPT produced a fully identical brand list only 46% of the time.
The environment moves without warning. Model updates, index refreshes and retrieval changes shift baselines for entire categories overnight. That is the specific reason a pre/post comparison is not good enough: a before-and-after chart cannot separate your change from the platform's change. A same-period control can.
The noise floor: how much an untouched prompt set moves on its own
Before choosing a design, decide how much movement means nothing. We pulled every prompt cohort in our data that received no deliberate change during the study window and measured week-over-week swing in brand mention rate.
How we measured this
- 41 MaxAEO accounts that ran at least one declared holdout between 5 January and 29 May 2026
- 63 distinct holdout tests inside those accounts
- Median 140 tracked prompts per account (range 60–410), one run per prompt per engine per day
- Eight engines: ChatGPT, Gemini, Perplexity, Claude, Copilot, Grok, Google AI Mode and AI Overviews
- Just over 6 million logged responses
- Mention rate = share of responses for a prompt naming the brand; citation rate = share linking a brand-owned URL
| Engine | Median week-over-week swing (pp) | 90th percentile swing (pp) |
|---|---|---|
| Claude | 1.4 | 4.9 |
| ChatGPT | 2.1 | 6.8 |
| Copilot | 2.2 | 7.4 |
| Gemini | 2.6 | 8.3 |
| AI Overviews | 2.9 | 9.6 |
| Google AI Mode | 3.4 | 11.2 |
| Perplexity | 3.8 | 12.1 |
| Grok | 4.1 | 13.7 |
Untouched prompt cohorts, 21 weeks, 41 accounts. Percentage points of absolute change in brand mention rate.
Read the right-hand column carefully. On Grok and Perplexity, a one-week "win" of ten percentage points falls inside ordinary background noise one time in ten — with no change shipped at all. On Claude the same ten-point move is genuinely unusual.
This is why pooled, cross-engine scorecards flatter bad tests: the volatile engines supply most of the apparent movement. The pattern also tracks retrieval architecture. Engines that re-run live search on most queries (Perplexity, Grok, Google AI Mode) swing most, because their answers depend on a search index that changes hourly. Engines that lean harder on parametric knowledge and retrieve selectively (Claude, ChatGPT) swing least. The more live retrieval an engine does, the noisier your baseline, and the bigger the effect you need to prove anything.
Your minimum detectable effect must clear the p90 column for every engine you intend to make claims about. If you plan to report a Perplexity result, you need more than 12 points of movement before the word "worked" is defensible.
Four holdout designs that work when you only have one brand
Each design answers the same question — what would have happened without the change? — using a different source of counterfactual.
| Design | What acts as the control | Best when | Minimum viable size | Biggest failure mode |
|---|---|---|---|---|
| Prompt holdout | Untouched prompts in your own tracked set | Change is page- or entity-specific | 60 prompts per arm | Spillover between arms |
| Page cohort holdout | Matched pairs of your own URLs | Change is a repeatable template edit | 20 matched pairs | Poor pair matching |
| Competitor baseline | Rivals' trajectory over the same window | You cannot withhold anything | 4–6 tracked rivals | Rivals ship changes too |
| Staggered rollout | Later cohorts, before their turn | Change must eventually reach everything | 3 waves | Long calendar time |
Pick by asking one question: can the change be confined? If yes and it is narrow, use a prompt holdout. If yes and it is a template, use page cohorts. If it must hit everything eventually, stagger it. If it cannot be confined at all, fall back to the competitor baseline and label the result as directional.
Design 1: Prompt holdout — freeze part of your tracked set
Split your monitored prompts into a treated arm and a frozen arm, then ship changes that only affect the treated arm's answer path. This is the cleanest single-brand design and the easiest to get wrong.
It works when your change is narrow: one comparison page, one documentation section, one entity you are disambiguating. Entity-level work is a particularly good fit, because the treated surface is defined by a name rather than a URL — the entity disambiguation playbook for brand-name collisions is the kind of change where treated and held-out prompts stay genuinely separate.
It fails when the change is site-wide — new schema everywhere, a homepage rewrite, a fresh round of third-party placements — because there is no prompt left that the change cannot reach.
Assign arms before you look at baselines, and freeze the assignment in writing. Post-hoc reassignment ("this prompt was clearly always going to move") is the single fastest way to manufacture a result. In our review it was the rarest failure mode (9% of tests) and the most damaging, because it is invisible in the final readout.
Design 2: Page cohort holdout — matched pairs of your own URLs
Pick pages that already behave similarly, pair them, and treat one of each pair. Twenty matched pairs is a workable floor; forty is comfortable.
Match on the things that actually predict citation behaviour:
- Current citation rate (the strongest single predictor)
- Page type — comparison, docs, glossary, landing
- Topical cluster
- Publish age
- Inbound link count
Do not match on traffic. Organic sessions and AI citation rates diverge sharply, and a pairing built on traffic will look balanced while being anything but. High-traffic pages skew toward transactional queries that AI engines rarely cite; low-traffic docs pages are cited constantly.
This design suits template-level work. If you are adding a summary block, a specification table or a definitions section across a content type, the untreated half of each pair tells you what the template edit was worth. Documentation is the highest-yield place to run it — developer docs are a distinct citation source with their own retrieval behaviour, and template changes there move numbers faster than on marketing pages.
Design 3: Competitor baseline — difference-in-differences without a control of your own
When you cannot withhold anything, borrow a counterfactual from the category. Track four to six rivals across the same prompt set, then compare your change over the window against their change over the same window. The estimate is your delta minus theirs — difference-in-differences, the design Card and Krueger made standard in applied economics.
This is the weakest of the four designs and the most often available. It assumes your rivals were not simultaneously running their own AEO programme, which in a competitive category is optimistic. It also assumes you picked the right rivals — the ones the models actually place beside you, not the ones on your sales battlecards. Build that list from observed co-mentions rather than internal assumption, using the method in building an AI-native competitive set.
One thing this design does uniquely well: detect category-wide shocks. If every tracked rival moves the same direction in the same week, a model update or index refresh happened and none of your own numbers that week mean anything. Use it as that alarm, and as a sanity check on your primary test — not as your primary evidence.
Design 4: Staggered rollout — every cohort is its own control
Roll the change out in waves. Wave one gets it in week one, wave two in week three, wave three in week five. Until a wave is treated, it is a control; after it is treated, it contributes to the effect estimate.
Statisticians call this a stepped wedge, and it has one property the other designs lack: nothing is permanently withheld, which usually ends the internal argument about deliberately under-optimising part of the site. It also separates the change from the calendar, because each wave's "before" and "after" sit at different points in time. If a model update lands in week four, it hits treated and untreated cohorts in a pattern the analysis can absorb.
The cost is calendar time. A three-wave rollout with a two-week observation window per wave takes eight to ten weeks minimum. That is the design's real constraint — and the reason it usually loses internal arguments to a faster, weaker test.
How far apart do holdout prompts need to be? The contamination radius
Every prompt holdout leaks. The practical question is how much, and the answer depends on how much source overlap exists between arms — not on how different the prompts sound to a human.
We measured leakage as the share of the treated arm's lift that also appeared in the held-out arm, grouped by the relationship between the holdout prompt and the treated page.
| Relationship of held-out prompt to the treated page | Median share of treated effect that leaked | Usable as control? |
|---|---|---|
| Same cluster, same core entity | 61% | No |
| Same cluster, different entity | 34% | Weak |
| Adjacent cluster, shared category prompt ("best X tools") | 18% | Borderline |
| Unrelated cluster, no shared cited URL at baseline | 4% | Yes |
| Different product line entirely | 2% | Yes |
63 holdout tests, 41 accounts, January–May 2026.
That gives a rule you can apply in an afternoon, and it does not require judgement about semantics:
A held-out prompt is only clean if the URL you are about to change does not appear in that prompt's cited-source set at baseline. If it does, the prompt is in the blast radius. Move it to the treated arm or drop it from the test.
Run this as a set operation, in four steps:
- Export cited sources for every prompt in your monitoring tool, across the full baseline window.
- Build the set of URLs you intend to change, including any page that links to them from within your own site.
- For each candidate control prompt, compute the intersection between its cited-source set and that treatment set.
- Require under 10% overlap for control-arm membership. Drop or reassign the rest.
Most teams find that roughly a fifth of their intended holdout fails this check — usually the prompts they were most confident about, because those are the prompts where their own pages already rank. The same source-overlap view explains why AI engines cite competitor pages instead of yours, and it is the most useful single export any ai visibility tool can give you.
Deep research modes make this worse. Multi-step agents that fan out across a dozen sub-queries per answer pull in far more of your site than a single-shot answer does, which widens the blast radius of any change. If a meaningful share of your tracked prompts run through deep research modes, tighten the overlap threshold to 5% and expect a smaller usable control arm.
How to size an AEO holdout test
Sizing is where most single-brand tests fail before they start, because repeat runs of the same prompt are not independent observations. They cluster: a prompt that never mentions you tends to keep not mentioning you.
In our data, intra-prompt correlation produced a design effect of roughly 2.5 to 4. Treat each repeat run as worth about a third of an independent observation. Seven daily runs of one prompt buy you roughly the statistical power of two or three distinct prompts — not seven.
| Prompts per arm | Runs per prompt per week | Realistic detectable lift at 80% power |
|---|---|---|
| 30 | 7 | ~15 pp |
| 60 | 7 | ~11 pp |
| 120 | 7 | ~7.5 pp |
| 240 | 7 | ~5.3 pp |
Two-proportion comparison at a 40% baseline mention rate, design effect 3.0.
Two consequences follow. First, a 30-prompt holdout can only detect changes so large you would have noticed them anyway. Second, adding prompts beats adding runs — doubling your prompt set buys far more power than doubling your daily frequency, because new prompts add independent information and repeat runs mostly re-measure the same cluster.
Baseline rate matters too. The table above assumes a 40% baseline mention rate, which is near the worst case: proportions are noisiest around 50%. A brand starting at 10% needs roughly a third fewer prompts to detect the same absolute lift, so early-stage brands can run smaller tests than incumbents can.
For scale reference on what a serious public experiment looks like: Search Engine Land's fake-brand AI search experiment ran 825 distinct prompts and logged 15,835 responses in its first month.
A six-week protocol you can copy
- Week 0 — pre-register. Write down the hypothesis, the arms, the primary metric (one metric — mention rate or citation rate, not both), the engines you will judge on, and the stopping date. Store it somewhere with a timestamp.
- Week 0 — run the overlap check. Export cited sources per prompt. Remove any intended control prompt whose source set overlaps your treatment URLs by more than 10%.
- Weeks 1–2 — establish baseline. Collect at least ten days of pre-change data on both arms. Confirm the arms track each other; if they already diverge, re-split.
- Week 3 — ship one change. One change, to the treated arm only, on a single day. Log the exact timestamp and diff.
- Weeks 3–4 — wait out the lag. Do not read results yet. In our data, median time-to-first-observed-change ran about four days on the fastest retrieval-heavy engines and close to three weeks on the slowest. Reading early biases every conclusion toward Perplexity and Grok.
- Weeks 5–6 — measure and call it. Compare the change in the treated arm against the change in the held-out arm, per engine. Report the difference-in-differences, not the treated arm's raw movement.
Freeze everything else for the full six weeks. A test with two changes in it has no interpretable result — and 19% of the tests we reviewed had exactly that problem.
Who runs it, and what it costs
A holdout test is roughly 6–10 hours of analyst time spread over six weeks: two hours on the overlap check and pre-registration, an hour a week of monitoring, and two to three hours on the readout. The hard part is not the labour — it is the organisational discipline of leaving a cohort untouched while someone senior asks why you are not optimising it.
That is a structural question as much as a methodological one. Agencies are usually paid for shipped changes, not withheld ones, which makes holdouts a predictable point of friction; the tradeoffs are laid out in who should own your AI search visibility. If you use an external partner, put the holdout arm in the scope of work explicitly, before the engagement starts.
Reading the result without fooling yourself
Report per engine, always. Pooling eight engines into one "AI share of voice" number hides the fact that a single volatile engine can carry an entire apparent win. If the effect only exists on Grok, you have found noise; if it appears consistently on three or more engines, you have found something.
Compare deltas, not levels. The number that matters is (treated after − treated before) − (control after − control before). A treated arm that rose eight points while the control rose seven has bought you one point, not eight.
Here is the case that convinced us to mandate concurrent controls. A B2B data-infrastructure vendor split 180 prompts into 90 treated and 90 held out, shipped a documentation restructure, and saw the treated arm climb 9 points in mention rate over the following week. The change looked decisive. Their untouched holdout arm rose 8 points in the same week — a model update had lifted the whole category. Net effect: one point, well inside noise. Without the holdout, that documentation project would have been declared a winner and copied across four more content types, at an estimated six weeks of writer time.
Write down the negative results too. In the 63 tests we reviewed, 22 produced an effect that could not be distinguished from noise after the control was subtracted. Those are not failed tests. They are the tests that stopped a team from scaling something that did nothing — which is most of the value a holdout produces.
Failure modes, ranked by how often they ruined a test
Across the 63 holdout tests we reviewed, these are the problems we actually found. Percentages exceed 100 because most broken tests had more than one.
| Failure mode | Share of tests affected | Tell | Fix |
|---|---|---|---|
| Contamination between arms | 38% | Control arm moves nearly as much as treated | Run the source-overlap check before splitting |
| Underpowered prompt set | 31% | Fewer than 60 prompts per arm | Add prompts, not runs |
| Pre/post with no concurrent control | 24% | Result is a single before-and-after chart | Add any control, even a competitor baseline |
| Window shorter than engine lag | 22% | Called inside 14 days of shipping | Pre-register the stopping date |
| Multiple changes shipped mid-test | 19% | Change log shows two or more edits | Freeze the treated surface |
| Pooled reading across engines | 17% | One engine supplies most of the delta | Report per engine |
| Post-hoc arm reassignment | 9% | Arm definitions edited after week 1 | Freeze arms in writing at week 0 |
Contamination and underpowering together account for most wasted tests, and both are fixable in the hour before the test starts.
When not to run a holdout at all
Skip the holdout when the change is unambiguous and universal. Fixing a robots.txt rule that blocks AI crawlers does not need a control group — you can verify it directly in server logs. The same goes for correcting factually wrong information about your company in a widely cited source: that is reputation repair, not an experiment.
Skip it when withholding is unethical or commercially silly. Nobody should leave a security-relevant documentation error unfixed for six weeks in the name of clean measurement. The same applies when AI answers are actively steering your customers elsewhere — if engines are recommending alternatives to your brand on high-intent prompts, fix it everywhere and measure with a competitor baseline instead.
Skip it when you genuinely cannot get 60 clean prompts per arm. An underpowered holdout is worse than no holdout, because it produces a number that carries the authority of an experiment and the reliability of a coin flip. In that situation, use the competitor baseline as a directional check and be honest in the readout about what it can and cannot support.
Frequently asked questions
How long should an AEO holdout test run?
Six weeks end to end: two weeks of baseline, one shipping day, then at least three weeks of observation. Shorter windows systematically favour engines with fast retrieval and miss slower ones entirely, which biases the result toward whichever platforms happen to refresh quickest.
Can I run more than one holdout test at a time?
Yes, if the tests touch non-overlapping source sets. Run the overlap check across every treatment URL in every active test — if two tests share cited sources for the same prompts, their effects are entangled and neither is interpretable. In practice most single-brand teams can sustain two concurrent tests, not five.
What if my control arm outperforms the treated arm?
Take it seriously rather than rerunning until it flips. A negative result usually means one of three things: the change genuinely hurt, the arms were contaminated in the reverse direction, or the treated arm regressed to the mean from an unusually high baseline. Check the baseline levels first — arms split when one was at a temporary peak will show a fake negative.
Do I need a monitoring platform, or can I do this by hand?
Manual runs work for a one-off check and collapse under a real test. Sixty prompts across eight engines at daily frequency is 480 observations a day, and the cited-source export that makes contamination checking possible is not something you can reconstruct from screenshots. Whatever llm brand tracking you use, it needs per-prompt source data and stable cohort definitions — a gap worth checking before you commit, as we cover in our comparison of MaxAEO and Profound.
Does a holdout test tell me why the change worked?
No. It tells you whether it worked. Attribution to a mechanism — schema, third-party sources, page structure, entity clarity — requires either a follow-up test that isolates one component or a qualitative read of which sources the engines started citing after the change.
Can I use a geographic holdout instead?
Sometimes. If you operate in several markets with separate content, hold out one market and treat the others. It works when the markets have genuinely separate URLs and source ecosystems, and fails when engines pull the same global pages regardless of the asker's location — which is common for English-language B2B. Check whether your tracked prompts in the held-out market already cite the pages you plan to change; if they do, geography is not buying you separation.
How do I explain a holdout to a leadership team that wants everything optimised now?
Frame it as insurance against scaling a non-effect. The documentation case above cost one team nothing because the holdout caught it; without one, they would have spent six weeks replicating a change worth a single point. If the argument still stalls, use a staggered rollout — nothing is permanently withheld, only sequenced.