Someone in the other thread asked whether anyone had compared the same prompt set across platforms. That is the useful question, and it never really got answered.
I have not completed a clean four-platform run that I can honestly present as benchmark results. What I can share is the manual protocol I built to make such a run comparable. It needs no monitoring product or browser extension, and it avoids the biggest trap in these comparisons: turning unlike answers into one precise-looking score.
The unit is simple:
one prompt × one platform × one pass
Keep every observation at that level until collection is finished.
1. Build a prompt pack that can survive comparison
Do not start by copying 50 loosely related prompts from a keyword tool. A smaller pack with explicit intent is easier to rerun and audit.
I would start with 18 prompts in three groups:
- Category discovery: six prompts such as “What are the best tools for [job]?”
- Problem diagnosis: six prompts such as “Why is [type of company] missing from AI recommendations?”
- Direct comparison: six prompts such as “Which is better for [use case], Brand A or Brand B?”
Before opening any platform, give every prompt an ID and freeze the exact wording. If one prompt turns out to be ambiguous, do not quietly edit it halfway through. Create a new ID and rerun the revised version everywhere.
Also freeze four conditions in a small run header:
- language;
- region or account location;
- signed-in versus signed-out state;
- whether follow-up questions are allowed.
Those details matter because a prompt is not the only input. Account context and follow-ups can change the answer shape enough to ruin a direct comparison.
2. Run the four platforms in a controlled sequence
Use ChatGPT, Perplexity, Gemini and Grok in the same 48-hour window. Run the complete prompt pack once on every platform, wait several hours, then repeat the full pack for pass two.
The sequence should be interleaved by prompt, not completed platform by platform over several days:
- Run P01 on all four platforms.
- Run P02 on all four platforms.
- Continue through P18.
- Repeat the same order for pass two.
This reduces the chance that a news event or index update affects only one platform’s batch. It also makes drift visible: if P07 changes between passes on Gemini but remains stable elsewhere, that disagreement stays attached to P07 instead of disappearing inside an average.
For 18 prompts, four platforms and two passes, the design produces 144 possible observations. That is the planned sample size, not a claim that those observations have already been collected.
3. Capture the answer before interpreting it
For the first pass through the sheet, record only observable fields:
- Mentioned: whether the target brand appears,
yesorno; - Cited domains: every visible source domain, or
none; - Answer saved: a screenshot or pasted answer reference;
- Collection time: timestamp for that observation.
Do not score sentiment, rank position or source quality while collecting. Those judgments are useful later, but mixing collection and interpretation creates a moving standard. The tenth answer gets judged differently from the first because you have already seen a pattern.
There are two platform-specific edge cases worth recording explicitly.
First, if a platform shows no citations, write none; do not infer sources from phrases in the answer. Second, if a platform cites the same domain multiple times, retain the domain once for source-mix analysis but keep the raw answer so citation frequency can be reviewed separately.
4. Turn cited domains into a source map
After collection, classify each visible domain into one of five broad buckets:
- competitor website;
- directory or software marketplace;
- community;
- media or editorial site;
- video platform.
Keep the buckets broad on purpose. The first question is not whether a source is a review site or an affiliate review site. It is whether one answer engine repeatedly depends on communities while another prefers vendor pages or editorial lists.
For each prompt, build a four-column source map:
| Prompt | ChatGPT | Perplexity | Gemini | Grok |
|---|---|---|---|---|
| P01 | Mention + source buckets | Mention + source buckets | Mention + source buckets | Mention + source buckets |
That row supports concrete questions:
- Which platforms mention the brand in both passes?
- Which platforms omit it in both passes?
- Which source bucket dominates each platform?
- Which domains recur for the same prompt?
- Where does the platform expose no visible source at all?
This is already more actionable than a vendor ranking. If community sources recur only on Grok, the publishing opportunity is different from a case where editorial lists recur on Perplexity.
5. Separate stable differences from answer drift
One answer is an observation, not a finding. Use pass two as a minimum stability check.
Mark a pattern stable only when it repeats under the same prompt and platform. Examples:
- the brand appears in both ChatGPT passes and neither Gemini pass;
- community sources dominate both Grok answers for P04;
- the same editorial domain is cited in both Perplexity passes for P11.
Mark it unstable when the two passes disagree. Do not break the tie by choosing the answer that supports your preferred story.
A compact decision table helps:
| Pass 1 | Pass 2 | Label | What you may say |
|---|---|---|---|
| Mentioned | Mentioned | Stable mention | Repeatedly mentioned in this test window |
| Not mentioned | Not mentioned | Stable absence | Repeatedly absent in this test window |
| Mentioned | Not mentioned | Unstable | Answer drift; rerun before acting |
| Source bucket A | Source bucket B | Unstable source mix | Retrieval pattern is not yet repeatable |
Even two passes are only a screening rule, not proof of a permanent platform behavior. The defensible conclusion is limited to this prompt pack, account context and 48-hour window.
6. Keep platform results separate
Do not add ChatGPT, Perplexity, Gemini and Grok into one visibility score. They expose different numbers of sources, use different retrieval systems and return different answer formats. A blended score hides the exact difference the benchmark is meant to reveal.
Report four platform-level outputs instead:
- stable mention rate by intent group;
- stable absence rate by intent group;
- recurring source buckets;
- recurring domains for the same prompt.
Only compare like with like. A platform with no visible citations should not receive a lower “source quality” score than one that exposes citations; the fields are not comparable.
Copyable collection sheet
| Prompt ID | Intent | Prompt | Platform | Pass | Mentioned | Cited domains | Source buckets | Timestamp | Stability |
|---|---|---|---|---|---|---|---|---|---|
| P01 | Discovery | What are the best tools for [job]? | ChatGPT | 1 | Yes/No | domain.com or none | Community | YYYY-MM-DD HH:MM | Pending |
| P01 | Discovery | What are the best tools for [job]? | ChatGPT | 2 | Yes/No | domain.com or none | Community | YYYY-MM-DD HH:MM | Stable/Unstable |
If anyone here runs this exact four-platform, two-pass format, I would be interested in comparing one thing: do your source buckets remain stable for the same prompt, or does the source mix move as much as the brand mentions do?