Run-to-run noise: what one AI answer tells you and what three do

Broadcastwell Research ran 140 buyer questions three times each on ChatGPT, Claude, Perplexity, Google AI Overviews and Google AI Mode on 22 to 23 Sep 2026 (UTC), then asked how often one run would have told you something the other two did not. Study 3 of the September 2026 series. Data and report CC BY 4.0.

Key figures

1,785 of 8,055 single-run vendor namings (22.2%) were not repeated by both of the other two runs of the same question on the same engine.

Base: 8,055 namings in 14,192 question x engine x vendor cells with three valid runs. Strict variant: 9.1% of namings were contradicted by both other runs (733 of 8,055). Cell variant: 37.6% of cells that named a vendor at all did not do so in all three runs (1,259 of 3,349).

What we found

• The three runs agreed on 12,933 of 14,192 cells (91.1%), and on 2,090 of 3,349 cells where the vendor was named at all (62.4%).
• Across active cells, ChatGPT disagreed across runs in 159 of 580 (27.4%), Claude in 253 of 812 (31.2%), Perplexity in 200 of 653 (30.6%), Google AI Overviews in 302 of 599 (50.4%) and Google AI Mode in 345 of 705 (48.9%).
• A single run called 57 to 59 of 286 vendors never named across all questions and engines; three runs called 46. Per engine, one run called on average 16 to 31 more vendors absent than three runs did.
• 28 of 286 vendors (9.8%) were absent in some runs and present in others. Ranked from one run, a vendor's rank within its category differed from its three-run rank in 421 of 858 vendor-run comparisons (49.1%, competition ranking; with ties broken in the three-run order, 243 of 858 comparisons differed, 28.3%); the top vendor changed in 3 of 42 single-run category rankings.
• Across 286 vendor records, the mean Wilson 95% interval around the mention rate was 16.9 percentage points wide with one run and 9.4 with three (rounded to one decimal).

Chart 1: was a single run's naming repeated?

Namings not repeated by both other runs: ChatGPT, 233 of 1,496 (15.6%); Claude, 362 of 2,039 (17.8%); Perplexity, 283 of 1,642 (17.2%); Google AI Overviews, 431 of 1,322 (32.6%); Google AI Mode, 476 of 1,556 (30.6%).

Chart 2: absence is overstated by a single run

Among 286 checked vendors, one run calls on average 16 to 31 more vendors absent per engine than three runs do (means across the three single runs). On ChatGPT, 115 of 286 vendors were never named across three runs against 131.3 of 286 per single run on average; on Google AI Mode, 101 of 286 against 131.7 of 286.

Chart 3: three runs cut the interval almost in half

The widest case was Jira in project management: named in 16, 22 and 25 of 50 answers across the three runs, a single-run mention rate from 32.0% to 50.0%. Across three runs Jira was named in 63 of 150 answers (42.0%; Wilson 95% interval 34.4% to 50.0%).

Why it matters

A single screenshot is a sample of one. A vendor missing from one answer is a weaker signal than a vendor present in one: 6,270 of 8,055 namings (77.8%) were repeated by both other runs, while one run added 11 to 13 vendors to the three-run never-named count of 46 (23.9% to 28.3% overstatement). Google's two surfaces varied most in this data; this study does not say why.

Method

Source: The Absence Index, release 2026-09 (Broadcastwell), DOI 10.5281/zenodo.22907695, CC BY 4.0. 14 categories, 286 checked vendors, 140 questions, five engines, three scheduled repeats per engine and question, captured 22 to 23 Sep 2026 (UTC); 2,095 valid target answers. Matching rule as published: names and listed aliases match case-insensitive word boundaries; plain-word vendors require a listed qualifier within five words in the same sentence or the canonical domain in the cited URLs; Wilson 95% intervals. Broadcastwell's published method (broadcastwell.com/methodology, version 1.1, 8 September 2026) covers US English and United States location. It is a controlled sample, not a replica of any one buyer's screen. Every figure was recomputed from the raw answers under the published matching rule and cross-checked against the Index: 0 of 14,300 cells differ. Reproduction materials are in the linked dataset. A cell is one question x engine x vendor; 14,192 cells have three valid runs and 108 that lost a run to an excluded answer are left out. This study measures within-day repeat variation, not change over time.

Limitations

Three runs is a small sample per cell. Same-day runs understate longer-horizon variation. A controlled sample, not every buyer's screen. The question set is fixed. Single-run ties are common among rarely named vendors; rank changed in 421 of 858 vendor-run comparisons (49.1%) under competition ranking and in 243 of 858 (28.3%) with ties broken in the three-run order. Checked vendor lists come from buyer directories. Plain-word vendors need a qualifier or domain to count. Excluded answers drop 108 cells. No causal claim about why engines vary. Engines change their models and retrieval without notice.

Dataset

Report, charts and CSVs: https://doi.org/10.5281/zenodo.22982432 (CC BY 4.0). Files: run_cells_2026-09.csv (14,192 rows), vendors_by_run_2026-09.csv (286 rows), README, scripts.

How to cite

Broadcastwell Research. (2026). Run-to-run noise in AI search answers: three runs of 140 buyer questions on five engines (Version 1.0) [Data set and report]. Broadcastwell. https://doi.org/10.5281/zenodo.22982432

Read the current Answer Operations scopes and prices. Each service has a defined deliverable, delivery clock and refund term.