Tool
How many times do you need to ask an AI engine before a change in citations is real?
One answer is one sample. Enter how many questions you ask, on how many engines, how many times. You get the range your citation share sits in, and the number of runs it takes before a change stands out from noise.
Updated
Your panel
What the numbers support
Share 67.3%, interval 59.6% to 74.2%. 7 runs per question needed.
- Citation share this round
- 67.3%, with a 95% interval from 59.6% to 74.2%.
- Smallest change this panel can tell from noise
- a rise to 81.1% or a fall to 51.8%, comparing two rounds of 156 answers.
- To detect a move from 67.3% to 77.3%
- 314 answers in each round, which is 7 runs of each question on each of 4 engines.
The address bar keeps your five numbers, so you can bookmark a panel or send it to a colleague. "Copy link to these numbers" copies that address. The numbers stay in the link and in your browser. The page sends them nowhere.
The numbers it starts with
They are ours. In launch week we asked four engines 13 questions that named us, three times each. 105 of the 156 answers cited synapsereality.io. That is 67.3%, and the calculator puts the 95% interval at 59.6% to 74.2%. A second round of the same size would have to reach 81.1% before the rise stood out. The full run is in AEO vs SEO: what we measured on our own site.
Questions that didn't name us were cited once in 204 answers, or 0.5%. With rounds of 204, only a rise to about 5% would stand out. A rise of one point would need 1,538 answers in each round. On those 17 questions and four engines, that is 23 runs of every question.
How it works it out
Each answer either cites your site or it doesn't. The share is cited answers divided by all answers, and the number of answers is questions times engines times runs.
The range around your share
The Wilson score interval at 95% confidence, as given in the NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.4.1. NIST notes that its lower limit can't go below zero, which matters at the low shares citation panels see.
The change you can detect
Two rounds of the same size, compared with the two-proportion test in NIST section 7.3.3. The answers needed in each round come from the sample-size formula for that test in Fleiss, Levin and Paik, Statistical Methods for Rates and Proportions. NIST cites the same book for its one-sample version. The test is two-sided at 5%, with an 80% chance of catching a change of the size you enter.
The code is in our site's repository and checked against NIST's worked example: a share of 0.13 in 200 gives a one-sided lower limit of 0.09577.
What the result can't tell you
The formulas treat every answer as an independent draw. Answers to the same question on the same engine aren't. When we re-asked five questions three times on three engines, all 15 pairs landed within one citation of the first run. Repeats that agree with each other add less information than new questions do. Read the result as the least you need, and add questions before you add runs.
It does not predict whether an engine will cite you, or what a change to your site will do. It says how big a difference has to be before your own measurements can show it.
answer-engine-benchmark, our open-source tool, runs a panel like this and records every answer, so the counts you enter here can be traced.