Dream Roofing appeared in 70.6% of the answers from one model. In the other, it appeared in 10%.
Monkeys Roofing moved in the opposite direction, from 19.4% to 76.8%.
Those numbers came from the same question: “Who's the best roofer in the Eastern Suburbs of Sydney?” On 14 November 2026, we asked it 500 times using GPT-5.4 mini and another 500 times using GPT-6 Luna, through the OpenAI API with web search enabled.
We wanted to know what a local business could reasonably learn from appearing in an AI answer. The results gave us a much better question to ask: how often does a business appear, under which conditions, and how much does that picture change when the model changes?
This was an API study, not a test of the ChatGPT app. The distinction matters throughout the results.
Share of 500 answers per model that included each business as a recommended option. Selected businesses, not a ranking of roofing quality. Source: Serse study data, 14 November 2026.
The most frequent recommendation depended on the model
Here are six of the businesses that appeared in both batches. A business counted once per answer, even if the model mentioned it several times.
| Business | GPT-5.4 mini | GPT-6 Luna |
|---|---|---|
| Dream Roofing | 353 of 500 · 70.6% | 50 of 500 · 10.0% |
| Monkeys Roofing | 97 of 500 · 19.4% | 384 of 500 · 76.8% |
| APT Roofing | 179 of 500 · 35.8% | 267 of 500 · 53.4% |
| All Sydney Roofing | 250 of 500 · 50.0% | 36 of 500 · 7.2% |
| Marshall Roofing | 248 of 500 · 49.6% | 44 of 500 · 8.8% |
| AAA Metal Roofing | 190 of 500 · 38.0% | 108 of 500 · 21.6% |
Dream Roofing's appearance rate differed by 60.6 percentage points between the batches. Monkeys Roofing's differed by 57.4 points in the other direction. No business appeared in every answer from either model.
Imagine reporting AI visibility for Dream Roofing using only the first batch. You would describe a business with a strong presence in this particular question. Using only the second, you would describe a much less frequent recommendation. Both reports could accurately describe their own samples while leaving the reader with very different impressions.
That is why a model label belongs beside the result. “We appeared in 70.6% of answers” is incomplete without the prompt, model, date and number of answers behind it.
We did not observe a live model rollout or measure a business before and after an update. These were two separate batches run that afternoon, with live web retrieval. The differences are real in the saved answers; the experiment does not isolate every cause of them.
Being mentioned and being named first gave different results
Dream Roofing was the most frequent extracted recommendation in the GPT-5.4 mini batch. It was also the first extracted business in 118 answers, or 23.6% of the batch. Its 70.6% appearance rate and 23.6% first-mention rate describe different things.
The distinction is even clearer in the Luna batch. Monkeys Roofing appeared most often, but APT Roofing was named first more often: 229 answers for APT, compared with 213 for Monkeys.
Even “named first” needs careful reading. An answer can introduce several businesses, then make a different recommendation for roof repairs, slate work or a full replacement. Our measure records the order in which recommended options first appeared in the text. It does not turn that order into an independent judgement of quality.
For an agency report, we would show how often a client appears and include a few complete answers to explain the context. A single position number loses too much of that detail.
The citation mix changed sharply
The two batches also presented very different supporting sources.
GPT-5.4 mini cited hipages, including its subdomains, in 411 of 500 answers: 82.2%. In 362 answers, or 72.4%, every recorded citation pointed to hipages.
Luna cited hipages in 147 answers, or 29.4%. Only one Luna answer cited hipages alone.
Percentage of answers with at least one recorded citation to each source. An answer can cite several sources, so the percentages do not sum to 100. hipages includes its subdomains; the other rows use the specified hostnames.
Luna's frequently cited sources included the NSW licence register at verify.licence.nsw.gov.au in 240 answers, Australian Roofers in 234, APT Roofing's website in 212, and Monkeys Roofing's website in 183.
There is a tempting story here: one model relied on a directory while the other checked businesses more thoroughly. The citation data supports a narrower finding. The mix of sources shown to the reader was different. It does not tell us how much weight the model gave each source or prove that it verified a particular contractor's licence.
There is another boundary worth keeping clear. Our files contain the URLs cited in the answers, not a complete record of every page retrieved. OpenAI documents those as different outputs: inline citations are a selection, while the separate sources field can return the broader set of consulted URLs. Our collection script did not request that full source list. OpenAI web search documentation
For a local business, the useful observation is that a report built around one directory could miss a large part of the citation picture in another model. Before investing in a particular listing or rewriting a website, inspect the sources across the models and questions you are actually measuring. This experiment did not test whether buying a listing or changing a page would improve visibility.
Luna repeated its shortlists more often
There was meaningful variation within each model as well as between them. But the degree of variation was quite different.
| Measure within each 500-answer batch | GPT-5.4 mini | GPT-6 Luna |
|---|---|---|
| Distinct business names after normalisation | 64 | 15 |
| Average recommended options per answer | 4.42 | 2.37 |
| Distinct sets of businesses | 240 | 94 |
| Distinct lists including their order | 334 | 126 |
| Pairs containing exactly the same business set | 1.28% | 8.31% |
| Pairs containing the same businesses in the same order | 0.49% | 5.54% |
For the last two rows, we compared every possible pair of different answers within each batch: 124,750 pairs per model. We compared the extracted business lists, not whether the full answer text was identical.
Luna had a narrower pool of names and shorter lists. Its most common business set, APT Roofing and Monkeys Roofing, appeared in 102 answers. GPT-5.4 mini returned that same pair as its most common set too, but in only 29 answers.
So the study is not simply a story about randomness. Repeated sampling revealed strong patterns, and Luna's shortlists were more concentrated in this test. A single response would hide both the variation and the concentration.
The long tail differed too. Of the 64 normalised business names in the mini batch, 38 appeared in fewer than five answers, below 1%. Five of Luna's 15 names fell below that threshold. These are counts of names returned in our sample. We did not compile a census of every roofer operating in the Eastern Suburbs, so they cannot tell us what share of the local market was invisible.
How this relates to the SparkToro and Gumshoe research
This study was inspired by Rand Fishkin and Patrick O'Donnell's research with SparkToro and Gumshoe. Their initial experiment collected 2,961 responses across 12 prompts and three AI tools, with a much broader set of users and conditions than ours. They found that recommendation lists varied substantially, while repeated appearances could still reveal a useful pattern of brand visibility. Read the original research
Our experiment is a narrow local follow-on, not a replication. We held one prompt constant and compared two API model configurations. Its contribution is a concrete Sydney example of how much the measured picture can differ across models, even within one afternoon.
What we would change in an agency visibility report
The practical change is to make the measurement inspectable.
Start with an exact question and show the reader how it was tested. For example, a defensible result from this dataset is: “Monkeys Roofing appeared as a recommended option in 384 of 500 GPT-6 Luna API answers to this question on 14 November 2026, with web search enabled.” That sentence contains the denominator and conditions needed to interpret the percentage.
Keep each model's results separate. An average across these two batches would conceal the largest finding in the study. If a model configuration changes between reporting periods, flag the change and establish a comparable baseline before attributing movement to SEO work.
Keep the source evidence separate from the business appearance measure too. A citation to a directory does not necessarily name your client. A mention of your client does not necessarily link to their website. Both observations can be useful, but they answer different questions.
Then broaden the questions. Our prompt did not say whether the person needed emergency leak repair, a metal roof replacement or heritage slate work. A more useful local research plan would include those different needs and several locations, with results reported by question. Repeating one prompt 500 times tells us much more about that prompt; it does not make it representative of every customer.
Finally, keep visibility alongside actual business outcomes. This study contains no enquiries, booked jobs or revenue data. A change in appearance rate can justify investigation, but it cannot establish that a business has gained customers.
How we ran and checked the study
The collection used the OpenAI Responses API with the identical prompt supplied as a new request each time. The script passed no prior conversation. Web search was available, and all 1,000 saved answers contained citations.
| Method detail | Recorded configuration |
|---|---|
| Question | Who's the best roofer in the Eastern Suburbs of Sydney? |
| Collection date | 14 November 2026 |
| GPT-5.4 mini response model | `gpt-5.4-mini-2026-03-17` |
| GPT-6 Luna response model | `gpt-6-luna` |
| Sample | 500 saved answers per model |
| Mini collection window | 2:26–2:33 pm Sydney time |
| Luna collection window | 2:40–3:04 pm Sydney time |
| Temperature, reasoning effort and user location | Not explicitly set in the collection request |
| Extraction | A separate GPT-5.4 mini pass identified recommended business options in their first-appearance order |
| Name handling | Supplied merge maps normalised spelling, case, legal suffixes and selected variants; duplicates within an answer counted once |
Directories and government services were excluded from the business lists. A name mentioned only in a cautionary passage could be excluded by the recommendation-extraction step. These figures therefore describe extracted recommendations and options, rather than every literal occurrence of a business name.
We reproduced the reported business counts, appearance rates, list sizes and list-consistency totals from the saved extraction files and merge maps. We recalculated citation rates from the raw answer records. All 1,000 raw records had unique run IDs with matching extraction records. A text-presence check found each extracted name in its corresponding answer after punctuation normalisation, and we inspected sample answers against their extracted lists.
That validates the calculations against the supplied records; it is not a complete human re-annotation of all 1,000 answers. The extraction model can still make judgement errors. One unresolved identity example was kept separate: “First Slate Roofing” in the mini batch was not assumed to be “First Class Slate Roofing” in the Luna batch.
The study covers one question, one trade, one region and one day. The batches were sequential rather than interleaved, and web results were not frozen. We did not test consumer-app personalisation, measure the quality of the roofing businesses, or establish whether these frequencies will persist. The percentages describe this sample, not guaranteed probabilities for a future customer.
The verified analysis, per-answer recommendation lists and cited URLs and method notes accompany this article so the calculations can be inspected. The public data files do not include the full generated narratives. The recorded recommendations are model output, not endorsements or independently verified claims about the businesses they name.
Turning an AI answer into useful SEO work
A business appearing once is a reason to investigate. Repeated answers, saved evidence and clearly stated conditions make that investigation more useful.
That is the approach behind Serse's AI visibility and citation research: inspect the answers and their context, then decide what deserves attention on the website. A screenshot is a starting point. The useful work comes from understanding what it represents.