If you ask ChatGPT for the “best headphones” 100 times, you will almost never get the same list twice. Not slightly different. Not one brand swapped out. Statistically, almost never the exact same response.
Yet brands are spending millions of dollars trying to rank in AI tools and understand their AI visibility. That is where the problem begins.
There is a growing industry built around AI visibility. These companies promise to show brands where they rank in ChatGPT, how often they appear in AI answers, and whether competitors show up more frequently. Large companies are investing real budget into these reports. The assumption is simple: if you can measure AI ranking, you can improve it.
But until recently, no one had tested whether AI rankings are even consistent enough to measure.
Watch the video below👇
The Large-Scale Experiment That Tested AI Visibility Consistency
Recently, Rand Fishkin and Gumshoe.ai ran a large experiment to answer that question. They recruited 600 volunteers who used 12 different prompts across three major AI tools: ChatGPT from OpenAI, Claude from Anthropic, and Google AI. In total, they gathered nearly 3,000 responses.
The prompts covered topics like:
- "best chef's knives under $300"
- "best digital marketing consultants"
- "best cancer hospitals on the West Coast"
- "best headphones for travel"
Each prompt was run dozens of times to test consistency.
The central question was simple. If you run the exact same prompt 100 times, how often do you get the exact same list back?
The answer was less than 1 percent of the time. And if you look at the exact order of brands in those results, that consistency dropped to less than one in a thousand.
That means the brands change, the order changes, and even the number of recommendations changes. Sometimes the model gives three results. Sometimes it gives ten. These systems are probability engines. They are designed to produce variation. Trying to track a fixed ranking position inside an AI response does not make statistical sense.
Why AI Ranking Positions Are Impossible To Track
One of the most serious examples in the study focused on cancer hospitals. When asked for the best cancer hospitals on the West Coast, one organization, City of Hope, appeared in 69 out of 71 responses. That is 97 percent visibility. However, it was only ranked number one in 25 of those responses.
This does not prove it is objectively the best hospital. It shows that the language model strongly associates that brand with that topic. That distinction matters. AI models reflect patterns and associations in their training data. They do not operate like traditional search engines with stable ranking positions.
When a tool promises you a specific ranking in AI, it is presenting a snapshot of a system that is constantly shifting.
AI Visibility Percentage May Be the Only Useful Metric
While ranking position appears unstable and noisy, visibility percentage across many runs of the same prompt may carry meaning. If a brand shows up repeatedly across dozens or hundreds of responses, across multiple models, that suggests it is consistently part of the AI’s consideration set.
That is measurable. It is still imperfect, but it is grounded in data rather than a single screenshot.
This shifts the conversation from “What rank are we?” to “How often are we included?” That question better reflects how large language models actually function.
AI Search Behavior Is Not the Same as Google Search
The study also explored how people write prompts. Researchers asked 142 participants to come up with their own prompt to find the best headphones. Almost none of the prompts looked alike.
People do not interact with AI the same way they type into Google. They are more conversational, more specific, and often more detailed. Instead of short keyword phrases, they ask full questions with context.
Despite that variation, brands like Bose, Apple, and Sony still appeared between 55 and 77 percent of the time. That suggests AI systems are capable of capturing intent across different phrasing. What they are not capable of doing is delivering consistent ranking positions.
What This Means Before You Invest in AI Visibility
If a service promises to get you to number two in ChatGPT, that claim has very little meaning right now. Ranking position inside AI responses shifts constantly.
However, if a provider can demonstrate that your brand appears in a high percentage of responses across a large sample of prompts and multiple models, that is at least a measurable signal.
Before investing in AI visibility tracking, it is worth asking direct questions about methodology. How many prompts were tested? How many runs per prompt? How many models were included? Is the reporting focused on ranking position or overall presence?
Without transparent data and meaningful sample sizes, you may simply be paying for a report that cannot be replicated.
The Real Takeaway for SEO and AI Search Strategy
The larger takeaway is straightforward. AI ranking positions are unstable. AI visibility across repeated, large-sample testing may indicate brand association strength.
If you are going to invest in this space, invest in measurement that reflects how these systems actually work. Focus on building brand authority, topical relevance, and strong associations in your industry. Those are the signals AI models appear to surface consistently.
If you want to go deeper into SEO and AI search strategy, join our Skool community. Inside, you will find structured SEO courses, practical frameworks for improving visibility in both Google and AI systems, and guidance on what actually drives measurable results.
- AI Ranking Is Chaos: What the Latest Study Means for Brands - February 20, 2026
- Stop Relying On Social Media To Grow Your Local Business: Use Local SEO Instead! - February 13, 2026
- Best Local SEO Course Now Available from BKA Content - February 5, 2026
