ChatGPT names brands far less often in longer queries, study finds

A new study from London AI consultant Frank Vitetta found that ChatGPT is far less likely to mention brands in longer shopping and buying queries than in short ones. The results suggest brands may need to rethink how they measure AI visibility as answer quality shifts with prompt length.

Why it matters: - Brand visibility in ChatGPT can change sharply depending on how a question is phrased, which affects how brands measure performance in AI search. - The findings matter for generative engine optimisation, AI optimisation and answer engine optimisation as more consumers use chatbots for purchase decisions. - Brands that rely on AI visibility scores may draw misleading conclusions if test prompts are not balanced by query length.

What happened: - Frank Vitetta, an AI consultant based in London, analyzed 8,555 ChatGPT answers to shopping and buying questions collected by LLM Scout over nine months. - Short questions of about one line, often in the “best X for Y” format, produced brand mentions in 46.7% of answers. - Questions of 120 characters or more produced brand mentions in 6.5% of answers. - That means long queries were about seven times less likely to produce a brand name in ChatGPT’s response.

The details: - Questions between 35 and 69 characters produced brand mentions in 44% to 47% of answers, the strongest range in the study. - Questions from 70 to 89 characters saw brand mentions fall to 35%. - Questions from 90 to 119 characters saw brand mentions fall further to 28%. - At 120 characters or more, only 75 of 1,147 answers included a brand name. - Short “best X” questions tended to generate lists of brands. - Longer, scenario-based questions tended to generate advice instead of specific brand recommendations. - The full study, including methodology, data charts and limitations, is available here.

Between the lines: - The research suggests AI visibility scores can move simply because the test set includes more short or long prompts, even if the assistant itself has not changed. - Vitetta said, “Nothing about the assistant has changed. Only the mix has.” - Vitetta also said, “If you build test prompt sets to measure how assistants answer, the length mix of the set moves the result.” - Removing the longest questions from the dataset lifted the overall brand mention rate from 36% to 40.5%, showing how much prompt composition can skew results.

What's next: - Vitetta recommends that brands ask for AI visibility scores broken out by short and long questions. - He also recommends testing longer, more detailed prompts that reflect how real customers ask for advice. - Brands should keep question sets consistent across reporting periods to make results comparable over time. - The research data was collected using LLM Scout more information. - Frank Vitetta says he helps teams prepare data for AI applications and writes about how AI assistants find and name brands.