ChatGPT, Claude and other LLMs for survey comments

We gave a large language model (LLM) the same 982 reviews twice and asked which themes drive the rating. It found sensible themes and counted them well. Its numbers were the problem: raw differences that mix each theme with everything else unhappy customers wrote, different from run to run, with guessed ranges. Use an LLM for ideas and wording, not for numbers you report. ChatGPT, Claude and Gemini are all used the same way, as a chat prompt over pasted comments, so the checks on this page apply to each of them.

The test

The data: 982 Google reviews of 55 restaurants in Osaka, each with a rating from 1 to 5 stars. We pasted the reviews into a chat prompt twice and asked the same question both times: which themes drive the rating, and by how much?

We then compared both answers with the same reviews labeled comment by comment, and with each theme’s separate effect as a Leverboard report shows it.

The LLMOne current chat model, used as a plain chat prompt over the pasted reviews. The numbers below are from that model; the pattern is what to check for in any LLM.

What we comparedWhich themes it found, how many mentions it counted, the effect it gave each theme, and whether two runs agree.

What the LLM did well

Reading and sorting text is what LLMs are built for, and it showed.

Sensible themesBoth runs found the themes a restaurant manager would expect: staff, price, atmosphere, authenticity.

Counts close to the labelsIt counted 82 and 72 staff complaints in the two runs, against 85 when every review is labeled one by one.

New theme ideasAsked for themes freely, it found ones a fixed list lacked: unequal treatment of foreigners, and quality that had declined since earlier visits.

What went wrong

The numbers. The LLM’s effect for a theme was a raw difference: the mean rating of reviews that mention it against the rest. That is not what the theme costs on its own.

Theme	LLM run 1	LLM run 2	Separate effect
Staff complaints	−2.67	−2.5	about −1.5
Atmosphere praise	+0.59	+0.6	+0.01 to +0.13
Price praise	+0.45	+0.5	−0.12 to −0.01

Stars per mention, 982 Google restaurant reviews, rated 1 to 5.

Complaints came out about twice too largeBad reviews raise several complaints at once, so each complaint gets the blame for all of them. Staff complaints scored −2.67 and −2.5 against about −1.5 on their own.

Praise that moves nothing ranked highAtmosphere and price praise were ranked at +0.5 to +0.6. On their own, they move the rating by close to nothing.

Two runs, two answersThe runs disagreed by 0.24 stars on average and by up to 1.4 stars on single themes: the authenticity complaint came out at −2.17 in one run and −0.8 in the other.

Ranges were guessedThe ranges in its answers were guessed, not computed from the data.

No theme per commentA prompt like ours returns a summary, not the theme of each comment, so you cannot click from a number to the comments behind it.

Why raw differences mislead

Picture a one-star review: the wait was long, the food was cold and the staff were rude. A raw difference credits the full gap to each of the three complaints. Add them up and the themes explain far more than the rating ever dropped.

Praise has the mirror problem. Happy diners mention the atmosphere next to the food they loved, so atmosphere looks like a driver even when it adds almost nothing of its own.

What you need insteadEach theme’s effect with the other themes in the same comment held apart, a range computed from the data, and the count it rests on. The method page explains these numbers in plain words.

Why it matters for decisionsIf staff complaints look twice as costly as they are, they win the budget over a fix that would have moved the score more.

How to use an LLM for your comments

An LLM is a good reading assistant. Give it jobs where a wrong number does no harm and you can check the result.

Ask for theme ideasPaste a few hundred comments and ask which topics customers raise, praise and complaints separately. Ask it to list themes it would not expect in your industry. This is where it found unequal treatment and quality decline in our test.

Ask for wordingHave it name each theme in your customers’ words, for example “waited too long” rather than “service latency”. Reports read better when the themes sound like the comments.

Ask for quotes, then check themRequest three verbatim comments per theme and search your export for each one. Quotes that do not exist in the file are a warning sign for everything else in the answer.

Run it twiceAsk the same question in a new chat. Themes that appear both times are solid ideas; numbers that change are not numbers.

Do not report its effectsTreat any “drives the rating by X stars” as a guess. The same goes for its ranges.

Remove personal data firstStrip names, emails and order numbers from comments before you paste them anywhere.

LLM or Leverboard

Use an LLM for ideas. Use Leverboard for numbers you can report: every theme ranked by what it costs your score, with the evidence. See it on the sample report.

What you need	LLM prompt	Leverboard
Theme ideas	Good, including themes a fixed list lacks	Themes for your industry, new ones shown as suggestions
Effect of each theme	Raw differences, complaints about twice too large	Separate effect per mention
How sure the number is	Guessed ranges	Likely range (95 %), count, low data under 30 comments marked
Click to the comments	Summary only, no theme per comment	Every number opens its comments

Questions

Can ChatGPT or another LLM analyze survey comments?

It can read them and suggest sensible themes, and it counts mentions reasonably well. In our test of an LLM, the numbers for how much each theme moves the score were raw differences, changed between runs and came with guessed ranges. Use it for ideas, not for numbers you report.

Which LLM did you test?

Claude (Opus) by Anthropic, as a plain chat prompt. ChatGPT, Gemini and other LLMs are used the same way. We did not measure them, so the numbers on this page are for the model we ran; the checks apply to any LLM.

Why do an LLM’s numbers change between runs?

An LLM writes a new answer each time, and nothing ties its numbers to a fixed calculation over every comment. In our test, two runs on the same reviews disagreed by 0.24 stars on average and by up to 1.4 stars on a single theme.

What is an LLM good for in survey analysis?

Finding theme ideas, naming themes in your customers’ words, pulling example quotes and summarizing what a group of comments says. It found themes in our test that a fixed list lacked.

Get numbers you can report

Your score says how many customers are unhappy. Your comments say why, and Leverboard measures which reason costs the most, with a likely range and the comments behind every number.

Try it on your data See a sample report

First analysis free. No card.