leverboard
Try it on your data

How to analyze open-ended survey responses

Read a sample of the comments, build a short list of themes, code every comment with the themes it mentions, and count them. Then do the step most guides skip: link each theme to the score. The theme mentioned most is often not the one that costs the most.

Six steps from comments to decisions

This works for the comment box of an NPS or CSAT survey, for reviews, and for any survey data that pairs a score with free text. You need one row per response: the score and the comment.

  1. Read a sample

    Read 100 to 200 verbatim responses from across the scale, low and high scores alike. Note what people talk about in their own words.

  2. Build a theme list

    Turn your notes into 10 to 25 themes, each one specific enough to act on: “delivery late” rather than “delivery”. Split praise from complaints.

  3. Code every comment

    Mark each comment with every theme it mentions. One comment often carries two or three. Add a theme when new comments keep not fitting.

  4. Count

    For each theme, the share of comments that mention it. This tells you what is common, not what matters.

  5. Link themes to the score

    Compare the score of comments that mention a theme with those that do not. Multiply by how often it comes up to see what it costs the average.

  6. Read the comments behind the top themes

    Before you act, read what customers wrote under the two or three themes that cost the most. The number says where to look; the verbatims say what to fix.

Counting stops at step 4. That ranks themes by how often they come up, which is not the same as how much they hurt the score.

In 4,000 public clothing reviews rated 1 to 5 stars, fit or size was the most mentioned complaint, in 32 % of reviews. “Doesn’t look flattering” came up in 15 %, but where it came up the rating dropped much further. Times how often each appears, it cost the average rating more: −0.14 stars against −0.09.

Complaints mentioned most

  1. 1Fit or size32 %
  2. 2Doesn’t look flattering15 %
  3. 3Fabric10 %

What costs the most

  1. 1Doesn’t look flattering−0.14
  2. 2Fit or size−0.09
  3. 3Fabric−0.06

Share of reviews on the left. On the right, stars each theme costs the average rating: the effect per mention times how often it comes up. Data: Women’s E-Commerce Clothing Reviews, public domain (CC0). An effect is an association, not proof of cause.

The spreadsheet approach for small sets

For a few hundred comments, a spreadsheet is enough to do all six steps by hand. Put the score in column A, the comment in column B, and one column per theme with 1 if the comment mentions it and 0 if not.

Share of comments=AVERAGE(C2:C301) on a theme column gives the share of comments that mention it.
Score with and without=AVERAGEIF(C2:C301,1,A2:A301) and =AVERAGEIF(C2:C301,0,A2:A301) give the average score of comments with and without the theme. The difference is a rough effect.
Cost to the averageRough effect times share. Sort themes by it, not by count.
Know the limitsUnhappy customers often raise several complaints at once, so a raw difference credits each complaint with some of the others. Themes with fewer than about 30 comments give numbers too uncertain to rank.

Where a chat model helps, and where it fails

A chat model is good at steps 1 and 2. Paste a sample of comments and ask which themes it sees: it finds sensible ones quickly, including some a fixed list would miss.

It is weak at step 5. We gave Claude 982 restaurant reviews twice and asked which themes drive the rating. See the full test.

Runs disagreeThe two runs differed by 0.24 stars per theme on average, and by up to 1.4 stars on a single theme.
Complaints mixed togetherIts effects were raw differences, about twice the separate effect of each complaint: staff complaints came out at −2.67 and −2.5 stars against about −1.5.
Praise overratedAtmosphere praise ranked at +0.59 and +0.6 had a separate effect of only +0.01 to +0.13.
Ranges guessedThe ranges it gave were not computed from the data.

When to use a tool

By hand works for one survey and a few hundred comments. Once you run the survey every month, or have thousands of comments, coding by hand stops being repeatable.

Tagging tools countThey sort comments into themes and show how often each comes up. That covers steps 3 and 4.
Leverboard measuresIt ranks themes by how much each moves the score: effect per mention with a likely range, what it costs the average, how many comments it rests on, and whether that is sure enough to act on. Every number opens its comments.
Long comments hold the costly complaintsOne-line answers carry themes too. In a test on NHS feedback, short comments were mostly praise; the complaints with large effects sat in longer ones.

Questions

How do you analyze open-ended survey responses?

Read a sample, build a list of themes, code every comment with the themes it mentions, count each theme, then compare the score of comments with and without each theme. Read the comments behind the themes that cost the most before you act.

What are verbatim responses?

The exact words respondents typed into an open-ended question, unedited. Keep them attached to every number in your analysis, so anyone can check what a theme stands for.

Can one comment have more than one theme?

Yes, and it often does. Code every theme a comment mentions. Forcing one theme per comment loses the second complaint and distorts the counts.

How many mentions does a theme need?

With fewer than about 30 comments, the effect of a theme on the score is too uncertain to rank. Keep such themes on the list, mark them as low data and wait for more responses.

Can I use ChatGPT to analyze survey comments?

For ideas, yes. In our test with Claude, a chat model found sensible themes, but its effects on the rating changed between runs and mixed each theme with the other complaints in the same reviews. We did not test ChatGPT.

Skip the coding, keep the verbatims

The score says how many customers are unhappy. The comments say why. Leverboard measures which reason costs the most, from a CSV or Excel export with a score and a comment per response, and links every number to its comments. See also NPS vs CSAT.

Try it on your data See a sample report

First analysis free. No card.