leverboard
Get a free analysis

How Leverboard measures what your comments cost

For every theme in your comments, the report shows its effect per mention with a likely range, what it costs your average score, how many comments it rests on, and whether that is sure enough to act on. Themes are ranked by what they cost, not by how often they come up. An effect is an association in your data, not proof of cause.

The sample report

The table below is the public sample: 4,000 women’s clothing reviews rated 1 to 5 stars. Open a row to see its comments. The sections after it explain each column.

What moves the rating

4,000 women’s clothing reviews, 1 to 5 stars
Costs points Gains points Likely range (95 %) Low data, under 30 comments Range crosses zero

Data: Women’s E-Commerce Clothing Reviews, public domain (CC0).

Leverboard themes discover what customers care about

A theme is a reason customers give for their score, named in plain words: “Fit or size”, “Doesn’t look flattering”, “Fabric”. Every number in the report belongs to one theme.

The test further down checks that the themes capture what drives the score.

Every comment is readEach comment is sorted into the themes it raises. One comment can raise several, for example a bad fit and a thin fabric.
Starts from your industryThe theme set starts from the themes common in your industry, so your first report needs no setup.
Finds what the list lacksThemes your customers raise that are not on the list show up as suggestions.
In your wordsOn a paid plan, you rename, merge or add themes, and the next report uses them.

Each column in plain words

Every number answers one question a manager asks before acting on a complaint: how bad is it, how sure are we, and how much does it cost the score.

Every number also opens the comments behind it, so you can read what customers actually wrote.

Effect per mentionHow much lower or higher the score is, on average, when a comment raises the theme, with the other themes in the same comment held apart. Reviews that say an item doesn’t look flattering give on average 0.94 stars less.
Likely range (95 %)The span the true effect most likely falls in. For “Doesn’t look flattering” it runs from 0.84 to 1.04 stars less: narrow, so the number is firm. For praise of service or returns, from 16 reviews, it runs from 0.55 stars less to 0.39 more: too wide to act on.
What it costs your averageThe effect per mention times how often the theme comes up. This is how many points or stars the theme takes off your average score across all responses, and the column the report ranks by.
Based on n commentsHow many comments raise the theme. More comments give a narrower range.
Sure enough to actYes, when the range sits clearly on one side of zero and rests on enough comments. No, when it does not.
Low data, under 30 commentsThemes with fewer than 30 comments are marked, not hidden. You see that the theme exists and know to wait for more comments.
Range crosses zeroThe theme could lower or raise the score; the data cannot tell which. Complaints about service or returns in the sample, from 67 reviews, range from 0.05 stars less to 0.34 more. Most of these reviewers like the item and are annoyed it sold out.

Why frequent is not expensive

Counting mentions ranks themes by how often they come up. That is not the same as what they cost. A common complaint can be a mild one, and a less common one can cost a lot each time.

In the sample, fit or size comes up in 32 % of reviews but costs 0.30 stars per mention: about 0.09 stars off the average. “Doesn’t look flattering” comes up in only 15 % of reviews, but each mention costs 0.94 stars: about 0.14 off the average. Fixing the second is worth more.

Complaints customers mention most

  1. 1Fit or size32 %
  2. 2Doesn’t look flattering15 %
  3. 3Fabric10 %

What actually costs you points

  1. 1Doesn’t look flatteringup 1−0.14
  2. 2Fit or sizedown 1−0.09
  3. 3Fabric−0.06

Share of reviews on the left. On the right, stars each theme costs the average rating: the effect per mention times how often it comes up.

An association, not a cause

An effect says that scores are lower when a theme comes up. It does not prove that the theme causes the lower score. Something else can sit behind both, for example a product line that both fits badly and looks cheap.

Use it to choose where to lookThe ranking tells you which themes to read first and which fixes are likely worth the most.
Read the comments before you actEvery number opens its comments. They show what customers mean by a theme, and whether one cause sits behind several themes.
Check after the fixIf a fix works, the theme should come up less often and the score should move in the next reports.

Short and empty comments

Short comments carry themes too: “staff were lovely” and “waited too long” each name one. Empty comments count toward your score, just not toward a theme.

One-liners are mostly praiseIn our test on the NHS Friends and Family Test, one-line comments were mostly praise. The complaints with large effects sat in longer comments.
Short comments are harder to predict fromIn the test below, the NHS data, with a median of 12 words and 7.4 % empty comments, gave the largest prediction error of the three datasets for both approaches that read the comments.

Why Leverboard is better

To fix what costs you points, you need two things from your comments:

  • Themes: reasons you can name and act on. A model that reads the full text of every comment can predict scores, but it cannot tell you which reason to fix.
  • The impact of each theme: numbers that explain the score. A list of themes tells you what customers talk about, but it is only worth acting on if it explains the score.

Leverboard gives you both. It sorts every comment into Leverboard themes and measures what each one costs. To check that the themes explain the score, we hid the ratings of reviews and predicted them from the comments alone.

On all three datasets, Leverboard’s themes predicted the rating more closely than the full-text model, and far more closely than guessing the average.

For you, that means the ranking in your report rests on reasons that explain your score, not just on how often something is said.

Typical error when predicting star ratings (lower is better)
Clothing reviews median 59 words per comment29 % less error than full text
Guess the average1.12
Full text0.86
Leverboard themes0.61
Google restaurant reviews median 32 words per comment28 % less error than full text
Guess the average1.26
Full text0.84
Leverboard themes0.60
NHS Friends and Family Test median 12 words per comment, 7.4 % empty19 % less error than full text
Guess the average1.14
Full text0.95
Leverboard themes0.77

Guess the average gives every review the average rating and ignores the comment. Full text predicts from text embeddings of the whole comment, a general measure of its meaning without named themes. Leverboard themes predicts only from which themes the comment raises. Typical error is the root mean squared error (RMSE), on the same held-out reviews for every approach.

Questions

What does effect per mention mean?

It is how much lower or higher the score is, on average, when a comment raises that theme, with the other themes in the same comment held apart. In the sample, reviews that say an item doesn’t look flattering give on average 0.94 stars less.

What does the likely range mean?

It is the span the true effect most likely falls in, at 95 %. A narrow range means the number is firm. A wide range means few comments, and the effect could be much smaller or larger than the middle value.

Why is a theme with few comments not hidden?

Because a rare theme can still matter, and you should know it exists. Themes with fewer than 30 comments are marked low data, so you know to wait for more comments before acting on them.

Does a large effect prove the theme causes low scores?

No. An effect is an association in your data, not proof of cause. It tells you where to look first. The comments behind each number tell you what is going on.

What does RMSE mean in the test?

It is the typical prediction error, in stars, when the rating of a review is predicted from its comment. Lower is better. All approaches were scored on the same held-out reviews.

See these numbers for your own comments

Your score says how many customers are unhappy. Your comments say why, and Leverboard measures which reason costs the most. More on the idea behind it: key driver analysis and what an LLM gets wrong.

Get a free analysis See a sample report

Up to about 1,000 comments. No card.