What each number in a Leverboard report means

The sample report

The table below is the public sample: 4,000 women’s clothing reviews rated 1 to 5 stars. Open a row to see its comments. The sections after it explain each column.

What moves the rating

4,000 women’s clothing reviews, 1 to 5 stars

Costs points Gains points Likely range (95 %) Low data, under 30 comments Range crosses zero

Data: Women’s E-Commerce Clothing Reviews, public domain (CC0).

Each column in plain words

Every number answers one question a manager asks before acting on a complaint: how bad is it, how sure are we, and how much does it cost the score.

Every number also opens the comments behind it, so you can read what customers actually wrote.

Effect per mentionHow much lower or higher the score is, on average, when a comment raises the theme, with the other themes in the same comment held apart. Reviews that say an item doesn’t look flattering give on average 0.94 stars less.

Likely range (95 %)The span the true effect most likely falls in. For “Doesn’t look flattering” it runs from 0.84 to 1.04 stars less: narrow, so the number is firm. For praise of service or returns, from 16 reviews, it runs from 0.55 stars less to 0.39 more: too wide to act on.

What it costs your averageThe effect per mention times how often the theme comes up. This is how many points or stars the theme takes off your average score across all responses, and the column the report ranks by.

Based on n commentsHow many comments raise the theme. More comments give a narrower range.

Sure enough to actYes, when the range sits clearly on one side of zero and rests on enough comments. No, when it does not.

Low data, under 30 commentsThemes with fewer than 30 comments are marked, not hidden. You see that the theme exists and know to wait for more comments.

Range crosses zeroThe theme could lower or raise the score; the data cannot tell which. Complaints about service or returns in the sample, from 67 reviews, range from 0.05 stars less to 0.34 more. Most of these reviewers like the item and are annoyed it sold out.

Why frequent is not expensive

Counting mentions ranks themes by how often they come up. That is not the same as what they cost. A common complaint can be a mild one, and a less common one can cost a lot each time.

In the sample, fit or size comes up in 32 % of reviews but costs 0.30 stars per mention: about 0.09 stars off the average. “Doesn’t look flattering” comes up in only 15 % of reviews, but each mention costs 0.94 stars: about 0.14 off the average. Fixing the second is worth more.

Complaints customers mention most

1Fit or size32 %
2Doesn’t look flattering15 %
3Fabric10 %

What actually costs you points

1Doesn’t look flatteringup 1−0.14
2Fit or sizedown 1−0.09
3Fabric−0.06

Share of reviews on the left. On the right, stars each theme costs the average rating: the effect per mention times how often it comes up.

An association, not a cause

An effect says that scores are lower when a theme comes up. It does not prove that the theme causes the lower score. Something else can sit behind both, for example a product line that both fits badly and looks cheap.

Use it to choose where to lookThe ranking tells you which themes to read first and which fixes are likely worth the most.

Read the comments before you actEvery number opens its comments. They show what customers mean by a theme, and whether one cause sits behind several themes.

Check after the fixIf a fix works, the theme should come up less often and the score should move in the next reports.

Short and empty comments

Short comments carry themes too: “staff were lovely” and “waited too long” each name one. Empty comments count toward your score, just not toward a theme.

One-liners are mostly praiseIn our test on the NHS Friends and Family Test, one-line comments were mostly praise. The complaints with large effects sat in longer comments.

Short comments are harder to predict fromIn the test below, the NHS data, with a median of 12 words and 7.4 % empty comments, gave the largest prediction error of the three datasets for both approaches that read the comments.

How we tested

If the themes capture what customers say about their rating, they should predict the rating of a review the analysis has not seen. We predicted the rating of held-out reviews from their comments, the same reviews for every approach, and measured the typical error in stars (RMSE). Lower is better.

Mean only guesses the average rating for every review, ignoring the comment: the error to beat. Text embeddings turn the full text of each comment into a list of numbers that capture its general meaning, without named themes. Leverboard themes predict from which themes each comment raises.

Dataset	Median words per comment	Mean only	Text embeddings	Leverboard themes
Clothing reviews (CC0)	59	1.121	0.855	0.610
Google restaurant reviews	32	1.256	0.835	0.598
NHS Friends and Family Test	12 (7.4 % empty)	1.137	0.952	0.771

Typical prediction error of the rating in stars, on the same held-out reviews per dataset. On all three, the named themes predicted the rating more closely than the full text did.

Questions

What does effect per mention mean?

It is how much lower or higher the score is, on average, when a comment raises that theme, with the other themes in the same comment held apart. In the sample, reviews that say an item doesn’t look flattering give on average 0.94 stars less.

What does the likely range mean?

It is the span the true effect most likely falls in, at 95 %. A narrow range means the number is firm. A wide range means few comments, and the effect could be much smaller or larger than the middle value.

Why is a theme with few comments not hidden?

Because a rare theme can still matter, and you should know it exists. Themes with fewer than 30 comments are marked low data, so you know to wait for more comments before acting on them.

Does a large effect prove the theme causes low scores?

No. An effect is an association in your data, not proof of cause. It tells you where to look first. The comments behind each number tell you what is going on.

What does RMSE mean in the test?

It is the typical prediction error, in stars, when the rating of a review is predicted from its comment. Lower is better. All approaches were scored on the same held-out reviews.

See these numbers for your own comments

Your score says how many customers are unhappy. Your comments say why, and Leverboard measures which reason costs the most. More on the idea behind it: key driver analysis and what a chat model gets wrong.

Try it on your data See a sample report

First analysis free. No card.

What each number in the report means