The internet is awash in positive product ratings. Here's how to decipher the good from the great

Through computational analysis of hundreds of thousands of online reviews, the researchers discovered that positive star and numerical ratings didn’t reliably correlate with how well products and businesses ultimately fared Image: Shutterstock

Ever had a great meal at a restaurant rated four out of five stars, but then had a lackluster meal at another restaurant with the exact same rating?

If so, you might have fallen victim to what Kellogg School researchers have dubbed “the positivity problem”: the vast majority of online reviews are positive, but those favorable reviews don’t always translate to real-world quality. “You can have two products with four-and-a-half stars, but they’re not equally good, nor are they equally successful in the marketplace,” explains Derek Rucker, a professor of marketing at Kellogg.

Luckily, the researchers developed another way to help parse apart products that appear similar based on ratings. In a new paper, Rucker—along with coauthors Loran Nordgren, a professor of management and organizations at the Kellogg School, and Matthew D. Rocklage of the University of Massachusetts Boston—finds that the emotionality of the written text in a review is a much better predictor of success than the numeric or star rating that goes with it.

Emotionality is different from valence—whether a feeling is positive or negative—and from extremity—how positive or negative the feeling is. Rather, it’s the extent to which a reaction is rooted in emotion. “‘Awesome’ is a very positive word that also conveys a lot emotion with it,” Rucker explains. “‘Perfect’ is also a very positive word, but it doesn’t have a lot of emotionality.” The same goes for “fantastic” as compared with “perfect,” or “terrible” as compared with “dumb.”

Through computational analysis of hundreds of thousands of online reviews, the researchers discovered that positive star and numerical ratings didn’t reliably correlate with how well products and businesses ultimately fared. But the emotionality of the accompanying written reviews did. (Though rarer, negative ratings are actually more useful; a one-star restaurant is probably both genuinely bad and doomed to failure.)

It’s an important finding for consumers, business owners, and industry experts.

“If you’re looking at ratings, you can figure out what’s a bad restaurant, but you really can’t figure out what’s the best choice for you within a set of good restaurants,” Nordgren says. “Analyzing the language that people use allows us to understand sentiment in a way that the rating system doesn’t reveal.”

Why “awesome” movies outperform “perfect” ones

To understand how people people’s emotionality informs success, the researchers started with movie reviews.

They gathered user-generated reviews from the popular rating site Metacritic. For each movie on the site released between 2005–2018, they analyzed numeric ratings, as well as the text, of the first 30 reviews. (These early reactions, the researchers posited, would be less influenced by the eventual success of the film or the opinions of other viewers and professional critics.)

The researchers used a computational tool that coauthor Rocklage developed called the Evaluative Lexicon (EL) to measure the emotionality of the written reviews. The EL scans the text and “distinguishes words that tend to convey more emotion from those that convey less emotion,” Rucker says—essentially, separating the “awesome” from the “perfect”—and it then generates an emotionality score.

The positivity problem is very real, the researchers found: of all the movies in the analysis, a whopping 81 percent got reviews of five or above on a ten-point scale. However, these favorable reviews didn’t necessarily translate into box-office success.

Instead, the emotionality of the positive reviews was a more accurate barometer of a true hit. The EL score of users’ text reviews predicted box-office earnings, a pattern that held true even when controlling for the movie’s genre, budget, length, and release date.

And it’s not just movies. The researchers repeated the analysis for book reviews on Amazon and found the same pattern: for books that garnered at least three stars out of five, more stars didn’t reliably predict higher sales. But more emotionally charged reviews did predict sales.

Was that bruschetta “amazing” or just “enjoyable”?

Next, the researchers turned their attention to restaurants, using Yelp to gather the first 30 star and text reviews of all restaurants in Chicago in 2017. They also gathered the average number of daily reservations for each establishment on OpenTable, the most popular online reservation service in the U.S.

As with books and movies, almost all Yelp reviews were favorable—92 percent of restaurants received a rating of greater than three out of five stars. But unlike movies and books, these star ratings did correlate with daily reservations. However, the emotionality of reviews as measured by the Evaluative Lexicon was an even better predictor of which restaurants’ tables would fill up.

From movies to restaurants, “at best, the star rating is an inconsistent signal. Sometimes it predicts success, and sometimes it doesn’t,” Rucker says. But “within the sea of positivity, one of the distinguishing factors is how much emotion is conveyed in the review.”

Harnessing the power of online reviews

In theory, online reviews are supposed to give consumers and industry experts new and more accurate ways to identify the best products.

“The promise of these online platforms is that now we can see and learn from others, and that should have extraordinary value,” Nordgren says. But the overwhelming number of positive reviews has dampened their usefulness, resulting in a missed opportunity. Looking at emotionality is one important way to capitalize “on that unrealized promise.”

Of course, this doesn’t mean business owners should ignore numerical ratings, Rucker points out. “If a consumer just left you a one-star review, that’s important too,” he says. “We aren’t suggesting ignoring reviews, but recognizing they are one piece of information. We should take advantage of the fact the data are much richer.”

But they can’t be the only way to measure success. “If people are giving you a positive review but there’s nothing strongly emotional in their language, they’re not really showing a passion for the brand. Maybe there’s an opportunity to improve, and our tool can help reveal that,” he says. “Emotionality is another piece of data that people should be thinking about and looking at.”

[This article has been republished, with permission, from Kellogg Insight, the faculty research & ideas magazine of Kellogg School of Management at Northwestern University]