Amazon’s Reviews Somewhat Slanted

According to a new study, reviews on Amazon.com are at least slightly skewed, in large part because positive reviews help reviewers achieve higher rankings.

Trevor Pinch, a professor at Cornell University, surveyed 166 of Amazon’s top 1,000 reviewers to probe whether Amazon’s wealth of outside insights was as much of a "free lunch" as it seemed.

Asked why they contribute reviews, respondents overwhelmingly mentioned "self-expression" and "enjoyment." Altruistic reasons — "Hope to help others decide whether to buy," "Wanting to share what I have liked with others" — were also given.

Some wrote reviews to keep track of new books or movies. (Books remain the highest reviewed category.)  Top reviewers also gain "small utilitarian benefits," largely free books although many reviewers said most of the free books aren’t worth reading.

The problem appears to be the ranking system of the reviews. Higher rankings were rewarded for three reasons: "overall helpfulness" of the review (as rated by customer votes), the number of reviews the person has written, and the recentness of the review.

The need to deliver numerous and ever-current reviews are leading some prolific reviewers to "cut corners" to drive productivity. Prof. Pinch writes that this "seems to go against the grain of serious book reviewing."

The bigger issue is that driving up the "helpfulness" grade appears to encourage positive reviews. Indeed, some 88 percent of respondents reported that most or all of the reviews they wrote were positive.

Writing negative reviews is avoided because such reviews are more likely to be found "unhelpful" or might solicit "mean spirited" comments from authors or other reviewers in the Amazon Comments section. Reviewers who themselves produce the same sort of material under review are also particularly cautious about posting negative reviews.

The authors of the study said one solution might be paying reviewers while admitting that such remuneration could "undermine the literary value of their endeavors" and also raise doubts that upbeat reviews are just being incentivized with straight cash.

Writing for paidcontent.org, which studies digital  content, Laura Hazard Owen said that although Amazon doesn’t appear to be manipulating upbeat reviews, the study provides some insight into why the soundness of reviews on other sites, such as Yelp, are also sometimes questioned.

"Reading Pinch’s interviews with reviewers, you get a sense of how hard it is maintain the integrity of a process that is dependent on a virtual army of unpaid but still presumably capitalist-minded laborers," she wrote. "If they’re not paid, they are going to find other incentives and motivations, which may in some cases work at cross purposes with their primary mandate, to produce honest and independent-minded reviews."

BrainTrust

Discussion Questions

Discussion Questions: What do you think of the credibility and value of online reviews, whether at Amazon or even more review-driven websites such as Yelp? Do you have an issue with how reviewers are being incentivized to post reviews?

Poll

13 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
David Biernbaum
David Biernbaum
12 years ago

Customer reviews are taken very seriously by other customers online. This is the case whether deserved or not, or whether it should be or not. It’s human nature for people to rely on other people to tell them if products are good or bad. Most people are followers that way. Ironically, professionally written reviews are taken with more a grain of salt by most readers.

Max Goldberg
Max Goldberg
12 years ago

Reviews can be helpful. Careful consumers will look at both positive and negative reviews, and weigh the value of each, before making a purchase. Websites like Amazon and Yelp have a vested stake in publishing truthful reviews to maintain their credibility. Consumers should find sites they trust and ignore those that seem to be too bombastic, either positive or negative.

Dick Seesel
Dick Seesel
12 years ago

It’s not surprising that those most motivated to post reviews on Amazon or elsewhere are the most enthusiastic about the product/book/service they are reviewing. As a regular reader of these reviews, I’ve learned to take them with a grain of salt. (Just as importantly, it’s worth recognizing when a book gets “down-rated” on Amazon because of reviewers who complain about the Kindle price even if they haven’t read the book.) No system of online reviews–Amazon, TripAdvisor, Yelp or others–is going to be flawless.

Ian Percy
Ian Percy
12 years ago

“Slightly skewed?” You’re kidding me.

If I had a dollar for every author who’s asked me to go in and write a positive review on their book (which I’ve never heard of never-mind read) I’d be writing this from my yacht.

In particular speakers who also write books love to get their entire mailing list to buy a copy on at a certain time on a certain day so they can then brag about being #1 on Amazon. The fact that was true for about 60 seconds doesn’t seem to bother them from claiming #1 status for the rest of their lives.

Not all reviews are set-up, I’ve written many myself. But to generally use Amazon’s reviews as a measure of a book’s value or popularity is pretty well pointless.

Paula Rosenblum
Paula Rosenblum
12 years ago

I think this door swings both ways. When I was looking for a moving company to take my “stuff” from Boston to Miami, I found something to dislike about every single moving company in the universe based on online reviews. In the end I just closed my eyes and picked one. Of course, everything turned out fine.

So we have the squeaky wheel syndrome on one side–where the most harmed are more likely to take the time to write something, and we have the other side–incentivized reviewers.

The net is the consumer must read a lot of reviews and then decide for him or herself. And then if your experience differs dramatically from what you read, you’ve got to write a review too. That’s the only way the system works.

Ed Rosenbaum
Ed Rosenbaum
12 years ago

Amazon has proven over time to be a valuable, reliable resource. Some not so glowing reviews will not change the public image at all.

Jonathan Marek
Jonathan Marek
12 years ago

The reviews may be slanted, but they are valuable, largely because it is pretty easy to see the quality of the actual content of the reviews (rather than just looking at stars). I suspect most consumers have a process for scanning reviews and gathering their own takeaways, but I’ve never seen any research on how customers engage with reviews.

It would be fascinating to push that idea, looking at click patterns and time spent on each review along with purchase information to understand how different customer segments put review info into a context to drive a decision.

Ryan Mathews
Ryan Mathews
12 years ago

I think they are suspect at best.

I once had one of my books reviewed on Amazon by somebody who gave it low marks because it didn’t have an index. The problem, of course, was that the book clearly does have an index. Despite numerous tries at bringing clarity (and perhaps sanity) to the reviewer and several nasty noted to Amazon the review stands.

By the same token, all three of my books have received glowing reviews from people who–I would guess–never actually read them.

Pass that salt shaker–I need a grain or two.

Anne Bieler
Anne Bieler
12 years ago

Amazon has built an amazing business model based on good value and great customer service. They have creating best practices that other retailers study.

Reviews are important to many shoppers as they decide–and yes there will probably be inconsistency as you depend on voluntary participation. Like any marketing approach in a fast changing digital world, it will be subject to evaluation and renewal.

Bill Hanifin
Bill Hanifin
12 years ago

Where there is smoke, there is normally fire and when examining reviews on Amazon, there is enough “smoke” to make the reader skeptical of its value.

The motivations of reviewers as stated in the survey indicated that an application of game theory could create an environment that would not only improve the value of reviews, but would impact sales through added credibility.

One social shopping network, Zavee.com only allows reviews to be posted following a purchase with the merchant being reviewed. This should be a best practice tactic that would discourage negative reviews from competitors and those with just too much time on their hands.

Game theory and opt-in via purchase sounds like a winning combination that Amazon could give a try.

M. Jericho Banks PhD
M. Jericho Banks PhD
12 years ago

I buy books only from Amazon, a lot of them, usually have three or more non-fiction tomes going simultaneously and at same time, and have never, ever read an Amazon book review. Instead, I count on certain authors whose every book I read and upon their recommendations of other authors. This includes books written by acquaintances, and the opinions of friends are always welcomed. I personally oversee the very loosely organized, informal, DocBanks Book Club & Complaining Society via email, where book titles are exchanged among trusted acquaintances, friends, family, in-laws, and outlaws. We haven’t gotten around to the Complaining part yet, because everybody seems to respect the others’ recommendations.

Herb Sorensen, Ph.D.
Herb Sorensen, Ph.D.
12 years ago

When “the crowd” is allowed to review, you can never eliminate strictly partisan comment, whether it is commercial or otherwise. All of the problems discussed here exist at places like Wikipedia, too. Personally, on reviews such as at Amazon or Yelp, I hope to see both positive and negative comments, and like to see quite a few ratings before taking any of them too seriously. It’s known as “The Wisdom of Crowds,” by James Surowiecki.

Ronald Stack
Ronald Stack
12 years ago

A review will be most valuable if the author is perceived to be credible (i.e., is an actual customer independently relating an authentic experience, not a PR rep or a hostile competitor) and knowledgeable (about the product and the space); if the review is timely; if the review contains information and opinion relevant to the user; and if the platform that hosts the reviews has a process for weeding out fictitious reviews, whether positive or negative.

That’s why transparency in the review process is so vital. I think compensating reviewers is likely to be a mistake, because any gain in expertise is likely to be offset by a loss in credibility. I think biasing reviews to be positive, even if done inadvertently, is also a mistake, especially because thoughtful negative reviews (not rants) can be very useful. But users can make their own decisions as to how to weight reviews *if* they know the rules.

That said, online reviews in general have tremendous value. They reflect consumers’ desire both to limit their exposure to mass market advertising and to depend more on other consumers for information and experiences. The cycle of reading and writing reviews is a key aspect of the user experience on TripAdvisor and is central to the Yelp experience as well. Both sites have developed loyal communities of engaged consumers who produce valuable content–for free–largely to “give back” after receiving value from the reviews on the site.