The consumer's always wrong

Why else would visitors to consumer rating sites like rank Rolling Rock the second-best beer and Alan Keyes the top presidential candidate?


As a child, I spent several years of afternoons happily engrossed in “Family Feud,” and ever since I have had a mania for polls and surveys. Not only have I always wanted to know what other people think — this is a basic human trait — but I have always had a particularly urgent need to know exactly what percentage of other people are thinking it. Since “Family Feud” asked contestants to guess the results of a survey of 100 Americans, (contestants would be presented with questions like, “Name an animal likely to be found on a farm,” and would score the most points for guessing the most common answers), my vision of utopia, is knowing what 100 Americans think about everything, all the time: what percentage of them like the new Fox sitcom; what percentage think Kenmore dryers chew up their clothes; and what percentage think Margaret Thatcher is still the prime minister of Great Britain.

My little fantasy of having instant access to a compendium of opinion surveys might have been preposterous before the Web, and yet in just a few months we have gotten much closer than we ever previously imagined to my peculiar vision of a plebiscite society.

When launched its online bookstore, its most striking feature (besides the simple fact that it let people buy books online) was that Amazon encouraged users to rate and discuss books in what amounted to virtual bookstore aisles. Being able to find out what other readers think about books that you’re thinking of buying was a natural and enticing use of the Internet.

Amazon itself has extended the ratings idea to a host of products besides books. And the initial idea has spawned a whole new sector of the Web: sites devoted to consumer ratings.

Consumers instantly reaching a wide audience with their take on whether their money was well spent is shaping up to be one of the most dramatic effects of the Internet. For professional marketers, it is undoubtedly also one of the scariest. Remember the advice your mother/teacher/best friend gave you in, oh, about third grade? “Don’t get mad, get even”? A lot of people now have a way to get even with companies they believe gave them a bad deal — and they see getting even as a public service.

Take Greg Plough, a onetime Prodigy customer who was so dissatisfied with the service he got from the Internet access provider that he posted a dismal review online. “I wanted to send out a warning to the millions of people who are getting shafted by [Prodigy's] rebate deal,” he wrote on one consumer review site.

The most interesting twist to consumer ratings, however, comes from sites like and Consumer Review. They don’t simply rate products (or, in’s case, not just products, but pretty much everything — from magazines to political candidates to cat breeds) but assemble the ratings into neat lists of the best and the worst.

A lot of the fun of rating things — most of the fun, actually — is that once you’ve rated them, you can rank them, too. Let’s face it, we don’t just want to know whether a particular movie is good or whether a particular video recorder is reliable. No, we want to know its particular and very precise degree of goodness, and before we shell out our money we want to know if that camcorder’s degree of goodness is even a shade of a fraction less than some other model’s.

Both Consumer Review and distill user comments into an average rating, from 1 to 5. does it in an especially elegant way, copying the “Zagat Survey” approach to rating restaurants by creating four measures in each category (like “comfort,” “features,” “reliability” and “cost” for sedans, or, a bit strangely, “delivery,” “interview style,” “reporting” and (!) “sex appeal” for television newscasters) to create a veritable feast of numerical ratings. links from the product ratings directly to retailers and Consumer Review has plans for a similar system. Both sites are betting that consumers will use the ratings to make purchase decisions — and entitle the sites to a cut of any sales made.

If, like me, you loved “Family Feud” and are a sucker for the nifty chart at the back of Entertainment Weekly which compresses the results of a dozen different critics’ responses to a movie into one handy little chart, you’ll probably also find yourself a junkie for the survey results at Consumer Review and But the more you look at the ratings, the more you are likely to notice some odd results.

The first peculiar result of the proliferation of online surveys is what I have come to think of as “the Philadelphia effect.” The term is mine, but credit for noticing it goes to Tim Zagat, the co-creator (with his wife, Nina) of the Zagat guides.

The Philadelphia effect takes its name from a survey conducted some time ago by a glossy magazine. “There was a magazine that surveyed its readers about what was the best restaurant city in America,” recalls Zagat. “The result was Philadelphia. I almost choked.”

Of course, there are undoubtedly people (most of them, presumably, native Philadelphians) who will be offended by Tim Zagat’s blithe dismissal of the opinions of hundreds of magazine readers. Who is Zagat to say that Philadelphia isn’t the best eating city in America?

If, however, you do not live in Philadelphia, you might be inclined to the view that the fact that Philadelphia was named as the best dining city in America says a lot less about Philadelphia or its restaurants than it does about the Philadelphians who we can safely presume are the restaurants’ most frequent customers. It might well mean not that the restaurants are the best, but that the patrons are among the least demanding. The ratings wind up telling you a lot more about the raters than they do about the city or its restaurants.

Online, it’s really easy to see the Philadelphia effect in action. Consider, for instance, the ratings for beer.

Even if you are not a beer drinker, it is likely that you know that supermarket shelves are filled with microbrews like Pete’s Wicked Ale or Samuel Adams, and plenty of high-priced imports like Guinness. Many of these are quite good. Expensive, but good. And some of the same beers that win competitions also score well in’s ranking.

But up at No. 2 in’s rankings is an inexpensive macro-brew much loved by college students: Rolling Rock. (The top spot goes to Yuengling Lager.) I know a lot about Rolling Rock because in college, when I drank beer, Rolling Rock was the beer that I drank. And it’s not a bad beer. But the second-best in the country? Better than all the beers that sell at three times the price? Well, maybe. But not very likely. In fact, it’s a good bet that even most of the people who praised Rolling Rock on the site, would not rank it so highly in a side-by-side comparison.

Of course, it’s hard to prove that Rolling Rock isn’t the second best beer in the country, just as it’s very hard to prove that Philadelphia doesn’t have the country’s best restaurants. But it’s an awfully suspect result.

The Philadelphia effect has a corollary, which you can think of as the Alan Keyes tsunami. The Philadelphia effect is the result of a lot of people, who are easy to satisfy, giving a surprisingly high rating to Philadelphia restaurants or Rolling Rock beer. The Alan Keyes effect is the result of a lot of people who have a very strong opinion on one product (or, in Alan Keyes’ case, one political candidate) sign on en masse and flood a site with reviews.

The Alan Keyes tsunami gets its name from a Republican presidential candidate. If you haven’t heard of him, you’re not alone. In August’s Iowa straw poll, Keyes got only 5 percent of the vote among a collection of Republican die-hards. Poor Alan Keyes is on his second run (very likely you didn’t hear about the first one either), gamely shambling between pancake breakfasts with nary a television camera in sight, but at least he has this consolation:’s raters are firmly in his camp.

In fact, Keyes scores highest of all the presidential candidates rated on the site. He is followed by John McCain, Bill Bradley, George W. Bush, Steve Forbes; Jesse Ventura, the former wrestler turned Minnesota governor scored higher than Keyes — but he’s not even running. Keyes’ popularity is no statistical fluke. At last count, 1,971 people put in scores for Keyes — fewer than the nearly 4,000 who expressed an opinion about George W. Bush — but a very respectable sample pool nonetheless.

Even the people behind seem to take some of the ratings with a grain of salt. “We’re definitely not the U.S. Census,” cautions executive vice president David Wilson.

Results like this can make old-school experts in the art of consumer surveys apoplectic. Says Zagat: “A lot of people who say they are doing surveys are just gathering a lot of comments.” Zagat points out, for instance, that professional surveyors will often poll a very large group of people and pull out a much smaller, demographically balanced sample to get more useful numbers.

You can ask a lot of people a question, but there’s no guarantee you will get a sensible answer. But while this fact of Web surveys might leave some experts and traditionalists aghast, consider this: We love surveys and ratings not so much because of what they tell us about the product, the movie or the candidate at issue, but because they tell us about the people doing the ranking. If the ratings were always accurate, we would surely be disappointed. After all we do not read them just to get good advice, but often read them (just as we watched “Family Feud”) for the delicious chance to disagree with the numbers and complain about how silly they are.

Mark Gimein is a staff writer for Salon Technology.

More Related Stories

Featured Slide Shows

  • Share on Twitter
  • Share on Facebook
  • 1 of 14
  • Close
  • Fullscreen
  • Thumbnails

    13 of "Girls'" most cringeworthy sex scenes

    Hannah and Adam, "Pilot"

    One of our first exposures to uncomfortable “Girls” sex comes early, in the pilot episode, when Hannah and Adam “get feisty” (a phrase Hannah hates) on the couch. The pair is about to go at it doggy-style when Adam nearly inserts his penis in “the wrong hole,” and after Hannah corrects him, she awkwardly explains her lack of desire to have anal sex in too many words. “Hey, let’s play the quiet game,” Adam says, thrusting. And so the romance begins.

    13 of "Girls'" most cringeworthy sex scenes

    Marnie and Elijah, "It's About Time"

    In an act of “betrayal” that messes up each of their relationships with Hannah, Marnie and Elijah open Season 2 with some more couch sex, which is almost unbearable to watch. Elijah, who is trying to explore the “hetero side” of his bisexuality, can’t maintain his erection, and the entire affair ends in very uncomfortable silence.

    13 of "Girls'" most cringeworthy sex scenes

    Marnie and Charlie, "Vagina Panic"

    Poor Charlie. While he and Marnie have their fair share of uncomfortable sex over the course of their relationship, one of the saddest moments (aside from Marnie breaking up with him during intercourse) is when Marnie encourages him to penetrate her from behind so she doesn’t have to look at him. “This feels so good,” Charlie says. “We have to go slow.” Poor sucker.

    13 of "Girls'" most cringeworthy sex scenes

    Shoshanna and camp friend Matt, "Hannah's Diary"

    We’d be remiss not to mention Shoshanna’s effort to lose her virginity to an old camp friend, who tells her how “weird” it is that he “loves to eat pussy” moments before she admits she’s never “done it” before. At least it paves the way for the uncomfortable sex we later get to watch her have with Ray?

    13 of "Girls'" most cringeworthy sex scenes

    Hannah and Adam, "Hard Being Easy"

    On the heels of trying (unsuccessfully) to determine the status of her early relationship with Adam, Hannah walks by her future boyfriend’s bedroom to find him masturbating alone, in one of the strangest scenes of the first season. As Adam jerks off and refuses to let Hannah participate beyond telling him how much she likes watching, we see some serious (and odd) character development ... which ends with Hannah taking a hundred-dollar bill from Adam’s wallet, for cab fare and pizza (as well as her services).

    13 of "Girls'" most cringeworthy sex scenes

    Marnie and Booth Jonathan, "Bad Friend"

    Oh, Booth Jonathan -- the little man who “knows how to do things.” After he turns Marnie on enough to make her masturbate in the bathroom at the gallery where she works, Booth finally seals the deal in a mortifying and nearly painful to watch sex scene that tells us pretty much everything we need to know about how much Marnie is willing to fake it.

    13 of "Girls'" most cringeworthy sex scenes

    Tad and Loreen, "The Return"

    The only sex scene in the series not to feature one of the main characters, Hannah’s parents’ showertime anniversary celebration is easily one of the most cringe-worthy moments of the show’s first season. Even Hannah’s mother, Loreen, observes how embarrassing the situation is, which ends with her husband, Tad, slipping out of the shower and falling naked and unconscious on the bathroom floor.

    13 of "Girls'" most cringeworthy sex scenes

    Hannah and the pharmacist, "The Return"

    Tad and Loreen aren’t the only ones to get some during Hannah’s first season trip home to Michigan. The show’s protagonist finds herself in bed with a former high school classmate, who doesn’t exactly enjoy it when Hannah puts one of her fingers near his anus. “I’m tight like a baby, right?” Hannah asks at one point. Time to press pause.

    13 of "Girls'" most cringeworthy sex scenes

    Hannah and Adam, "Role-Play"

    While it’s not quite a full-on, all-out sex scene, Hannah and Adam’s attempt at role play in Season 3 is certainly an intimate encounter to behold (or not). Hannah dons a blond wig and gets a little too into her role, giving a melodramatic performance that ends with a passerby punching Adam in the face. So there’s that.

    13 of "Girls'" most cringeworthy sex scenes

    Shoshanna and Ray, "Together"

    As Shoshanna and Ray near the end of their relationship, we can see their sexual chemistry getting worse and worse. It’s no more evident than when Ray is penetrating a clothed and visibly horrified Shoshanna from behind, who ends the encounter by asking if her partner will just “get out of me.”

    13 of "Girls'" most cringeworthy sex scenes

    Hannah and Frank, "Video Games"

    Hannah, Jessa’s 19-year-old stepbrother, a graveyard and too much chatting. Need we say more about how uncomfortable this sex is to watch?

    13 of "Girls'" most cringeworthy sex scenes

    Marnie and Desi, "Iowa"

    Who gets her butt motorboated? Is this a real thing? Aside from the questionable logistics and reality of Marnie and Desi’s analingus scene, there’s also the awkward moment when Marnie confuses her partner’s declaration of love for licking her butthole with love for her. Oh, Marnie.

    13 of "Girls'" most cringeworthy sex scenes

    Hannah and Adam, "Vagina Panic"

    There is too much in this scene to dissect: fantasies of an 11-year-old girl with a Cabbage Patch lunchbox, excessive references to that little girl as a “slut” and Adam ripping off a condom to ejaculate on Hannah’s chest. No wonder it ends with Hannah saying she almost came.

  • Recent Slide Shows



Comment Preview

Your name will appear as username ( settings | log out )

You may use these HTML tags and attributes: <a href=""> <b> <em> <strong> <i> <blockquote>