The consumer's always wrong

Why else would visitors to consumer rating sites like rank Rolling Rock the second-best beer and Alan Keyes the top presidential candidate?


As a child, I spent several years of afternoons happily engrossed in “Family Feud,” and ever since I have had a mania for polls and surveys. Not only have I always wanted to know what other people think — this is a basic human trait — but I have always had a particularly urgent need to know exactly what percentage of other people are thinking it. Since “Family Feud” asked contestants to guess the results of a survey of 100 Americans, (contestants would be presented with questions like, “Name an animal likely to be found on a farm,” and would score the most points for guessing the most common answers), my vision of utopia, is knowing what 100 Americans think about everything, all the time: what percentage of them like the new Fox sitcom; what percentage think Kenmore dryers chew up their clothes; and what percentage think Margaret Thatcher is still the prime minister of Great Britain.

My little fantasy of having instant access to a compendium of opinion surveys might have been preposterous before the Web, and yet in just a few months we have gotten much closer than we ever previously imagined to my peculiar vision of a plebiscite society.

When launched its online bookstore, its most striking feature (besides the simple fact that it let people buy books online) was that Amazon encouraged users to rate and discuss books in what amounted to virtual bookstore aisles. Being able to find out what other readers think about books that you’re thinking of buying was a natural and enticing use of the Internet.

Amazon itself has extended the ratings idea to a host of products besides books. And the initial idea has spawned a whole new sector of the Web: sites devoted to consumer ratings.

Consumers instantly reaching a wide audience with their take on whether their money was well spent is shaping up to be one of the most dramatic effects of the Internet. For professional marketers, it is undoubtedly also one of the scariest. Remember the advice your mother/teacher/best friend gave you in, oh, about third grade? “Don’t get mad, get even”? A lot of people now have a way to get even with companies they believe gave them a bad deal — and they see getting even as a public service.

Take Greg Plough, a onetime Prodigy customer who was so dissatisfied with the service he got from the Internet access provider that he posted a dismal review online. “I wanted to send out a warning to the millions of people who are getting shafted by [Prodigy's] rebate deal,” he wrote on one consumer review site.

The most interesting twist to consumer ratings, however, comes from sites like and Consumer Review. They don’t simply rate products (or, in’s case, not just products, but pretty much everything — from magazines to political candidates to cat breeds) but assemble the ratings into neat lists of the best and the worst.

A lot of the fun of rating things — most of the fun, actually — is that once you’ve rated them, you can rank them, too. Let’s face it, we don’t just want to know whether a particular movie is good or whether a particular video recorder is reliable. No, we want to know its particular and very precise degree of goodness, and before we shell out our money we want to know if that camcorder’s degree of goodness is even a shade of a fraction less than some other model’s.

Both Consumer Review and distill user comments into an average rating, from 1 to 5. does it in an especially elegant way, copying the “Zagat Survey” approach to rating restaurants by creating four measures in each category (like “comfort,” “features,” “reliability” and “cost” for sedans, or, a bit strangely, “delivery,” “interview style,” “reporting” and (!) “sex appeal” for television newscasters) to create a veritable feast of numerical ratings. links from the product ratings directly to retailers and Consumer Review has plans for a similar system. Both sites are betting that consumers will use the ratings to make purchase decisions — and entitle the sites to a cut of any sales made.

If, like me, you loved “Family Feud” and are a sucker for the nifty chart at the back of Entertainment Weekly which compresses the results of a dozen different critics’ responses to a movie into one handy little chart, you’ll probably also find yourself a junkie for the survey results at Consumer Review and But the more you look at the ratings, the more you are likely to notice some odd results.

The first peculiar result of the proliferation of online surveys is what I have come to think of as “the Philadelphia effect.” The term is mine, but credit for noticing it goes to Tim Zagat, the co-creator (with his wife, Nina) of the Zagat guides.

The Philadelphia effect takes its name from a survey conducted some time ago by a glossy magazine. “There was a magazine that surveyed its readers about what was the best restaurant city in America,” recalls Zagat. “The result was Philadelphia. I almost choked.”

Of course, there are undoubtedly people (most of them, presumably, native Philadelphians) who will be offended by Tim Zagat’s blithe dismissal of the opinions of hundreds of magazine readers. Who is Zagat to say that Philadelphia isn’t the best eating city in America?

If, however, you do not live in Philadelphia, you might be inclined to the view that the fact that Philadelphia was named as the best dining city in America says a lot less about Philadelphia or its restaurants than it does about the Philadelphians who we can safely presume are the restaurants’ most frequent customers. It might well mean not that the restaurants are the best, but that the patrons are among the least demanding. The ratings wind up telling you a lot more about the raters than they do about the city or its restaurants.

Online, it’s really easy to see the Philadelphia effect in action. Consider, for instance, the ratings for beer.

Even if you are not a beer drinker, it is likely that you know that supermarket shelves are filled with microbrews like Pete’s Wicked Ale or Samuel Adams, and plenty of high-priced imports like Guinness. Many of these are quite good. Expensive, but good. And some of the same beers that win competitions also score well in’s ranking.

But up at No. 2 in’s rankings is an inexpensive macro-brew much loved by college students: Rolling Rock. (The top spot goes to Yuengling Lager.) I know a lot about Rolling Rock because in college, when I drank beer, Rolling Rock was the beer that I drank. And it’s not a bad beer. But the second-best in the country? Better than all the beers that sell at three times the price? Well, maybe. But not very likely. In fact, it’s a good bet that even most of the people who praised Rolling Rock on the site, would not rank it so highly in a side-by-side comparison.

Of course, it’s hard to prove that Rolling Rock isn’t the second best beer in the country, just as it’s very hard to prove that Philadelphia doesn’t have the country’s best restaurants. But it’s an awfully suspect result.

The Philadelphia effect has a corollary, which you can think of as the Alan Keyes tsunami. The Philadelphia effect is the result of a lot of people, who are easy to satisfy, giving a surprisingly high rating to Philadelphia restaurants or Rolling Rock beer. The Alan Keyes effect is the result of a lot of people who have a very strong opinion on one product (or, in Alan Keyes’ case, one political candidate) sign on en masse and flood a site with reviews.

The Alan Keyes tsunami gets its name from a Republican presidential candidate. If you haven’t heard of him, you’re not alone. In August’s Iowa straw poll, Keyes got only 5 percent of the vote among a collection of Republican die-hards. Poor Alan Keyes is on his second run (very likely you didn’t hear about the first one either), gamely shambling between pancake breakfasts with nary a television camera in sight, but at least he has this consolation:’s raters are firmly in his camp.

In fact, Keyes scores highest of all the presidential candidates rated on the site. He is followed by John McCain, Bill Bradley, George W. Bush, Steve Forbes; Jesse Ventura, the former wrestler turned Minnesota governor scored higher than Keyes — but he’s not even running. Keyes’ popularity is no statistical fluke. At last count, 1,971 people put in scores for Keyes — fewer than the nearly 4,000 who expressed an opinion about George W. Bush — but a very respectable sample pool nonetheless.

Even the people behind seem to take some of the ratings with a grain of salt. “We’re definitely not the U.S. Census,” cautions executive vice president David Wilson.

Results like this can make old-school experts in the art of consumer surveys apoplectic. Says Zagat: “A lot of people who say they are doing surveys are just gathering a lot of comments.” Zagat points out, for instance, that professional surveyors will often poll a very large group of people and pull out a much smaller, demographically balanced sample to get more useful numbers.

You can ask a lot of people a question, but there’s no guarantee you will get a sensible answer. But while this fact of Web surveys might leave some experts and traditionalists aghast, consider this: We love surveys and ratings not so much because of what they tell us about the product, the movie or the candidate at issue, but because they tell us about the people doing the ranking. If the ratings were always accurate, we would surely be disappointed. After all we do not read them just to get good advice, but often read them (just as we watched “Family Feud”) for the delicious chance to disagree with the numbers and complain about how silly they are.

Mark Gimein is a staff writer for Salon Technology.

More Related Stories

Featured Slide Shows

  • Share on Twitter
  • Share on Facebook
  • 1 of 22
  • Close
  • Fullscreen
  • Thumbnails

    Once upon a time on the Bowery

    Talking Heads, 1977
    This was their first weekend as a foursome at CBGB’s, after adding Jerry Harrison, before they started recording the LP “Talking Heads: 77.”

    Once upon a time on the Bowery

    Patti Smith, Bowery 1976
    Patti lit up by the Bowery streetlights. I tapped her on the shoulder, asked if I could do a picture, took two shots and everyone went back to what they were doing. 1/4 second at f/5.6 no tripod.

    Once upon a time on the Bowery

    Blondie, 1977
    This was taken at the Punk Magazine Benefit show. According to Chris Stein (seated, on slide guitar), they were playing “Little Red Rooster.”

    Once upon a time on the Bowery

    No Wave Punks, Bowery Summer 1978
    They were sitting just like this when I walked out of CBGB's. Me: “Don’t move” They didn’t. L to R: Harold Paris, Kristian Hoffman, Diego Cortez, Anya Phillips, Lydia Lunch, James Chance, Jim Sclavunos, Bradley Field, Liz Seidman.

    Once upon a time on the Bowery

    Richard Hell + Bob Quine, 1978
    Richard Hell and the Voidoids, playing CBGB's in 1978, with Richard’s peerless guitar player Robert Quine. Sorely missed, Quine died in 2004.

    Once upon a time on the Bowery

    Bathroom, 1977
    This photograph of mine was used to create the “replica” CBGB's bathroom in the Punk Couture show last summer at the Metropolitan Museum of Art. So I got into the Met with a bathroom photo.

    Once upon a time on the Bowery

    Stiv Bators + Divine, 1978
    Stiv Bators, Divine and the Dead Boys at the Blitz Benefit show for injured Dead Boys drummer Johnny Blitz.

    Once upon a time on the Bowery

    Ramones, 1977
    “The kids are all hopped up and ready to go…” View from the unique "side stage" at CBGB's that you had to walk past to get to the basement bathrooms.

    Once upon a time on the Bowery

    Klaus Nomi, Christopher Parker, Jim Jarmusch – Bowery 1978
    Jarmusch was still in film school, Parker was starring in Jim’s first film "Permanent Vacation" and Klaus just appeared out of nowhere.

    Once upon a time on the Bowery

    Hilly Kristal, Bowery 1977
    When I used to show people this picture of owner Hilly Kristal, they would ask me “Why did you photograph that guy? He’s not a punk!” Now they know why. None of these pictures would have existed without Hilly Kristal.

    Once upon a time on the Bowery

    Dictators, Bowery 1976
    Handsome Dick Manitoba of the Dictators with his girlfriend Jody. I took this shot as a thank you for him returning the wallet I’d lost the night before at CBGB's. He doesn’t like that I tell people he returned it with everything in it.

    Once upon a time on the Bowery

    Alex Chilton, Bowery 1977
    We were on the median strip on the Bowery shooting what became a 45 single sleeve for Alex’s “Bangkok.” A drop of rain landed on the camera lens by accident. Definitely a lucky night!

    Once upon a time on the Bowery

    Bowery view, 1977
    The view from across the Bowery in the summer of 1977.

    Once upon a time on the Bowery

    Ramones, 1977 – never before printed
    I loved shooting The Ramones. They would play two sets a night, four nights a week at CBGB's, and I’d be there for all of them. This shot is notable for Johnny playing a Strat, rather than his usual Mosrite. Maybe he’d just broken a string. Love that hair.

    Once upon a time on the Bowery

    Richard Hell, Bowery 1977 – never before printed
    Richard exiting CBGB's with his guitar at 4am, about to step into a Bowery rainstorm. I’ve always printed the shots of him in the rain, but this one is a real standout to me now.

    Once upon a time on the Bowery

    Patti Smith + Ronnie Spector, 1979
    May 24th – Bob Dylan Birthday show – Patti “invited” everyone at that night’s Palladium show on 14th Street down to CBGB's to celebrate Bob Dylan’s birthday. Here, Patti and Ronnie are doing “Be My Baby.”

    Once upon a time on the Bowery

    Legs McNeil, 1977
    Legs, ready for his close-up, near the front door of CBGB's.

    Once upon a time on the Bowery

    Suicide, 1977
    Rev and Alan Vega – I thought Alan was going to hit me with that chain. This was the Punk Magazine Benefit show.

    Once upon a time on the Bowery

    Ian Hunter and Fans, outside bathroom
    I always think of “All the Young Dudes” when I look at this shot. These fans had caught Ian Hunter in the CBGB's basement outside the bathrooms, and I just stepped in to record the moment.

    Once upon a time on the Bowery

    Tommy Ramone, 1977
    Only at CBGB's could I have gotten this shot of Tommy Ramone seen through Johnny Ramones legs.

    Once upon a time on the Bowery

    Bowery 4am, 1977
    End of the night garbage run. Time to go home.

  • Recent Slide Shows



Comment Preview

Your name will appear as username ( settings | log out )

You may use these HTML tags and attributes: <a href=""> <b> <em> <strong> <i> <blockquote>