The consumer's always wrong

Why else would visitors to consumer rating sites like rank Rolling Rock the second-best beer and Alan Keyes the top presidential candidate?


As a child, I spent several years of afternoons happily engrossed in “Family Feud,” and ever since I have had a mania for polls and surveys. Not only have I always wanted to know what other people think — this is a basic human trait — but I have always had a particularly urgent need to know exactly what percentage of other people are thinking it. Since “Family Feud” asked contestants to guess the results of a survey of 100 Americans, (contestants would be presented with questions like, “Name an animal likely to be found on a farm,” and would score the most points for guessing the most common answers), my vision of utopia, is knowing what 100 Americans think about everything, all the time: what percentage of them like the new Fox sitcom; what percentage think Kenmore dryers chew up their clothes; and what percentage think Margaret Thatcher is still the prime minister of Great Britain.

My little fantasy of having instant access to a compendium of opinion surveys might have been preposterous before the Web, and yet in just a few months we have gotten much closer than we ever previously imagined to my peculiar vision of a plebiscite society.

When launched its online bookstore, its most striking feature (besides the simple fact that it let people buy books online) was that Amazon encouraged users to rate and discuss books in what amounted to virtual bookstore aisles. Being able to find out what other readers think about books that you’re thinking of buying was a natural and enticing use of the Internet.

Amazon itself has extended the ratings idea to a host of products besides books. And the initial idea has spawned a whole new sector of the Web: sites devoted to consumer ratings.

Consumers instantly reaching a wide audience with their take on whether their money was well spent is shaping up to be one of the most dramatic effects of the Internet. For professional marketers, it is undoubtedly also one of the scariest. Remember the advice your mother/teacher/best friend gave you in, oh, about third grade? “Don’t get mad, get even”? A lot of people now have a way to get even with companies they believe gave them a bad deal — and they see getting even as a public service.

Take Greg Plough, a onetime Prodigy customer who was so dissatisfied with the service he got from the Internet access provider that he posted a dismal review online. “I wanted to send out a warning to the millions of people who are getting shafted by [Prodigy's] rebate deal,” he wrote on one consumer review site.

The most interesting twist to consumer ratings, however, comes from sites like and Consumer Review. They don’t simply rate products (or, in’s case, not just products, but pretty much everything — from magazines to political candidates to cat breeds) but assemble the ratings into neat lists of the best and the worst.

A lot of the fun of rating things — most of the fun, actually — is that once you’ve rated them, you can rank them, too. Let’s face it, we don’t just want to know whether a particular movie is good or whether a particular video recorder is reliable. No, we want to know its particular and very precise degree of goodness, and before we shell out our money we want to know if that camcorder’s degree of goodness is even a shade of a fraction less than some other model’s.

Both Consumer Review and distill user comments into an average rating, from 1 to 5. does it in an especially elegant way, copying the “Zagat Survey” approach to rating restaurants by creating four measures in each category (like “comfort,” “features,” “reliability” and “cost” for sedans, or, a bit strangely, “delivery,” “interview style,” “reporting” and (!) “sex appeal” for television newscasters) to create a veritable feast of numerical ratings. links from the product ratings directly to retailers and Consumer Review has plans for a similar system. Both sites are betting that consumers will use the ratings to make purchase decisions — and entitle the sites to a cut of any sales made.

If, like me, you loved “Family Feud” and are a sucker for the nifty chart at the back of Entertainment Weekly which compresses the results of a dozen different critics’ responses to a movie into one handy little chart, you’ll probably also find yourself a junkie for the survey results at Consumer Review and But the more you look at the ratings, the more you are likely to notice some odd results.

The first peculiar result of the proliferation of online surveys is what I have come to think of as “the Philadelphia effect.” The term is mine, but credit for noticing it goes to Tim Zagat, the co-creator (with his wife, Nina) of the Zagat guides.

The Philadelphia effect takes its name from a survey conducted some time ago by a glossy magazine. “There was a magazine that surveyed its readers about what was the best restaurant city in America,” recalls Zagat. “The result was Philadelphia. I almost choked.”

Of course, there are undoubtedly people (most of them, presumably, native Philadelphians) who will be offended by Tim Zagat’s blithe dismissal of the opinions of hundreds of magazine readers. Who is Zagat to say that Philadelphia isn’t the best eating city in America?

If, however, you do not live in Philadelphia, you might be inclined to the view that the fact that Philadelphia was named as the best dining city in America says a lot less about Philadelphia or its restaurants than it does about the Philadelphians who we can safely presume are the restaurants’ most frequent customers. It might well mean not that the restaurants are the best, but that the patrons are among the least demanding. The ratings wind up telling you a lot more about the raters than they do about the city or its restaurants.

Online, it’s really easy to see the Philadelphia effect in action. Consider, for instance, the ratings for beer.

Even if you are not a beer drinker, it is likely that you know that supermarket shelves are filled with microbrews like Pete’s Wicked Ale or Samuel Adams, and plenty of high-priced imports like Guinness. Many of these are quite good. Expensive, but good. And some of the same beers that win competitions also score well in’s ranking.

But up at No. 2 in’s rankings is an inexpensive macro-brew much loved by college students: Rolling Rock. (The top spot goes to Yuengling Lager.) I know a lot about Rolling Rock because in college, when I drank beer, Rolling Rock was the beer that I drank. And it’s not a bad beer. But the second-best in the country? Better than all the beers that sell at three times the price? Well, maybe. But not very likely. In fact, it’s a good bet that even most of the people who praised Rolling Rock on the site, would not rank it so highly in a side-by-side comparison.

Of course, it’s hard to prove that Rolling Rock isn’t the second best beer in the country, just as it’s very hard to prove that Philadelphia doesn’t have the country’s best restaurants. But it’s an awfully suspect result.

The Philadelphia effect has a corollary, which you can think of as the Alan Keyes tsunami. The Philadelphia effect is the result of a lot of people, who are easy to satisfy, giving a surprisingly high rating to Philadelphia restaurants or Rolling Rock beer. The Alan Keyes effect is the result of a lot of people who have a very strong opinion on one product (or, in Alan Keyes’ case, one political candidate) sign on en masse and flood a site with reviews.

The Alan Keyes tsunami gets its name from a Republican presidential candidate. If you haven’t heard of him, you’re not alone. In August’s Iowa straw poll, Keyes got only 5 percent of the vote among a collection of Republican die-hards. Poor Alan Keyes is on his second run (very likely you didn’t hear about the first one either), gamely shambling between pancake breakfasts with nary a television camera in sight, but at least he has this consolation:’s raters are firmly in his camp.

In fact, Keyes scores highest of all the presidential candidates rated on the site. He is followed by John McCain, Bill Bradley, George W. Bush, Steve Forbes; Jesse Ventura, the former wrestler turned Minnesota governor scored higher than Keyes — but he’s not even running. Keyes’ popularity is no statistical fluke. At last count, 1,971 people put in scores for Keyes — fewer than the nearly 4,000 who expressed an opinion about George W. Bush — but a very respectable sample pool nonetheless.

Even the people behind seem to take some of the ratings with a grain of salt. “We’re definitely not the U.S. Census,” cautions executive vice president David Wilson.

Results like this can make old-school experts in the art of consumer surveys apoplectic. Says Zagat: “A lot of people who say they are doing surveys are just gathering a lot of comments.” Zagat points out, for instance, that professional surveyors will often poll a very large group of people and pull out a much smaller, demographically balanced sample to get more useful numbers.

You can ask a lot of people a question, but there’s no guarantee you will get a sensible answer. But while this fact of Web surveys might leave some experts and traditionalists aghast, consider this: We love surveys and ratings not so much because of what they tell us about the product, the movie or the candidate at issue, but because they tell us about the people doing the ranking. If the ratings were always accurate, we would surely be disappointed. After all we do not read them just to get good advice, but often read them (just as we watched “Family Feud”) for the delicious chance to disagree with the numbers and complain about how silly they are.

Mark Gimein is a staff writer for Salon Technology.

More Related Stories

Featured Slide Shows

  • Share on Twitter
  • Share on Facebook
  • 1 of 11
  • Close
  • Fullscreen
  • Thumbnails

    Ten spectacular graphic novels from 2014

    Beautiful Darkness by Fabien Vehlmann & Kerascoët
    Kerascoët's lovely, delicate pen-and-watercolor art -- all intricate botanicals, big eyes and flowing hair -- gives this fairy story a deceptively pretty finish. You find out quickly, however, that these are the heartless and heedless fairies of folk legend, not the sentimental sprites beloved by the Victorians and Disney fans. A host of tiny hominid creatures must learn to survive in the forest after fleeing their former home -- a little girl who lies dead in the woods. The main character, Aurora, tries to organize the group into a community, but most of her cohort is too capricious, lazy and selfish to participate for long. There's no real moral to this story, which is refreshing in itself, beyond the perpetual lessons that life is hard and you have to be careful whom you trust. Never has ugly truth been given a prettier face.

    Ten spectacular graphic novels from 2014

    Climate Changed: A Personal Journey Through the Science by Philippe Squarzoni
    Squarzoni is a French cartoonist who makes nonfiction graphic novels about contemporary issues and politics. While finishing up a book about France under Jacques Chirac, he realized that when it came to environmental policy, he didn't know what he was talking about. "Climate Changed" is the result of his efforts to understand what has been happening to the planet, a striking combination of memoir and data that ruminates on a notoriously elusive, difficult and even imponderable subject. Panels of talking heads dispensing information (or Squarzoni discussing the issues with his partner) are juxtaposed with detailed and meticulous yet lyrical scenes from the author's childhood, the countryside where he takes a holiday and a visit to New York. He uses his own unreachable past as a way to grasp the imminent transformation of the Earth. The result is both enlightening and unexpectedly moving.

    Ten spectacular graphic novels from 2014

    Here by Richard McGuire
    A six-page version of this innovative work by a regular contributor to the New Yorker first appeared in RAW magazine 25 years ago. Each two-page spread depicts a single place, sometimes occupied by a corner of a room, over the course of 4 billion years. The oldest image is a blur of pink and purple gases; others depict hazmat-suited explorers from 300 years in the future. Inset images show the changing decor and inhabitants of the house throughout its existence: family photos, quarrels, kids in Halloween costumes, a woman reading a book, a cat walking across the floor. The cumulative effect is serene and ravishing, an intimation of the immensity of time and the wonder embodied in the humblest things.

    Ten spectacular graphic novels from 2014

    Kill My Mother by Jules Feiffer
    The legendary Pulitzer Prize-winning cartoonist delivers his debut graphic novel at 85, a deliriously over-the-top blend of classic movie noir and melodrama that roams from chiaroscuro Bay City to Hollywood to a USO gig in the Pacific theater of World War II. There's a burnt-out drunk of a private eye, but the story is soon commandeered by a multigenerational collection of ferocious women, including a mysterious chanteuse who never speaks, a radio comedy writer who makes a childhood friend the butt of a hit series and a ruthless dame intent on making her whiny coward of a husband into a star. There are disguises, musical numbers and plenty of gunfights, but the drawing is the main attraction. Nobody convey's bodies in motion more thrillingly than Feiffer, whether they're dancing, running or duking it out. The kid has promise.

    Ten spectacular graphic novels from 2014

    The Motherless Oven by Rob Davis
    This is a weird one, but in the nervy surreal way that word-playful novels like "A Clockwork Orange" or "Ulysses" are weird. The main character, a teenage schoolboy named Scarper Lee, lives in a world where it rains knives and people make their own parents, contraptions that can be anything from a tiny figurine stashable in a pocket to biomorphic boiler-like entities that seem to have escaped from Dr. Seuss' nightmares. Their homes are crammed with gadgets they call gods and instead of TV they watch a hulu-hoop-size wheel of repeating images that changes with the day of the week. They also know their own "death day," and Scarper's is coming up fast. Maybe that's why he runs off with the new girl at school, a real troublemaker, and the obscurely dysfunctional Castro, whose mother is a cageful of talking parakeets. A solid towline of teenage angst holds this manically inventive vision together, and proves that some graphic novels can rival the text-only kind at their own game.

    Ten spectacular graphic novels from 2014

    NOBROW 9: It's Oh So Quiet
    For each issue, the anthology magazine put out by this adventurous U.K.-based publisher of independent graphic design, illustration and comics gives 45 artists a four-color palette and a theme. In the ninth issue, the theme is silence, and the results are magnificent and full of surprises. The comics, each told in images only, range from atmospheric to trippy to jokey to melancholy to epic to creepy. But the two-page illustrations are even more powerful, even if it's not always easy to see how they pertain to the overall concept of silence. Well, except perhaps for the fact that so many of them left me utterly dumbstruck with visual delight.

    Ten spectacular graphic novels from 2014

    Over Easy by Mimi Pond
    When Pond was a broke art student in the 1970s, she took a job at a neighborhood breakfast spot in Oakland, a place with good food, splendid coffee and an endlessly entertaining crew of short-order cooks, waitresses, dishwashers and regular customers. This graphic memoir, influenced by the work of Pond's friend, Alison Bechdel, captures the funky ethos of the time, when hippies, punks and disco aficionados mingled in a Bay Area at the height of its eccentricity. The staff of the Imperial Cafe were forever swapping wisecracks and hopping in and out of each other's beds, which makes them more or less like every restaurant team in history. There's an intoxicating esprit de corps to a well-run everyday joint like the Imperial Cafe, and never has the delight in being part of it been more winningly portrayed.

    Ten spectacular graphic novels from 2014

    The Shadow Hero by Gene Luen Yang and Sonny Liew
    You don't have to be a superhero fan to be utterly charmed by Yang and Liew's revival of a little-known character created in the 1940s by the cartoonist Chu Hing. This version of the Green Turtle, however, is rich in characterization, comedy and luscious period detail from the Chinatown of "San Incendio" (a ringer for San Francisco). Hank, son of a mild-mannered grocer, would like to follow in his father's footsteps, but his restless mother (the book's best character and drawn with masterful nuance by Liew) has other ideas after her thrilling encounter with a superhero. Yang's story effortlessly folds pathos into humor without stooping to either slapstick or cheap "darkness." This is that rare tribute that far surpasses the thing it celebrates.

    Ten spectacular graphic novels from 2014

    Shoplifter by Michael Cho
    Corinna Park, former English major, works, unhappily, in a Toronto advertising agency. When the dissatisfaction of the past five years begins to oppress her, she lets off steam by pilfering magazines from a local convenience store. Cho's moody character study is as much about city life as it is about Corinna. He depicts her falling asleep in front of the TV in her condo, brooding on the subway, roaming the crowded streets after a budding romance goes awry. Like a great short story, this is a simple tale of a young woman figuring out how to get her life back, but if feels as if it contains so much of contemporary existence -- its comforts, its loneliness, its self-deceptions -- suspended in wintery amber.

    Ten spectacular graphic novels from 2014

    Through the Woods by Emily Carroll
    This collection of archetypal horror, fairy and ghost stories, all about young girls, comes lushly decked in Carroll's inky black, snowy white and blood-scarlet art. A young bride hears her predecessor's bones singing from under the floorboards, two friends make the mistake of pretending to summon the spirits of the dead, a family of orphaned siblings disappears one by one into the winter nights. Carroll's color-saturated images can be jagged, ornate and gruesome, but she also knows how to chill with absence, shadows and a single staring eye. Literary readers who cherish the work of Kelly Link or the late Angela Carter's collection, "The Bloody Chamber," will adore the violent beauty on these pages.

  • Recent Slide Shows



Comment Preview

Your name will appear as username ( settings | log out )

You may use these HTML tags and attributes: <a href=""> <b> <em> <strong> <i> <blockquote>