On Sept. 17, the Gallup Organization, the polling industry's oldest and most respected firm, released a survey devastating to the prospects of John Kerry's campaign for the presidency. After interviewing more than a thousand people, the company determined that Americans "likely" to vote in November preferred George W. Bush to Kerry by a 13-point margin -- 55 percent to 42 percent.
Bush's lead in the poll marked "the first statistically significant edge either candidate has held this year," reported USA Today, which, along with CNN, is one of the main media sponsors of the Gallup Poll. The paper added: "The boost Bush received from the Republican convention has increased rather than dissipated, reshaping a race that for months has been nearly tied."
At the time, a number of other polls were suggesting a much closer race, though all gave Bush a lead among likely voters: A survey by the firm ICR had him up by 8 percent, one by the New Democrat Network showed Bush ahead by 5, and Democracy Corps, the political strategy organization run by a group of former Clinton strategists, put Bush's lead at just 1 percent.
Left-wing bloggers immediately suspected something was amiss with Gallup's apparently aberrant results, and one of them, Steve Soto of the Left Coaster, asked Gallup's representatives for a more detailed explanation of what kinds of voters Gallup considered "likely" to go to the polls on Election Day. Gallup courteously sent Soto a reply, one that convinced him, and Democrats all over the Web, that the firm's polls were rigged.
Gallup's survey, it turned out, included a large number of Republicans -- about 40 percent of the people polled identified themselves as being in the GOP, a number far higher than the Republican share of voters in recent presidential elections. Of course Gallup's poll was showing a huge Bush win, Democrats cried -- the company was calling too many Republicans! The brewing lefty anger at Gallup hit a fever pitch on Thursday, when MoveOn.org, the liberal advocacy group, purchased a full-page ad in the New York Times denouncing Gallup's methodology. "Why does America's top pollster keep getting it wrong?" the ad asked in bazillion-point type. It also noted that George Gallup Jr., the son of the Gallup Organization's founder and a longtime executive of the firm, is "a devout evangelical Christian." MoveOn didn't specify why that was important, but the implication was clear: Gallup is on George W. Bush's side in this election.
Is Gallup's poll pulling for Bush? The short answer is no; polling experts, even Democratic polling experts, consider Gallup transcendently nonpartisan, one of the survey industry's straightest shooters. Several pollsters say they resent MoveOn's attack on Gallup. But there's a more important side to the kerfuffle over the Gallup Poll, one that lays bare not only legitimate questions over Gallup's methodology but also, more generally, the possible shortcomings of all election polls as well as the mistakes the public and the media make in interpreting them. In addition, there are many reasons, these days, to be broadly suspicious of the truth according to pollsters. Not the least of them is that an increasingly large share of the population fails to respond to pollsters' calls (a phenomenon that may be responsible for Gallup's odd Sept. 17 poll results) and are possibly evading surveyors altogether by using cellphones and caller I.D. systems. In a tight race, these concerns are more consequential; and the polling industry sees no good way around the problems in the long run.
We live, today, in an era of polling ubiquity. In the 2004 election, we'll probably have more polls from more organizations over more topics than we've ever had before, and the public will enjoy far greater access to these polls than in the past. The many polls dictate media coverage and campaign strategy, determining from week to week and day to day how journalists and insiders call the race -- not only who's up and who's down but why, how, where and what they should do about it.
Indeed, right now, a new Gallup Poll, released Sunday, shows the presidential race tied, with Bush and Kerry at 49 percent among likely voters. Other polls this week variously show Kerry leading, Bush leading, or neither. What should you make of those numbers? Perhaps the best thing to do is resist their pull.
The Gallup Poll's mid-September survey showing a huge Bush lead was based on interviews the firm conducted with 1,022 American adults over the course of three days. The main question Gallup asked was, "Suppose that the presidential election were being held today, and it included John Kerry and John Edwards as the Democratic candidates, and George W. Bush and Dick Cheney as the Republican candidates. Would you vote for John Kerry and John Edwards, the Democrats, or George W. Bush and Dick Cheney, the Republicans?" (The order of the candidates' names was varied for each interview.)
Although it was later discovered that Gallup's poll included a large number of Republicans, Gallup did not choose to call people it already knew were Republicans, Democrats or independents -- a common misconception online. Instead, in conducting election polls, Gallup and other firms go to a great deal of trouble to ensure that the people they interview are chosen completely randomly. Theoretically, the greater the randomness with which the sample is selected, the closer the sample will approximate the views of the larger population. To get this random sample, Gallup dials phone numbers constructed by a random-digit dialing machine, thereby including unlisted phones in its survey. Polling firms repeatedly call numbers that are busy or unresponsive. Still, despite such efforts, not many people cooperate with pollsters. A typical response rate -- the number of interviews conducted out of all the phone numbers attempted -- is under 30 percent, a statistic that has been declining over the past two decades. In order to reach 1,022 people for its poll, Gallup had to call thousands more who didn't respond or who refused to take part.
When it gets people on the phone, Gallup peppers them with questions about the election -- but although they interview all the respondents, pollsters know that not all of the people contacted will make it to the polls on Election Day. Since you only want to know which candidate is preferred by the people who will vote, polling firms need to guess which respondents will cast a ballot -- and this turns out to be one of the trickiest things in election polling. "There's no single question you can ask people that will capture the future fact" of whether they will vote, says Michael Dimock, research director at the Pew Research Center for the People and the Press. "We ask people flat out, 'Do you plan to vote?' and 95 percent of the people say they do. But we know that's not the case -- we don't get 95 percent of the people voting on Election Day."
Different polling organizations use different methods for sorting likely voters from unlikely voters, but Gallup's method is generally acknowledged to be more elaborate than that of other polls. Gallup says that the system is designed to increase the accuracy of the poll; critics, though, charge that the likely-voter methodology gives some candidates an unfair edge in the results.
Gallup attempts to gauge a respondent's likelihood of voting through a series of questions about an impending election. For instance, "How much thought have you given to the upcoming election for president -- quite a lot, or only a little?" Or, " Do you happen to know where people who live in your neighborhood go to vote?" "How often would you say you vote -- always, nearly always, part of the time, or seldom?" "In the last presidential election, did you vote for [candidate name] or [candidate name], or did things come up to keep you from voting?"
Gallup asks seven such questions; respondents who answer "correctly" on all of them are deemed likely voters. Of the 1,022 people interviewed for Gallup's mid-September poll, 767 of them -- about 75 percent -- made it through this likely-voter screen, and it was among this smaller sample of respondents that Bush beat Kerry by a 13-point margin. But it's important to note what Gallup means by the term "likely voter." To Gallup, the 767 respondents who answered its likely-voter questions correctly are not necessarily going to vote in the November election. Rather, the firm is saying that if an election were held today, it's these 767 people -- and the tens of millions in the population they represent -- who are most likely to go to the polls.
This is a subtle point, one not frequently explained in media coverage of poll results. Pollsters are not psychics; when they issue their election surveys from on high, they are not telling us what will happen tomorrow or three weeks or three months from now. Instead, and far less usefully, they are telling us what the public mood is today. Frank Newport, editor in chief of the Gallup Poll, compares preelection polling results to football scores taken during the course of a game. Obviously, the scores can be useful. A third-quarter score tells you something -- which team is doing better, which team is winning so far. But as a predictor of the final result, the third-quarter score is not perfect, for one obvious reason: Things can change. In a political campaign, as on the football field, winners can become losers, or vice versa, dramatically, quickly. A team can squander its momentum or, out of nowhere, come from behind with a winning play.
Presidential debates, like passes on the football field, can be fumbled. Speaking to Salon last week before John Kerry met George Bush in the first debate, Newport cited the 1980 race between Jimmy Carter and Ronald Reagan. Prior to the first debate that year, Carter was polling ahead. "But Reagan had a great debate performance," Newport recounts, "and then the Iran hostage anniversary came in, and in the end Reagan won by 10." Newport added: "With three debates still coming in this election, we wouldn't want to predict what's going to happen on Election Day."
Even if pollsters would prefer that you look at their polls as a snapshot of where the race stands rather than as a crystal ball showing where the race will end, it's only natural to wonder, when you see a certain poll result, how it may approximate the final result. It's on this point that critics of Gallup sharpen their claws. The critics charge that in the early part of the race -- weeks before the election, say -- Gallup's "likely voter" methodology often results in numbers that look extremely unlikely to come about on Election Day.
Ruy Teixeira, a Democratic polling expert and Web pundit, is chief among these critics. He points out that in the last three presidential elections, the pattern in which voters in exit polls identify themselves as either Republican, Democrat or independent hasn't changed much. In 1992, 34 percent of the voters called themselves Democrats, 34 percent said they were Republicans, and 33 percent claimed to be independents. In 1996, 39 percent were Democrats, 34 percent were Republicans, and 27 percent were independents. In 2000, it was roughly the same, with 39 percent Democrats, 35 percent Republicans, and 26 percent independents. Teixeira says that these numbers show that, over time, how people think of themselves politically remains fundamentally steady. If you're a Democrat, you're a Democrat, and you'll call yourself a Democrat for a long time -- even if, say, 9/11 persuaded you to vote for a few Republicans, you'll still think of yourself as a Democrat. What this means, Teixeira says, is that in a sense we already know something about what we should expect to see on Election Day -- in all likelihood, we'll have slightly more Democrats at the polls than Republicans, and more Republicans than independents.
But Gallup's likely-voter results of Sept. 17 show far more Republicans than Democrats. Forty percent of the people whom Gallup tagged as likely to vote identified themselves as Republicans, 33 percent said they were Democrats, and 28 percent called themselves independents. As Teixeira sees it, this result could mean one of two things. The first is that a fundamental shift has occurred in the electorate, a recent event that turned a lot of Democrats into Republicans. Teixeira calls this idea ludicrous. "Does it make sense to have a change in party I.D. over this short period of time?" he asks. "I don't think so. I don't think it switches this much that fast."
The other possibility, the one that Teixeira says is more probable, is that Gallup's likely-voter method is biased. In short, he believes that the battery of questions Gallup asked its respondents in mid-September to determine whether they were likely to vote was allowing more Republicans into the likely-voter pool than Democrats. "They ask the questions in such way that what you get in the likely pool is one form of political interest," he says. "And it doesn't take much, because of the low response rates. Any kind of partisan behavior is going to produce a partisan result. All of that bias gets imported into your poll."
Teixeira doesn't think that Gallup's likely-voter methodology is biased in favor of Republicans per se. Instead, he says, it's biased in favor of the "more activated" party, the one that that feels more enthusiastic about its candidate. His idea is that voters who feel good about their candidate probably have an easier time answering the questions Gallup uses to determine who's going to vote. If you were excited about Bush's performance at the Republican Convention, you may have gone and found out where your polling place is and you may tell the interviewer that there's no way you're going to miss voting on Election Day. Meanwhile, if you're a Kerry supporter and Bush's convention speech demoralized you, gave you the idea that your candidate could never beat him, you may have become apathetic about the project of voting. You don't care where your polling place is; you're not even really thinking about the election. In this example, the Bush voter would be counted as likely; the Kerry voter unlikely. And even though no voters have really changed their preferences -- the Kerry voters are still sticking with Kerry, the Bush voters sticking with Bush -- Bush will suddenly appear to be doing better in the polls, because his voters are now considered likelier to vote.
This is no idle theory of Teixeira's. In a forthcoming paper in Public Opinion Quarterly, the bible of polling research, the political scientists Robert Erikson, Costas Panagopoulous and Christopher Wlezien report on a study they conducted of Gallup's likely-voter model during the 2000 presidential election. The likely-voter poll that year, they note, resulted in extraordinarily volatile shifts in opinion. On Oct. 3, 2000, the day of the first debate, Bush trailed Gore by 51 to 40 among likely voters. But a few days later, Bush was ahead, 49 to 41. Did people really swing so suddenly from Gore to Bush? After analyzing data provided by Gallup, the scientists conclude that such a shift did not occur. "Our evidence suggests that shifts in voter classification as likely or unlikely account for more observed change in the preferences of likely voters than do actual changes in voters' candidate preferences," they say. Much of the volatility in the poll "is an artifact of classification" and not the result of people changing their support from one candidate to another.
Gallup, for its part, doesn't dispute this idea. To Gallup, stronger results should be reported for a candidate whose base is "more activated." A candidate who has more "likely" voters would indeed probably win the election if one were held today. In the two polls that Gallup has conducted since the Sept. 17 poll, Kerry's base has been the more activated and, consequently, better represented in the likely-voter pool. In the poll released on Sunday, Kerry is tied with Bush among likely voters; but among registered voters, Bush is ahead by a couple of points. This means that Gallup deems at least some of Bush's voters "unlikely" -- and, considering that Bush's debate performance may have demoralized some of his voters, that's a reasonable assumption, isn't it?
While there are legitimate questions over Gallup's polling methodology among pollsters, almost nobody believes that Gallup has a political ax to grind. Mark Blumenthal, the Democratic campaign pollster who runs the fine poll-explaining blog Mystery Pollster, says that he has questions about Gallup's methodology. But when he saw MoveOn's anti-Gallup ad, he was "taken aback by its ferocity," he writes. In an interview, he added, "The nastiness of the ad was uncalled-for. I don't think these people are intentionally slanting the results. I've met some of the people who run Gallup and they're professionals. You may not agree with what they do, and it's perfectly appropriate to question their methodologies, but I think taking out a full-page ad to claim they're slanted is too far."
There are, to be sure, bad polls available these days -- polls with slanted questions, polls that try to save money by calling only people who are listed, polls that seem always to deliver the kind of result that the person paying for it seems to want. Gallup, pollsters say, is not one of those. "That's just garbage," says Michael Dimock of Pew. "It's the gold standard of polls in many ways. They have an incredible record behind them, they have very smart people there, and they're social scientists, not partisans."
Frank Newport, of Gallup, says that from its earliest days, people who run the Gallup Poll have tried to remain scrupulously nonpartisan. George Gallup, the company's founder, refused even to vote. Newport himself says that he does not vote in primary elections. It's true that George Gallup Jr., the founder's son, whom MoveOn criticized in its ad, is a devout Christian; but when he told an audience in Massachusetts in June that he thought "the most profound purpose of polls is to see how people are responding to God," he was referring specifically to polling on religion, not to political polling, as MoveOn's ad suggested. And, Blumenthal asks, what's wrong with being a devout pollster? "Does that mean that pollsters aren't allowed to go to church? We can't go to a synagogue?"
Of all there is in this world for a liberal advocacy group to complain about in a full-page ad in a national newspaper, why did MoveOn choose to go after Gallup? Because polls can also affect the public mood; a poll showing Kerry way behind, for instance, may demoralize his supporters and fire up Bush's base. The poll also clearly affects media coverage, determining whether journalists write about your campaign as if it has no chance or, instead, as if you're within fighting distance. Polls, especially Gallup's polls, are important. They're the only measure we have of how a race is going, and election season is mainly the act of constructing storylines to fit the newest numbers.
But in a razor-thin race, this is dangerous business. Pollsters this year really don't know how good their numbers are. On Sept. 16, a day before the much disputed Gallup poll was released, the columnist Jimmy Breslin wrote a blistering attack on the entire enterprise of polling. The whole thing is a "sham," he wrote, because pollsters call land-line telephones, skipping the hundreds of millions of cellphones in America. "Any editors of newspapers or television news shows who use poll results as a story are beyond gullible," he wrote. "On behalf of the public they profess to serve, they are indolent salesmen of falsehoods."
That column blazed around the Internet, and it's possible to find, now, people who refuse to believe that polls are accurate because of the cellphone problem. Pollsters, for their part, don't believe this is a big concern just yet, because the overwhelming majority of people who have cellphones also have land lines, and so are still reachable by traditional telephone polls. But Blumenthal says that recent survey industry conferences have been dominated by this question of what pollsters will do when cellphones become the only way people communicate. Can pollsters get around the hurdle of calling phones that charge by air time? Can they figure out a way to tie a phone to a particular geographic location in an age in which cellphone numbers are portable?
These are hard, hard problems the polling industry needs to solve. And they come at a time when national races seem especially close -- the era of the 50-50 nation. Polling has perhaps never been as crucial to the everyday pace of a race as it is in this presidential election. But consequently, polling has never been quite as disputed, as fought-over, before, either.
"Pollsters have been sucked into the battlefield of politics," Newport laments. But considering how important they are, it's perhaps naive to have expected them to remain pure.
Shares