Claim: Exit polls are usually accurate.
"Over the past decades, exit polling has evolved into an exact science," he writes. "The results are exquisitely accurate." Kennedy points out that exits are often used to verify the integrity of an election -- he refers to Ukraine, where in 2004 exit polling "exposed election fraud that denied Viktor Yushchenko the presidency."
Essentially, Kennedy's argument goes like this: Exit poll numbers -- which are derived from interviews with voters after they've cast their votes -- showed us what voters actually wanted. The discrepancy between the exits and the final count indicates that something funny happened in the casting or counting of ballots in Ohio. If the Ukranian exits proved fraud, why don't those in the U.S?
Reality: "Nonsense," says Mark Blumenthal, the professional Democratic pollster who runs Mystery Pollster, the poll-scrutinizing blog that has comprehensively covered the exit poll story since Election Day. Anyone who says that exit polls are the most reliable kind of survey "only demonstrates that the person making that statement knows very little about how surveys are done," Blumenthal says.
Warren Mitofsky, the veteran pollster who conducted the exit poll for the networks, told me last year that he doesn't think the exits represent the gospel truth of what happened during an election. The ACE Project, a group that advises democracies on how to conduct elections that is spearheaded by, among other groups, the United Nations, says this of exit polling: "Their reliability can be questionable. One might think that there is no reason why voters in stable democracies should conceal or lie about how they have voted, especially because nobody is under any obligation to answer in an exit poll. But in practice they often do. The majority of exit polls carried out in European countries over the past years have been failures."
As the MIT political scientists Charles Stewart has pointed out, it's not useful to compare the role of exit polls in Ukraine's 2004 election with exit polls in the U.S race. The two elections, and the two nations, are too different to come to any meaningful conclusion from such a comparison. In Ukraine, one exit poll showed opposition candidate and eventual president Viktor Yushchenko winning 54 percent to 43 percent nationally. Mitofsky's final national poll put Kerry at 51 percent and Bush with 48 percent. Compare this to the actual result, which had Bush at 51 percent and Kerry with 48 percent. The difference is not that significant.
Moreover, Stewart notes, pre-election polls in Ukraine agreed with the exits, bolstering the case that Yushchenko was the true winner. In the United States, though, the polls taken before the election tended to show either a very close race or a Bush win. (You can read Stewart's paper in PDF format here.)
When you talk to pollsters about what to make of the 2004 American exit polls -- as I have done, on and off, for the past year and a half -- you don't hear the degree of trust in the surveys that Kennedy suggests. Exit polls are sometimes wrong; indeed, examples abound. In 1992, the exits showed almost as great a pro-Clinton bias as the 2004 poll's pro-Kerry bias -- in other words, the poll showed Clinton with a lot bigger win than he ultimately had. The reason that poll didn't cause a firestorm is because the race wasn't as close as the one in 2004.
Claim: The exit polls showed an insurmountable Kerry lead, one that made a Bush win impossible.
"As the last polling stations closed on the West Coast, exit polls showed Kerry ahead in ten of eleven battleground states -- including commanding leads in Ohio and Florida -- and winning by a million and a half votes nationally." Kennedy adds, "Based on exit polls, CNN had predicted Kerry defeating Bush in Ohio by a margin of 4.2 percentage points. Instead, election results showed Bush winning the state by 2.5 percent. Bush also tallied 6.5 percent more than the polls had predicted in Pennsylvania, and 4.9 percent more in Florida."
Kennedy then includes a blockbuster quote from Steven Freeman, a visiting scholar at the University of Pennsylvania, who puts the odds of the polls being as wrong as they were in Pennsylvania, Ohio and Florida at 1 in 660,000. ''As much as we can say in sound science that something is impossible,'' Freeman says, ''it is impossible that the discrepancies between predicted and actual vote count in the three critical battleground states of the 2004 election could have been due to chance or random error.''
Reality: Kennedy is right that the polls in battleground states showed Kerry ahead. What he fails to say is that in many states, the exits didn't show Kerry ahead by the margin of error, meaning, statistically, that his lead wasn't secure. Way back in December of 2004, pollster Mark Blumenthal pointed out the key fact in this debate. Of the ten battleground states that the exit poll showed Kerry winning, he ultimately lost four -- states that, you could say, cost him the election. These were Ohio, Iowa, Nevada and New Mexico. But in none of those states was Kerry's lead outside the poll's margin of error. In other words, the poll results showed a race that was too close to call, and it is impossible to use such a poll to prove that fraud occurred. As Mitofsky told me, television news networks, looking at the exit poll data, seemed to understand that Kerry did not top the margin of error, and so did not call these states for him.
As for Freeman's 660,000 to 1 statistic, it is irrelevant. (His comment to Kennedy -- "As much as we can say in sound science that something is impossible..." -- appears almost verbatim in the paper he put out in December 2004; I included it in a story on exit polling a year and a half ago.) The statistic measures the probability that the errors in Pennsylvania, Ohio and Florida occurred due to chance or random error, and according to Freeman, that probability is very low. But nobody argues the errors happened by chance. Everyone in the exit poll debate agrees that there was a systematic cause for the errors in the poll. Freeman, Kennedy, et al., claim that the systematic cause was fraud, while Mitofsky and many in the polling community claim the cause was a problem with the poll. So Freeman's argument that it would take preposterous odds to produce a random sampling error is a straw-man assertion.
Claim: The exit pollsters can't explain how their poll failed.
Kennedy says that Edison/Mitofsky, Warren Mitofsky's polling group, "was unable to identify any flaw in its methodology -- so the pollsters, in essence, invented one for the electorate."
Reality: This claim is misleading. In January 2005, Mitofsky released a 77-page report detailing how his poll performed on Election Day. You can read the PDF here. It is not stingy about possible methodological flaws in the survey: "Our detailed analysis by polling location and by interviewer has identified several factors that may have contributed to the size of the Within Precinct Error that led to the inaccuracies in the exit poll estimates. Some of these factors are within our control while others are not."
As I reported last year, Mitofsky has outlined a clear and convincing explanation for what went wrong with his survey. According to Mitofsky, interviewers assigned to talk to voters as they left the polls appeared to be slightly more inclined to seek out Kerry voters than Bush voters. Kerry voters were thus overrepresented in the poll by a small margin. According to Mitofsky's report, the polling error tended to be larger in precincts where interviewers had been recently hired or reported being insufficiently trained; where precinct officials, lawyers or other vote observers interfered with pollsters' opportunity to approach the voters as they left the precinct; where pollsters were made to stand far away from the precinct; and where the weather wasn't great (remember the rain in Ohio?). The report went on to outline various fixes in polling practices that might mitigate such flaws in the future.
Claim: Researchers have conclusively disproved the official explanation for the exit poll's error.
Kennedy says that Mitofsky's theory that Kerry voters were oversampled in the poll -- thus leading to a pro-Kerry poll bias -- doesn't hold water. "Now, thanks to careful examination of Mitofsky's own data by Freeman and a team of eight researchers, we can say conclusively that the theory is dead wrong. In fact it was Democrats, not Republicans, who were more disinclined to answer pollsters' questions on Election Day. In Bush strongholds, Freeman and the other researchers found that fifty-six percent of voters completed the exit survey -- compared to only fifty-three percent in Kerry strongholds. 'The data presented to support the claim not only fails to substantiate it,' observes Freeman, 'but actually contradicts it.'"
Reality: To begin with, Freeman and his team did not "find" the survey-completion rates that Kennedy cites. Mitofsky released that data in a public report. This data was not discovered "now" -- Freeman and others have been touting it ever since Mitofsky put it out in January 2005. You can see the data on page 37 of Mitofsky's report. There, Mitofsky indeed shows that in precincts where Bush got 80 percent or more of the vote, an average of 56 percent of people who were approached volunteered to take part in the poll, while in precincts where Kerry got 80 percent or more of the vote, a lower average of 53 percent of people were willing to be surveyed. But these numbers don't reveal how Bush voters or Kerry voters behaved, they only show how all voters, taken together in average, responded in certain precincts. They are irrelevant to the question of whether fraud occurred.
As Mark Lindeman, a political scientist at Bard College, explained to me, the numbers Kennedy cites fit the theory that Kerry voters were more likely to respond to pollsters than Bush voters. For instance, in the Bush strongholds -- where the average completion rate was 56 percent -- it's possible that only 53 percent of those who voted for Bush were willing to be polled, while people who voted for Kerry participated at a higher 59 percent rate. Meanwhile, in the Kerry strongholds, where Mitofsky found a 53 percent average completion rate, it's possible that Bush voters participated 50 percent of the time, while Kerry voters were willing to be interviewed 56 percent of the time. In this scenario, the averages work out to the same ones Kennedy cited: a 56 percent average response rate in Bush strongholds, and a 53 percent average response rate in Kerry strongholds. But in both Bush strongholds and Kerry strongholds, Kerry voters would have been responding at a higher rate, skewing the poll toward Kerry.
What's more, these numbers are not set in stone. That's because, as Mitofsky has pointed out, it's not possible to measure the actual completion rate by Kerry voters and by Bush voters. (When someone refuses to talk to a pollster, it's not possible to say whether he was a Bush voter or Kerry voter.) Mitofsky says that a hypothetical completion rate of 50 percent for Bush voters and 56 percent for Kerry voters would have led to the error we saw in the poll. In other words, Kerry voters were very slightly more likely to talk to pollsters than were Bush voters.
Ultimately, nothing in Kennedy's article, and nothing in the research he cites, refutes Mitofsky's theory that there was a true difference in the willingness of Kerry voters to participate in the poll compared to that of Bush voters. Mitofsky noted a broad array of methodological errors that could have contributed to this difference in participation rate by Kerry and Bush voters. Such a difference would not have been a surprise; Democrats have historically been overrepresented in exit polls. There is no reason to think that the error in 2004 was anything substantively different.
Claim: Tens of thousands of people were disenfranchised due to voter registration errors.
Kennedy points to an analysis conducted by the nonpartisan Greater Cleveland Voter Coalition. He says it showed that "16,000 voters in and around the city were disenfranchised because of data-entry errors by election officials, and another 15,000 lost the right to vote due to largely inconsequential omissions on their registration cards." He adds the study concludes that statewide, "a total of 72,000 voters were disenfranchised through avoidable registration errors -- one percent of all voters in an election decided by barely two percent."
Reality: Kennedy has misread the Greater Cleveland Voter Coalition report in a small but important way. The report examines the numbers of people whose registrations were bungled due to their own or their county officials' error. Some of those errors -- for instance, submitting a registration form without an address -- disqualified people to vote. Other errors, such as someone's making a mistake while typing in your name, might or might not have disqualified you. So not all of the 16,000 people in Cleveland whose registrations included data-entry errors were disenfranchised. In fact, many of them got to vote. (You can read the coalition's PDF report here.)
Kennedy's error is important when you consider the number of people disenfranchised through registration errors statewide, which he puts at 72,000. In fact, the coalition reports the number as an estimate of about 42,500 votes that were "lost," and 30,000 votes it says were "at risk" of being lost; it is not clear how many of those that were "at risk" were actually lost.
The report simply does not say that 72,000 people were disenfranchised.
About the writer
Farhad Manjoo is a Salon staff writer.
Related Stories
Was the election stolen?
The system is clearly broken. But there is no evidence that Bush won because of voter fraud.
11/10/04
Don't get "Fooled Again"
In his new book, Mark Crispin Miller tries to prove that Republicans rigged the 2004 election, but his evidence is thinner than a butterfly ballot.
11/14/05
Story finder (3 ways to search Salon)
Salon Directory (browse by topic)