Polling Pitfalls

Polling Pitfalls

Are Stephen HARPER's CONSERVATIVES surging toward a majority win on Oct. 14, or simply holding their own? Have Stéphane DION's LIBERALS lost the trust of voters in key "battleground" ridings, or are they making unexpected inroads in vote-rich Quebec? In the early days of the federal election campaign, the answer depends on which opinion survey and media outlet you choose to believe. The owners of the crystal balls blame the divergent results on a volatile electorate, but a new book says the problems with polling run much deeper: skewed samples, inaccurate methods, and a largely apathetic and ignorant public.

In The Opinion Makers (Beacon Press), David Moore, former senior editor of America's Gallup Poll, charges that polling firms and news organizations are undermining democracy by "systematically misleading" voters about the accuracy and insights of their ever-multiplying surveys. "Media polling gives us distorted readings of the electoral climate, [and] manufactures a false public consensus on policy issues," he writes. Voter indecision is played down or simply not reported, while many respondents are cajoled or coached into giving top-of-mind answers. The result, says Moore, is "manufactured opinion based on a mythological public" - an idealized vision of what "a rational, informed and engaged citizenry might be thinking," and a fiction that politicians seize upon to buttress or justify their own policy positions.

The fallibility of opinion surveys has been a mostly ignored fact since Harry Truman decisively beat Thomas Dewey in the 1948 U.S. presidential election, confounding the predictions of the only three players then polling. But Moore contends such mistakes are now endemic - citing from recent American history Hillary Clinton's "upset" win in the New Hampshire primary, Rudy Giuliani's abrupt transformation from Republican "front-runner" to no-hoper, and the support for the invasion of Iraq that was neither as widespread nor as profound as pollsters had claimed. The samples, says Moore, are now drawn from an ever-shrinking pool of people who have land-line home phones and are willing to answer the questions (pollsters are reluctant to discuss it, but as many as eight out of 10 people they try to contact either screen out the call or hang up). Internet panels are even less representative, tilting toward young, urban and conservative men - Zogby International recently tried to balance out its Net surveys by striking a deal with the Marijuana Policy Project, gaining access to the pro-toking lobby group's 12,000 supporters.

The way questions are framed is an even bigger obstacle, with respondents routinely pressured into providing answers on subjects they neither know nor care about. (Do 73 per cent of Americans really believe their country should always comply with unfavourable World Trade Organization rulings, as per a 2007 WorldPublicOpinion.org survey?) "I don't think we should require or expect someone to pay close attention to all the candidates and issues, especially months before an election," Moore told Maclean's in an interview. "But clearly the pollsters and the news media are less interested in finding out the truth about the public than coming up with something to put in their stories."

While Moore doesn't specifically address Canadian polling, many of the practices he questions are also employed in this country. For example, most Canadian pollsters now follow their U.S. counterparts and no longer report on undecided voters in their "horse race" polling. "We don't think undecideds are all that important, because, frankly, they don't vote," says Tim Woolstencroft, managing partner of the Strategic Counsel, which conducts surveys for the Globe and Mail and CTV. (In their recent survey, which placed the Tories on "the brink" of a majority - 37 per cent support, versus 27 for the Liberals, and 17 for the NDP - Woolstencroft says 17 per cent of the 1,000 respondents still had no party preference. That was even after they were prodded to say which way they were "leaning.") Environics, which works with the CBC, similarly didn't report its undecideds - 13 per cent, according to senior vice-president Donna Dasko, in a recent survey that had the Tories at 38 per cent, followed by the Liberals at 28 and the NDP at 19. " Everyone focuses on the decided voters. That's how it has evolved over the last three decades," she says.

Woolstencroft, who is familiar with Moore's criticisms, says he believes there are some mitigating factors in Canada. Response rates, although deteriorating, are not as bad here, and our citizens tend to be more urban, and urbane. With parties now so reliant on polling, Woolstencroft argues media surveys have become a democratic necessity. "The question is, should the media give this information to the electorate, or should only the political parties have access to it?" he asks.

Canadian pollsters, however, do have much to prove in the current federal campaign, given their less than stellar predictions in the last two go-arounds. Mark Pickup, a political scientist at Simon Fraser University, tracks bias and error in all the major polls on his website (http://pollob.politics.ox.ac.uk/). In the 2004 and 2006 general elections, he says, almost all the major firms systematically underestimated Liberal support, and overestimated the NDP vote. (The exceptions were SES, Leger and Pollara.) By comparing the results of overlapping surveys, Pickup and Richard Johnston of the University of Pennsylvania created a complicated mathematical formula to correct for these shortcomings in sampling, the wording and ordering of questions, and other more random errors. And if you trust their science more than the industry's, the 2008 campaign polls are already off to a shaky start. In the first week, Pickup places the Tories at 35.1 per cent and the Liberals at 32.5 per cent - a virtual tie given the margin of error. "These aren't deliberate errors. Pollsters want to get it right," says Pickup. "But the fact that papers are already saying the Tories are on the brink of a majority suggests people haven't learned their lesson." Come Oct. 14, we'll know for certain.

Maclean's September 22, 2008