(Sep 20, 2008)

There are lies, damned lies and statistics, according to the line made popular by Mark Twain, and a cynical person might be excused for thinking that the best evidence of that can be found in the middle of an election campaign.

We'll let the politicians argue over the lies and the damned lies for now and focus on statistics, because they don't lie.

Or do they?

Election polls have joined death and taxes as the only sure things in life. Each day brings a new blizzard of poll results as the country moves closer to the Oct. 14 election, with the various parties inching up and down a percentage point or two.

But do we really understand the science behind polls and polling?

It's a job made tougher at a time when pollsters must take into account factors such as the high rate of unresponsive or undecided voters -- even the huge explosion in cellphone users.

"A poll is a snapshot of what's happening at the time if it's well designed, but things can change very quickly," said Nancy Reid, an award-winning professor of statistics at the University of Toronto. "Any type of political issue can have an effect and render yesterday's poll obsolete."

So, rule No. 1: polls have a short shelf life.

Rule No. 2: Polls report a range of possibilities, not just one static number.

What often gets glossed over by the media is the fact that polls are supplying a deceptively wide range of potential support because polls have a built-in margin of error.

In a poll of 1,000 people there's a margin of error of plus or minus 3.1 percentage points.

But that still gives a possible range of 6.2 percentage points. If that same poll, for example, puts one party's support at 38 per cent, what it really means is that the number could fall anywhere from 34.9 per cent to 41.1 per cent.

"If you want to half the margin of error you have to quadruple the sample size," said Reid, noting that the cost of sampling people has to be balanced against the desire to sharpen a poll's accuracy.

"It's not really worth it."

On top of that, statistical methodology builds confidence about a poll's accuracy.

Polls are considered accurate 19 times out of 20. In other words, one out of 20 polls could be expected to give results that would fall outside the margin of error.

But in this age of nonstop polling, that one in 20 anomaly would stand out in the crowd.

If problems are going to show up with a poll, it's more likely to be a factor of which people were asked the questions, not how many.

Rule No. 3: The success of any poll depends on the quality of the sample being surveyed.

For a poll's results to be reliable, the people selected to respond have to accurately match the characteristics of the entire population.

Surprisingly, a sample size of as few as 1,000 people can reasonably reflect the preferences of Canada's 33 million citizens.

What's also surprising is that pollsters don't set out to build a target sample group to match the country's demographics. They rely on random-digit telephone dialing to capture the right mix.

"They trust their randomness to catch it for them," said Reid. "It sounds paradoxical - and people don't like to accept it - but it's actually safer than trying to set out to get an absolutely representative group."

But if the sample is somehow biased, all bets are off.

The best example of an opinion survey that was doublecrossed by sampling bias occurred during the 1936 U.S. presidential election between Franklin Roosevelt and Alf Landon, when the science of election polling was still in its infancy.

Three independent scientific polls at the time predicted that Roosevelt would defeat Landon, the Republican candidate. But an influential magazine, called the Literary Digest, predicted Roosevelt would earn less than 41 per cent of the popular vote and that Landon would win in a landslide.

The Literary Digest seemed to have an overwhelming mountain of evidence on its side. The magazine had mailed out about 10 million questionnaires and received more than 2.3 million responses.

But the ballots had been sent to people whose names were taken from lists of those who owned a telephone or a car. In the 1930s - during the middle of the Depression - car and telephone owners were affluent people who were more likely to vote Republican.

Roosevelt won more than 62 per cent of the popular vote in one of the largest landslides in U.S. election history. Two years later, the Literary Digest folded.

There's also the issue of less-than-truthful answers.

Even though election polls are conducted anonymously, people don't always give an honest answer, for a variety of reasons.

That was the case during the 1992 British election, where polls - including exit polls taken after people had already voted - predicted a victory for the Labour party.

Yet John Major's Conservative party won a majority government, confounding everyone.

It turns out that the Conservative party's policies had become so unpopular that people were either embarrassed or afraid to admit that they'd be voting Conservative.

A similar problem - this time over the issue of race - may show up in this year's U.S. election.

"They worry a lot about that with (Barack) Obama, that people won't admit to a pollster that they wouldn't vote for him (because of race)," said Reid.

A savvy pollster these days will even take into account how long it takes a person to respond to a question about voting preference.

John Bassili, a psychologist at the University of Toronto's Scarborough campus, has conducted research looking at the relationship between voting behaviour and response times to a polling question.

Bassili found that for every one second delay in stating who a person would vote for, the probability that person would eventually change their vote rose by 8 per cent.

In other words, if there was a five-second delay in naming a preference, the chance that person would actually stick with that choice dropped by 40 per cent.

A poll could even be biased by those people who refuse to participate in the poll in the first place.

All polls have some people who won't participate, and if that group doesn't have the same characteristics as those who do respond, there could be a sampling bias.

That's a growing problem for polling companies in an age where people use call display to shield themselves from telemarketers.

Another emerging problem is the growing number of people who only use a cellphone.

Those people, generally younger and more urban, would be excluded from sampling, which could also skew results.

sbuist@thespec.com

905-526-3226