So, I open up my web browser this weekend to check the news, and I see the following three polls, all on the same page:

• Rasmussen: Obama up by 5 points
• Gallup: Obama up by 10 point
• Zogby: McCain up by 1 point

These can’t all be right, can they?

Actually, they can. Or, at least, they can all be properly conducted, and just lead to wildly different results.

The only way to get a perfect result is to interview everyone in the country. (In fact, that’s exactly what we do on Election Day.) But that takes so much time and money that no individual pollster can do it. Instead, they interview several hundred people, maybe a couple thousand, and from there extrapolate what the country as a whole will do.

Now, mathematically, you can do this. You just can’t be sure of your answer. Here are a few of the reasons why.

Margin of error

Most opinion polls will state the margin of error. For example, they may say that that Candidate X is ahead by, say, 5 points, with a margin of error of plus-or-minus 3 points. Meaning, the real answer could be as high as 8 points or as low as 2 points.

(Sometimes, the margin of error is actually larger than the result. The poll shows Candidate X leading by 2 points, but with a margin of error of 4 points. Meaning, he could be ahead by 6, or he could actually be behind by 2! This seems to have happened a lot this year.)

A range of a few percentage points, when applied to a country with over 100 million voters, can lead to some pretty huge differences.

Confidence interval

In addition to reporting a margin of error, polls also report a confidence interval, usually 90% or 95%. This means that, according to the laws of mathematics, there is a 95% probability that the real result is the same as the poll result, within the margin of error.

But what about the other 5% or 10% of the time? Well, the folks reporting the numbers don’t like to tell you this, but, mathematically speaking, the poll can do everything right, and still be completely wrong, as much as 10% of the time.

There have been over 700 polls released this election season, and over 200 just in October. No doubt, many of the polls you have heard about fall into this category.

Weighting

In most elections, more women vote than men. If you conduct a survey and talk to 100 men and 100 women, you are going to have to give the women’s answers more weight to accurately reflect the Election Day results.

How much more weight? That depends. Do you think this election will be pretty much the same as previous years? Is there something happening this year that will make a lot more women come out to vote? Or, perhaps, something that will attract a lot more men?

The fact is, nobody knows. Weighting is just educated guesswork. And this year, it is more complicated than usual:

• Black voters are expected to come out in record numbers to support Barack Obama. How many will actually vote? Nobody knows.
• Young people generally do not vote as much as other groups. But many analysts expect more young people to vote this year. How many more? Nobody knows.
• Democrats, having lost the last two elections, are likely to turn out in larger numbers. How much larger? Nobody knows.
• New voters. There has been a great push to register new voters, many of them poor or minority members. These groups tend to vote Democratic. But, the NY Times reports that as many as 60% of those registrations may be fake. If you are a poll taker, you really have no idea how many new voters there actually are.
• Likely voters. Every pollster ends up talking to some people who will not vote on Election Day. Different polls use different methods of figuring out who is likely to actually vote—based on whether they voted last time, how much interest they have in the election, or just taking the voter’s word for it.

The different weighting factors used by the different polls probably accounts for most of the variability we see in the results.

Human factors

Let’s face it – humans are complicated and sometimes uncooperative beings. There are lots of ways they can foul up a perfectly good poll.

• Lying. People have a tendency to tell a pollster what they think he wants to hear. Maybe they just want to be nice; maybe they want to avoid an argument. This skewing has been found in many, many types of polls.
• Refusal. In any poll, a certain number of people refuse to participate—they don’t want to be bothered, or they don’t want to talk politics with a stranger. If refusals are more likely to come from one party than the other, this can skew results.
• Hard-to-reach folks. For years, pollsters called people on the telephone. But today more and more people have cell phones, or call screening, and are hard to reach. Again, if people with such gadgets are more likely to support one candidate or the other, it will skew the results. (One blogger has noticed that John McCain does better in polls conducted during the week than on the weekend, and speculates that's when McCain supporters are home.)
• Bias. Poll takers are only human. They have their own thoughts and opinions. And while they take great pains to be neutral, those opinions sometimes come through in the questions asked or the way the answers are interpreted. Most of the major polling companies are based in big cities like New York, Los Angeles, Boston, or Washington, DC – all cities that are heavily Democratic. This may explain why, in the last two elections, the Democrats did better in the polls than they did on Election Day.

So, with all these problems, how can we figure out who is going to win the election? Well, never fear – there is one sure-fire way to find out the winner:

Read the newspaper Wednesday morning.

And don’t forget to vote!

All tags:

Tags : math polls voting elections statistics human behavior probability opinion polls election

How would you tag this story?

My tags:

• Email