By Michael Starbird, PhD, The University of Texas at Austin
What’s the one essential ingredient for a trustworthy poll? While there are plenty of pitfalls that can taint results, the key for validity rests in randomness.
Start With A Sample
When used in statistics, the word “population” refers to the entirety of the collection of people or things that are of interest. A sample is a subset of the total population.
In general, the goal is to infer information about the whole population from information about the sample. It’s not in our interest to know only about the people who are asked in the sample. What we’re interested in is those aspects of the entire population.
This is a transcript from the video series Meaning from Data: Statistics Made Clear. Watch it now, on Wondrium.
Randomness Is The Key To Good Sampling
If you choose the sample randomly, using probability gives you the advantage to make inferences about how well the sample represents the opinions of the whole population.
On the other hand, if you intentionally choose certain groups to reflect what you believe to be reflective of reality, you may bring your own biases to the selection process, and those biases are then going to be reflected in the people whom you ask. A representative of the whole population means that the sample should have the same characteristics that the whole population does.
The whole concept of choosing the sample randomly is that you have a better chance that the proportion of people in the sample with a certain opinion will be, in fact, the same as the entire population.
Learn more about induction within polling and scientific reasoning
Roosevelt Versus Landon
The most familiar occasion where this comes up is before an election when pollsters try to find out what proportion of the voters will vote for the Democratic candidate and what proportion will vote for the Republican candidate.
There are several major pitfalls in the way sampling can be done. In the 1936 US presidential election, the two primary contenders for the presidency were the incumbent, Franklin Delano Roosevelt, and the Republican opponent, Alfred Landon. At the time, the magazine The Literary Digest had for several elections conducted polls to predict who would win the coming election. They had successfully predicted the outcomes in several elections, so this was a major poll.
In the 1936 election, The Literary Digest sent out 10 million voting surveys, and they received 2.4 million replies. Based on those surveys, The Literary Digest predicted that Landon would win in a landslide, with 370 electoral votes to Roosevelt’s 161.
You may not recall reading about President Landon in your American history books, as he did not win the presidency.
The only correct aspect of The Literary Digest’s prediction was that the election was a landslide, but unfortunately for them, the landslide was the other way. Roosevelt won the election with 62% of the popular vote and by an incredible 523 electoral votes to 8 for Landon.
Learn more about what makes aggregation more effective than any single poll
The Literary Digest’s sampling method was not representative of the whole population.
Pitfall 1: A Wealthy Audience
What went wrong? The Literary Digest got their samples from several different kinds of lists, including subscribers to their own magazine. They also looked at car registration records—an available list of many names—and they sent their surveys to those people. Their sampling method also including using telephones.
The people to whom The Literary Digest had sent their survey were likely wealthy people and their opinions were not representative of the population at large.
The people to whom The Literary Digest had sent their survey were likely wealthy people and their opinions were not representative of the population at large.
The year 1936 was in the middle of the Great Depression, and many people were having financial problems and were cutting back on their budgets. Probably one of the first things to go in tight times would be one’s subscription to The Literary Digest. Also, not many people owned cars or telephones. These were luxury items for many people in 1936. Because of this, the people to whom The Literary Digest had sent their survey were likely wealthy people and their opinions were not representative of the population at large.
Learn more about gathering data from which deductions can be drawn confidently
Pitfall 2: A Voluntary Response
The Literary Digest poll’s second pitfall was that it was a voluntary response survey.
The magazine sent out all these surveys and only some people replied. The problem with this is that sometimes people who send back replies have a particular bias. Instead of sending back replies in the same proportion, maybe some people with a certain opinion are more apt to reply. The bias that can come from voluntary responses may not just give an answer that’s a little off, but it can give a completely erroneous view of reality.
Because of this story, The Literary Digest, which otherwise would simply be lost in the dustbin of history, will now live on forever in statistics textbooks as a great example of bias in sampling.
The Father of Random Polling
A success that came from this Literary Digest fiasco is the story of George Gallup.
At the time, Gallup was a young statistician just starting out, and he did his own poll for the 1936 election. He took a survey of 50,000 people and made two predictions of his own for the election.
- He correctly predicted that Roosevelt would win the election.
- He also predicted that The Literary Digest poll would be wrong and estimated how wrong they would be before their poll came out.
He was one of the people who introduced the concept of randomness in political polling as a key feature of sampling techniques. It is one of the fundamental criteria to look for when you’re evaluating whether a sample survey is, in fact, a good one.
Learn more about sampling; a technique for inferring features of a whole population from information about some of its members
The Gold Standard
Randomness is a basic ingredient of essentially all of the standard statistical techniques. The reason it’s an ingredient is that the analysis of randomness and probability allows us to apply mathematics to the understanding of the results that we get.
The most basic way to get an accurate sample is to take a sample that’s called a simple random sample. As the name implies, simply take the entire population you’re interested in, say how many people you want to survey, randomly select them from that group, and then get the answer from each member of that selected sample.
Of course, there are lots of problems in getting the answer from that selected sample. But the simple random sample is the gold standard for finding a representative sample.
Common Questions About Polling Results and Statistics
Political polling is used both to predict a campaign’s results and to give a candidate or supporters of that candidate a metric by which to measure the candidate’s results. The first poll, called a benchmark, helps a candidate to design a campaign strategy by identifying such factors as the candidate’s overall popularity, the demographics of the people most likely to vote for that candidate, and the issues that matter most to the candidate’s main audience.
An exit poll is conducted after an election. As voters leave the polling station, reporters ask them who they voted for. This is used to predict election results, since the votes can sometimes take a few days to count.
Polling data is used by a political candidate to gather information about how well he/she is resonating with potential voters at the start of and during a campaign. It provides the candidate with information such as the demographical features of the individuals who would most likely vote for the candidate and allow the candidate to test the popularity of various messages. His/her overall approval rating is also a good indicator of whether or not it is worth staying in the campaign because running a political campaign is very expensive.
Push polls are intended to “push” an issue to the forefront of the voter’s mind. For example, a push poll might ask potential voters to evaluate candidates based on their support of healthcare.
