As someone who spends their days working with samples and statistical methods to make inferences about a larger population (think A/B testing), I've been especially curious about why election polling seems to have missed the mark lately—particularly in the last few election cycles. It feels like every time, we’re caught off guard by the final results versus what the polls predicted.
Take the popular vote: recent forecasts based on polls had Harris at 51.1% and Trump at 48.9% in popular votes (according to FiveThirtyEight). But in the final result, the trend flipped, Harris came in at 48% and Trump at 50.5%. That’s a 3-point miss on each candidate, making for a total 6-point swing in the difference.
What causes this difference?
FiveThirtyEight forecast based on polling results
Sample Size, Margin of Errors, and Other Errors
When you're polling only a sample of the entire population, a margin of error is always expected.
In fact, Pew Research points out that for a single candidate's support, the margin of error is typically around 3 percentage points. And when comparing two candidates? That error margin doubles.
A big driver behind this sampling error is, of course, the sample size itself. And here’s the part that might surprise you: for a state poll, only around 500-1000 samples are gathered. For a national poll, that number jumps to about 1000-2000.
To put it in perspective, most A/B tests we run often require samples 100x larger than what’s collected in political polling.
But according to major research firms, there's a reason they don’t push for a bigger sample size.
Apparently, the margin of error stabilizes past a certain point, meaning that the cost of adding more participants simply doesn’t justify the minimal reduction in error margin.
Note the reported margin error does not include other errors incurred in the polling process, especially given the various outreach methods used—whether by phone calls, text messages, online surveys, or mail.
- Noncoverage Error: Certain groups may never even have a chance to be included in the survey. For example, if a poll only surveys landline users, younger people who primarily use cell phones would be excluded, creating a gap in representation.
- Nonresponse Error: This occurs when individuals who are part of the sample don’t respond to the survey. Nonresponse can lead to bias if those who don’t respond hold different views than those who do. For instance, if older voters or certain demographic groups respond at lower rates, this can skew results. It’s been reported that Trump supporters are harder to reach in polling efforts.
- Measurement Error: Respondents may not always be completely candid, especially on sensitive topics. someone might give answers that cast them in a favorable light, like claiming to be registered to vote when they aren’t.
- Measurment Error: Different polling firms may use varied question wording or survey techniques, leading to differences in results and potential errors.
Samples are adjusted to mimic the entire population
Since polling samples are relatively small, they’re adjusted—or "weighted"—to better represent the broader population.
This weighting process aligns the sample with real-world demographics like location, age, gender, income level, and education.
In 2016, polling was off for Hilary Clinton vs. Trump partly due to flaws in this process: not enough weight was given to people without college degrees, a group that leaned more heavily toward Trump, leading to a skewed picture of the electorate.
Polling results are just a snapshot in time
Polls capture public opinion at a specific moment, reflecting how people feel based on current events and available information.
As new news and developments unfold, public opinion can shift quickly, meaning poll results can be changed overnight, and tend to be more accurate the closer they are to Election Day, so they’re not always a prediction of final outcomes.
Why do election polls if the predictions are not accurate?
While election polls may not always predict final outcomes with precision, they serve several vital functions in the political landscape:
- Issue Prioritization: Polls highlight what matters most to voters—such as healthcare, the economy, or education—helping campaigns focus their messaging on topics that resonate.
- Tracking Public Sentiment Trend: Regular polling captures shifts in voter attitudes over time, allowing campaigns and policymakers to respond to emerging concerns and changing perspectives.
- Engaging the Public: Polls spotlight important issues, encouraging voter awareness and participation, ultimately fostering a more engaged and informed electorate.
- Policy Insights: Beyond elections, issue-based polls provide policymakers with a sense of public opinion, guiding decisions and helping align policies with the electorate's preferences.
