# Study: Probability Surveys More Accurate than Online Panels

September 29, 2009

Though the research industry has long raised doubts about the credibility of non-probability-based surveys conducted online, a new study from Stanford University researchers and others provides even more evidence that today’s increasingly popular opt-in internet surveys are not as accurate as thosse conducted with probability sampling.

The academic study, entitled “Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples,”? (pdf) commissioned nine firms to administer the same survey questionnaire in 2004/2005 via (1) RDD telephone interviewing with a probability sample of American adults, (2) internet data collection from a probability sample of American adults, and (3) internet data collection from samples of American adults who volunteered to do surveys for money or prizes and were not randomly sampled from the American adult population (“opt-in” samples).

Key findings from the study:

• The probability sample surveys done by telephone or the internet were consistently highly accurate, while the opt-in sample surveys done via the internet were always less accurate and were sometimes strikingly inaccurate.
• The average errors for the two probability samples were 3.5% and 3.3%, whereas the average errors for the opt-in surveys ranged from 4.9% to 10.0% and averaged 6.0%.

• Best- practices weighting of the opt-in samples sometimes improved their accuracy and sometimes reduced their accuracy but never made them as accurate as the RDD telephone and probability sample internet surveys. The largest unweighted error for a single item for the probability samples was 12 percentage points, whereas the largest such error for the opt-ins was 35.5 points. The comparable weighted numbers are 9% and 18%, respectively.
• The? standard deviation of the errors is nine times larger for the opt-in sample surveys than for the RDD telephone survey.

Though critics of the study have raised questions about the relevance of older study data and assert that online sampling methods have improved since the study was conducted,? two of the study’s authors, David Yeager and Jon Krosnick, assert that survey accuracy remains largely the same as it was five years ago. “We see no evidence that the accuracy of probability sample surveys by these firms declined, or that the accuracy of this opt-in survey firm’s data improved since 2004,” they said in a blog post.

Despite the fact that the research found probability sample surveys to be more consistently accurate,? the authors did concede that opt-in surveys occasionally produce data points that are close to accurate, though they usually don’t.

Nonetheless, Yeager and Krosnick still do see “tremendous value in opt-in survey data”? from a social-science perspective.

About the study:? One of the internet probability samples used in the study was provided by KnowledgeNetworks, which is making this study available on the web.

Explore More Articles.

## CMOs to Increasingly Prioritize CX and Innovation in the Coming Year

The primary roles of the marketing function over the next year will be ensuring effective brand management and developing the overall customer experience.

Over the past couple of years, podcast ad revenues have shifted from direct response to brand-building.

## How Are Shoppers Getting to Product Detail Pages?

More than one-quarter report often arriving at product detail pages from social media.

## Mobile Banking Apps Grow in Popularity

Only 18% of Americans say they don’t use smartphone banking, down from 28% in 2019.

## Which Video Types Are Proving Effective for Content Marketers?

Social media videos and case studies/customer stories lead the way in perceived effectiveness.

## Baby Boomers Are Frequent Online Shoppers, Present Lucrative Market

Almost one-quarter of Baby Boomers worldwide report having high purchase power, triple the rate of Gen Zers.

## The US’ Pay-TV Penetration Rate Continues to Fall

Some 64% of TV households subscribe to pay-TV, down from 78% in 2018.

## Where DTC Brands See Opportunities in 2024

DTC brands are looking to improve on testing, but may face troubles with budgets and buy-in.

## How Are CMOs Chasing Consumers’ Attention?

Investments in live events appear to be growing, while enthusiasm around virtual events is waning.

## About Half of Marketing Operations Teams Work Closely with the C-Suite

The higher the company’s revenue, the more likely that the Marketing Ops team works in close concert with the IT team.

## B2B Demand Gen Budgets to Continue Growth amid Focus on Pipeline and Revenue

The most common way to define demand gen campaign success is through new pipeline opportunities/revenue sourced.

## The Vast Majority of CMOs Expect Their MarTech Spend to Grow

7 in 10 CMOs say they can accurately measure the effectiveness of their martech investments.

## Gen Z Consumers Plan to Stick With Their Inflation-Driven Spending Changes

Some 96% of those who have changed their spending habits plan to maintain those changes over the next year.

## In the US, Premium Video Ads Are Mostly Being Seen on Connected TV

H1 saw an 80% year-over-year increase in the US of premium video ad impressions using audience targeting.

## B2B Demand Gen Marketers to Increasingly Prioritize High-Quality Lead Programs

Lead nurturing is also set to be prioritized more in the coming year, while content syndication falls out of favor.

## CMOs’ Budget Outlook Appears to Have Rebounded from Earlier This Year

CMOs are more likely to be reporting revenue increases and to be expecting budget growth.

## The Availability of New Content Isn’t That Important to Free TV Streaming Service Users

Most agree that being able to watch for free makes up for the lack of newer content.

## B2B Demand Gen: Smaller Companies Catch Up to BDR Best Practices

B2B demand gen teams that met or exceeded their KPIs are more likely to have a lead qualifying team.

## Brands’ Portrayal of Women in Advertising Affects Their Purchase Decisions

Adult women feel that older women ages 50+ are presented authentically in media and advertising less than a third of the time.