Though the research industry has long raised doubts about the credibility of non-probability-based surveys conducted online, a new study from Stanford University researchers and others provides even more evidence that today’s increasingly popular opt-in internet surveys are not as accurate as thosse conducted with probability sampling.
The academic study, entitled “Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples,”? (pdf) commissioned nine firms to administer the same survey questionnaire in 2004/2005 via (1) RDD telephone interviewing with a probability sample of American adults, (2) internet data collection from a probability sample of American adults, and (3) internet data collection from samples of American adults who volunteered to do surveys for money or prizes and were not randomly sampled from the American adult population (“opt-in” samples).
Key findings from the study:
- The probability sample surveys done by telephone or the internet were consistently highly accurate, while the opt-in sample surveys done via the internet were always less accurate and were sometimes strikingly inaccurate.
- The average errors for the two probability samples were 3.5% and 3.3%, whereas the average errors for the opt-in surveys ranged from 4.9% to 10.0% and averaged 6.0%.
- Best- practices weighting of the opt-in samples sometimes improved their accuracy and sometimes reduced their accuracy but never made them as accurate as the RDD telephone and probability sample internet surveys. The largest unweighted error for a single item for the probability samples was 12 percentage points, whereas the largest such error for the opt-ins was 35.5 points. The comparable weighted numbers are 9% and 18%, respectively.
- The? standard deviation of the errors is nine times larger for the opt-in sample surveys than for the RDD telephone survey.
Though critics of the study have raised questions about the relevance of older study data and assert that online sampling methods have improved since the study was conducted,? two of the study’s authors, David Yeager and Jon Krosnick, assert that survey accuracy remains largely the same as it was five years ago. “We see no evidence that the accuracy of probability sample surveys by these firms declined, or that the accuracy of this opt-in survey firm’s data improved since 2004,” they said in a blog post.
Despite the fact that the research found probability sample surveys to be more consistently accurate,? the authors did concede that opt-in surveys occasionally produce data points that are close to accurate, though they usually don’t.
Nonetheless, Yeager and Krosnick still do see “tremendous value in opt-in survey data”? from a social-science perspective.
About the study:? One of the internet probability samples used in the study was provided by KnowledgeNetworks, which is making this study available on the web.
