Can Nonprobability Samples be Used for Social Science Research? A cautionary tale

  • Elizabeth S. Zack Indiana University
  • John Kennedy Indiana University
  • J. Scott Long Indiana University
Keywords: Nonprobability Samples, Online Panels, MTurk, GSS

Abstract

Survey researchers and social scientists are trying to understand the appropriate use of nonprobability samples as substitutes for probability samples in social science research. While cognizant of the challenges presented by nonprobability samples, scholars increasingly rely on these samples due to their low cost and speed of data collection. This paper contributes to the growing literature on the appropriate use of nonprobability samples by comparing two online non-probability samples, Amazon’s Mechanical Turk (MTurk) and a Qualtrics Panel, with a gold standard nationally representative probability sample, the GSS. Most research in this area focuses on determining the best techniques to improve point estimates from nonprobability samples, often using gold standard surveys or census data to determine the accuracy of the point estimates. This paper differs from that line of research in that we examine how probability and nonprobability samples differ when used in multivariate analysis, the research technique used by many social scientists. Additionally, we examine whether restricting each sample to a population well-represented in MTurk (Americans age 45 and under) improves MTurk’s estimates. We find that, while Qualtrics and MTurk differ somewhat from the GSS, Qualtrics outperforms MTurk in both univariate and multivariate analysis. Further, restricting the samples substantially improves MTurk’s estimates, almost closing the gap with Qualtrics. With both Qualtrics and MTurk, we find a risk of false positives. Our findings suggest that these online nonprobability samples may sometimes be ‘fit for purpose,’ but should be used with caution.

Author Biographies

Elizabeth S. Zack, Indiana University
Ph.D. Candidate Department of Sociology
John Kennedy, Indiana University
Senior Research Director Center for Survey Research
J. Scott Long, Indiana University
Distinguished Professor and Chancellor's Professor of Sociology and Statistics
Published
2019-06-19
How to Cite
Zack, E. S., Kennedy, J., & Long, J. S. (2019). Can Nonprobability Samples be Used for Social Science Research? A cautionary tale. Survey Research Methods, 13(2), 215-227. https://doi.org/10.18148/srm/2019.v13i2.7262
Section
Articles