Comparing Probability-Based Surveys and Nonprobability Online Panel Surveys in Australia: A Total Survey Error Perspective

Abstract

In this paper we report the findings from our study which was undertaken to learn if the findings of Chang et al. (2009), Yeager et al. (2011), Erens et al. (2014), MacInnis et al. (2018) and Cornesse et al. (2020) would be replicated in Australia. Our Australian Online Panels Benchmarking Study (AOPBS) involved administering the same questionnaire across eight independent national Australian samples aiming to achieve approximately 600 completed questionnaires/interviews from each sample. The questionnaire was administered by the Social Research Centre (SRC), a subsidiary of the Australian National University (ANU), to three probability samples, and to five nonprobability samples drawn from the online panels operated by five independent nonprobability online panel providers. A dual-frame telephone sampling methodology was used for two of the probability surveys and the third used an address-based sampling (ABS) frame. The target population for the AOPBS was persons aged 18 years and older living in Australia who were fluent in English. The three probability sample surveys likely had little Coverage Error, a known amount of Sampling Error, a nonignorable amount of Nonresponse Error, little Adjustment Error, and a small-to-modest amount of Measurement Error. Overall, the three probability samples as a group were less biased on the substantive measures and had less variance from the benchmark values, compared to the nonprobability surveys.  The five nonprobability surveys likely had a nonignorable amount of Coverage Error, an unknowable amount of Sampling Error, a nonignorable amount of Nonresponse Error, unknown Adjustment Error, and a small-to-modest amount of Measurement Error. Overall, the five nonprobability panel surveys as a group were more biased on the substantive measures and had more variance from the benchmark values, compared to the probability surveys.  Our AOPBS study replicates very closely with previous comparison studies conducted in other countries.