Oops, They Didn’t Answer Again! Dealing with Systematic Item Nonresponse in a Factorial Survey Experiment

Abstract

Factorial survey experiments (FSEs) are widely used tools for identifying judgment princi-ples and subgroup differences, leveraging their experimental designs to draw causal inferences.Systematic item nonresponse can undermine this advantage, arising from diverse and frequentlyunobserved causes before data collection, such as questionnaire design, or during data collection,such as respondent characteristics. This paper investigates an FSE on entrepreneurs’ likelihoodof purchasing voluntary unemployment insurance in Germany, which experienced unexpectedlyhigh item nonresponse. The paper addresses the following research questions: 1) What designflaws or contextual factors may have contributed to the high item nonresponse rates? 2) How canresearchers consider – and possibly prevent – systematic item nonresponse in factorial surveyexperiments? The analyses reveal that the item nonresponse is primarily driven by the perceivedimplausibility of certain dimension levels. Furthermore, the study shows that the impact can bemitigated by excluding the problematic dimension level(s) without compromising level balanceor the independence of all dimension levels. The findings highlight the importance of thoroughquestionnaire pretesting, especially for cross-national policy adoption, and of identifying andaddressing potential causes of item nonresponse to ensure the validity of FSE results. Ulti-mately, this “failed” study offers a valuable learning opportunity and contributes to transparencyin methodological reporting.