Learning Through Failures. What Unsuccessful Survey Experiments Teach Us About Design, Measurement, and Inference

Abstract

Survey experiments have become an indispensable tool in the social sciences for estimatingcausal effects in diverse populations. Yet, the apparent simplicity of experimental designs canbe deceptive, as weak treatments, noncompliance, or implementation breakdowns may leadto statistically nonsignificant or otherwise unexpected results that leave researchers wonderingwhether to conclude that their studies have failed or show true null effects. This special issueof Survey Research Methods presents empirical and methodological contributions that examinesurvey experiments that did not work as planned. Rather than dismissing unexpected or non-sig-nificant results, the papers address key challenges in treatment validation, context dependency,treatment-related nonresponse, precisely articulated causal states, field implementation, and in-direct questioning techniques. Collectively, the contributions illustrate that unexpected findingsreveal critical assumptions connecting experimental design to causal inference. By emphasiz-ing pretesting, field monitoring, paradata, manipulation checks and other diagnostic measures,this special issue highlights the value of publishing carefully diagnosed ‘failures’ to advancecumulative methodological knowledge.