Losing Control (of the Party): Conjectural Bias in Survey Experiments

Abstract

Researchers often operationalize “control” conditions in survey experiments by omitting treat-ment information or labels in vignettes. With our own “failed” prior use of this practice whenrandomizing political party labels as our impetus, we show that holding a highly salient labelconstant by excluding it from experimental stimuli can result in a robust bias. In the politicalparty context, this conjectural bias colors evaluations because partisans actively infer the partyof the vignette’s subject through an interaction between the content of the stimuli and the re-spondent’s own party identity. We showcase this effect in two separate experiments, using botha positive description of a candidate (prompting respondents to rate the target as copartisan), anda negative description of a candidate (prompting respondents to assume the candidate is fromthe other party). The only certain mechanism for eliminating this bias is abandoning no-labelconditions, and we argue that scholars consider doing so whenever possible. Otherwise, controlchecks can serve as valuable tools to ensure that respondents do not make unwanted inferencesregarding omitted information.