Facilitated by greater access to survey platforms, researchers have increasingly turned to survey experimental paradigms to examine voter decision-making (Druckman et al. 2011; Mutz 2011; Sniderman and Grob 1996). These efforts frequently involve vignettes or other stimuli that directly manipulate information about real or hypothetical political actors. Despite the many advantages of survey experimentation in addressing endogeneity inherent in observational data, the enterprise is not without challenges. While much attention has been paid to external validity concerns in survey experimental research (Barabas and Jerit 2010), we highlight a design challenge more related to internal validity.
What we term conjectural bias describes a special case in which respondents systematically go beyond the intentionally incomplete information presented to them under a “control” condition and impute characteristics researchers seek to manipulate through their stimuli. This can occur when the context of an experimental study makes 1) a piece of information about the experimental target’s characteristics or group membership conspicuous in its absence and 2) a respondent’s own group identity makes them more likely to assume that positively valenced descriptions are of ingroup members and negatively valenced descriptions refer to members of the outgroup. In certain circumstances, this loss of experimental control can produce problematic biases, even if respondents indicate explicitly that they are not making an inference. We first began to suspect this effect as we sought to interpret findings from an experimental study on partisan motivated reasoning in which we had included a no-party control condition alongside Republican and Democratic conditions. Our suspicion prompted us to field a follow-up experiment, which we use to demonstrate and explore this phenomenon.
Beyond simply being the subject of our initial survey experimental “failure,” the case of party identification and partisan labels is well suited to our examination for a number of reasons. For starters, party identification and partisan cues are well established as central forces guiding political behavior in the United States (Arceneaux and Nicholson 2024; Goggin and Theodoridis 2017; Goggin et al. 2020; Nicholson 2012; Heit and Nicholson 2016; Nicholson 2011; Druckman et al. 2013; Theodoridis et al. 2023). Furthermore, the current hyper-polarized climate in American politics has heightened scholarly interest in parsing the effects of party in voter behavior. Voters are known to have deep partisan identities (Theodoridis 2017) and, in the American context, the omission of a partisan label in most political stimuli is likely conspicuous.
Political party shapes political cognition through at least two theoretically distinct pathways: effects of identity, or one’s affiliation with a party, and effects of stereotypes, or the role of a stable party image. While the role of party images and stereotypes is important, party’s role as a perceptual screen and social identity colors the political world for partisans. These dual effects, combined with the highly salient nature of party in American politics, create just the sort of environment in which conjectural bias can substantially confound efforts to operationalize experimental controls. Because party labels are seemingly ubiquitous and undeniably powerful, many researchers have sought to minimize or circumvent the effects of party for the purpose of studying party cues and other topics. This could be done by omitting party from stimuli to generate a “control” condition or omitting party from all experimental conditions in studies that manipulate other variables. However, the same ubiquity and importance that motivate researchers to manipulate or exclude party labels drive respondents to impute them when they are absent from experimental stimuli.
Others have shown that information provided in vignettes can lead to inferences. Westwood and Peterson (2022) demonstrate how race and partisanship are fused in the public’s mind, something that Butler and Broockman (2011) effectively address by manipulating both dimensions in a field experimental context. With a different methodological approach, Goggin et al. (2020) find a large number of characteristics associated with party and ideological labels. In a study closely related to our inquiry, Dafoe et al. (2018) find that respondents make inferences based upon information provided regarding countries in vignette experiments, and offer an approach for dealing with these inferences. The conjectural bias that we describe is related to but distinct from the inferences Dafoe et al. (2018) and others highlight. Rather than highlighting statistical inferences prompted by information provided in an experiment, we emphasize inferences prompted by the omission of conspicuously absent information and driven by an interaction between the valence of the vignette and each respondent’s identity.
The pronounced bias that can result is relevant to scholars producing or consuming work on the experimental effects of party labels, and more broadly, studies of any political subject using vignettes but not specifying political party. Furthermore, while we focus on the partisan dimension here, conjectural bias likely represents a threat to inference in other areas of study (e.g., race, religion, ethnic group, gender) where contextually conspicuous absence of labels, vignette valence, and respondent identity might interact. This bias is problematic not only because it means respondents may infer stereotypical information about a target, but because it also creates motivation to change their responses based on their own identity.
The only certain mechanism for eliminating this bias is abandoning no-label conditions, and we suggest that scholars consider doing so whenever possible. However, we also suggest best practices for the design of studies and presentation of results to minimize the effects of conjectural bias when a no-label condition is deemed necessary. We highlight the types of work susceptible to conjectural bias and detail its effects. We then describe results from two separate studies that empirically demonstrate both the presence of conjectural bias and its problematic consequences.
Survey experimental work on the topic of political candidate evaluation has generally attempted to isolate one or two dimensions of candidates’ policy, personality, performance, or demographic characteristics, while controlling for all other attributes, including party of the candidate. McGraw (2011) describes three general ways of doing this:
“A third important distinction is how the partisan affiliation of the candidate is handled in the experimental design. Three possibilities exist. The first is to vary the partisan affiliation of the stimulus candidate so that partisanship becomes an independent variable that is fully crossed with other manipulated variables. The second approach is to make explicit the partisanship of the target candidate and hold it constant. Finally, in some studies, the partisan affiliation of the candidate is left out altogether.” (McGraw 2011, p. 189)1
McGraw herself notes a very common criticism of both the second and third design options: “In short, when researchers do not manipulate partisan affiliation, they risk failing to detect contingency effects or overstating the influence of other factors.” (McGraw 2011, p. 190) We share this concern, as well as others raised in the literature. McGraw’s is not an exhaustive list of options for ‘controlling’ for political party label in the U.S. context. One could also identify a target in a vignette as an Independent or specify that a candidate being described is running for a local or municipal office, which are often nonpartisan. These approaches could work for certain studies, but come with obvious limitations. For starters, Independents in the U.S. electoral context are rare and the Independent label does not necessarily signal partisan neutrality, but rather a separate, intentional political identity. For instance, several elected officials have run as Independents as a way station in the process of switching from one party to the other. Others remain Independents, but consistently caucus with one of the two parties. In practice, political independence in the United States is not necessarily tantamount to the absence of party identity, and may invoke its own set of inferences. Specifying that a candidate is seeking local office does provide cover for not providing a party attachment in experimental stimuli, and this may be a useful option in some cases. One obvious limitation of this approach is that it limits the scholar to topics that can be studied using local races. Also, just because many local offices and the races for them may be nonpartisan does not mean the individuals seeking them do not have party affiliations that are well established or known. In fact, voters in these elections may be particularly attuned to subtle cues indicating party leanings. Most importantly, given the dominance of the two parties in American politics at all levels, neither of these two approaches can guarantee that respondents will not attempt to infer the party of the described candidate.
The practice of omitting the party in the descriptions of candidates is widespread in the literature. A systematic search of the literature finds more than two dozen articles published in American Political Science Review, American Journal of Political Science, The Journal of Politics, Political Behavior, and Political Psychology in the past 15 years alone.2 There are numerous examples across political science literatures—on gender (e.g., Kahn 1992), race (e.g., Sigelman et al. 1995), religion (e.g., Campbell et al. 2011), appearance (e.g., Riggle et al. 1992), scandal (e.g., Miller 2010), policy positions (e.g., Tomz and Van Houweling 2009), and others in candidate evaluation (e.g., McGraw et al. 2003).
Why is this omission of party labels potentially problematic? Person perception, and more specifically politician perception, is dominated by a drive to categorize people.3 Certain types can be more or less salient depending on the domain in which the target person is judged. In the political arena, party defines an important type. When a party label is omitted in a vignette or informational treatment, whether real or fictional, respondents may seek to fill in the party label based on other information. The information in a vignette may lead to a party guess in two ways. First, some experimental treatments contain information stereotypical of one party or another, such as an issue position or priority (Egan 2013) or a personality trait (e.g., Hayes 2005). This is the party version of the information equivalence highlighted by Dafoe et al. (2018) and can be extremely problematic, as it confounds the effect of party with any information that is stereotypically partisan. Given the number of issues, demographic characteristics, and personality traits with well-documented associations with the Democratic and Republican parties, many literatures are affected. Second, simpler partisan identity-related processes may lead mere valence of information in the vignette to signal partisanship differentially based on respondent partisan identification. That is, because most voters identify with one of the two major political parties, respondents will engage in motivated reasoning and seek to categorize positively valenced politicians as copartisans and negatively valenced politicians as members of the out-party. Therefore, we would expect generally positive experimental vignettes to create copartisan biases, while negative vignettes would be rated as the opposite party of the respondent.
We term this second inferential process conjectural bias. Some prior work has described the more general problem of respondents inferring information based on that presented in a vignette (Sher and McKenzie 2006; Dafoe et al. 2018). Conjectural bias is an especially problematic case, because it does more than just create unmeasured systematic variance in the overall evaluations of the candidate, but it requires that the partisanship of the sample be taken into account, as this distribution can otherwise bias estimates of evaluation averages. Additionally, theoretical claims from studies regarding the absence of party effects may be unfounded; party is not absent, it is inferred.
Effective design-based inference requires a researcher to precisely define both the treatment in an experiment and that treatment’s relevant counterfactual, often operationalized as a control condition. In the case of political party, omission amounts to misspecification of the appropriate counterfactual. When considered this way, it becomes clear that studies can only realistically examine the effect of a candidate being a Democrat as opposed to a Republican. The often assumed baseline condition of “no party” is not meaningful, as it varies across respondents. In other words, the realistic counterfactual for one party label is the other party label, not the absence of one. Conjecture on the part of respondents does not invalidate the randomization. Instead, it renders the no-label cell a blend of respondents not treated and respondents who are self-selecting into the other treatment conditions. This biases resulting estimates.
We first began to suspect the potential effects of conjectural bias when interpreting the results of a survey experimental study designed to measure partisan motivated reasoning. In that experiment, fielded through YouGov as part of the 2010 Cooperative Election Study, respondents (N = 1674) were randomly assigned to read a version of a fictional news report about a Senator admitting to lying about his opponent (see Fig. 1).4 They were then asked to respond to a series of 9‑point Agree/Disagree items about the report and the Senator. The statements to which subjects responded were designed to ascertain the extent to which they were engaging in various mechanisms of motivated reasoning and possessed more negativity towards the candidate: “This report seems fair,” “The person who wrote this is probably biased” (reverse-coded), “This sort of thing is important to me when deciding which candidate to support,” “The Senator deserves credit for admitting this” (reverse-coded), and “The behavior that got the Senator in trouble is typical.”5 Together, a higher rating indicates more negativity towards the candidate. The versions of the fictional news report were identical except that the politician was described as either a “Democratic Senator,” “Republican Senator,” or simply “Senator”—with the last description intended to serve as a “control” condition.
The purpose of the no-party condition was to allow us to determine the direction of bias. Is partisan motivated reasoning more a punishment of the out-party or in-group favoritism? Figure 2 shows the overall ratings of the candidate by respondent partisanship and the party label of the candidate.6 A consistent pattern emerges. For both Democratic and Republican respondents, the average for the “control” condition, in which no party label is provided, is far closer to that of the out-party condition. Taken at face value, this result might suggest that motivated reasoning is mostly in-group favoritism. But we were skeptical, providing the impetus for our examination of conjectural bias. Were respondents simply assuming the Senator being described in a negative light hailed from the out-party? If so, would the reverse happen if the description were positive?
While the study described above produces results consistent with what we have come to call conjectural bias, it does not prove that respondents are actually inferring the party of the “control” candidate. To more cleanly establish the presence of conjectural bias, we fielded an experiment in which respondents were either shown a positive or negative news story about a fictional candidate. We presented the news stories shown in Fig. 3 to a sample of the U.S. voting-age population recruited through Survey Sampling International (SSI) (N = 1609) in 2014. The experimental design was between-subjects, with respondents only seeing one news story. Following the article excerpt, respondents were asked two questions: whether “the Senator deserved credit for [this or admitting this]” on a 1–5 Likert scale, with higher numbers corresponding to more credit, and whether they think the Senator is more likely a Democrat or a Republican, also on a 1–5 scale, rescaled to 0–1 for the figure, with lower numbers indicating the candidate is more likely a Democrat and higher numbers indicating the candidate is more likely a Republican.
Figure 4 shows the source of conjectural bias more directly, with partisan respondents always more likely to guess the positive story is about a copartisan candidate while the negative story is about a candidate of the other party. If we condition on the guessed party of the candidate, we see results consistent with our initial “failed control” study. Figure 5 displays ratings of the credit deserved by the candidate pooling across both positive and negative news stories, conditioned by respondent partisanship and the presumed partisanship of the candidate.7 As in the initial study, we see Democratic and Republican respondents rate the candidate presumed to be copartisan more positively. In this case, since we are not providing a party label, they appear to be basing that rating on their conjecture regarding the Senator’s party.
What about survey respondents who did not offer a guess of the party of the candidate? Because the center option of this scale indicates that respondents said they did not think the candidate was more or less likely to be Republican or Democratic, we can analyze these respondents who did not make a guess. Figure 6 shows the dependent variable of credit deserved by respondent partisanship and respondent guess for the negative vignette, while Fig. 7 shows the results for the positive vignette. Notably, while the “no guess” ratings on the positive vignette in Fig. 7 appear to be very close to the average/midpoint between the Democratic and Republican guesses across partisanship, we do not see this for the negative information in Fig. 6. Specifically, we see respondents who offered no guess rating the candidate much more closely to the out-party among Democratic and Republican respondents, suggesting that with negatively valenced information conjectural bias may still be at work even if respondents did not explicitly indicate they guessed the candidate’s party.
Scholars could address conjectural bias by narrowly defining the treatment as the party label itself in the experimental stimuli. But, such an assertion is likely disingenuous in most cases, as the validity of the experimental manipulation relies on interpretation of a party label as something more than just the presence or absence of a word or phrase in an experimental vignette, but an extrapolation to some real-world representation of the described candidate or policy. In practice, there are steps we suggest scholars take to curtail conjectural bias.
First, experimentalists should simply not intentionally omit party or other highly salient information as a way of controlling for it. The only certain mechanism for eliminating this bias is abandoning no-label conditions, both as controls or across experimental conditions in studies where party is likely relevant and conspicuous in its omission, but perhaps not the focus of the study. Even if the party of a candidate or entity is not of primary interest, one can keep the party constant or tailor it so that respondents always encounter copartisan or opposite party candidates.8
Second, if one does omit information that may be relevant to respondent decisions, control checks should be implemented. In her canonical work on survey experiments, Mutz (2011) describes at length the importance of manipulation checks when designing experimental treatments. Specifically, one should measure post-outcome whether the treatment was perceived in the manner the researcher expects. Analogous to these, control checks should ask respondents to “guess” or categorize the experimental target on a variety of pieces of omitted information. If these checks show no differential guessing on the omitted information, both overall and heterogeneous based on respondent characteristics, it provides some reassurance that the results are not due to the omitted information. Also, if one wishes to treat subjects who infer party labels as ‘non-compliers’ and model average treatment effects, these control checks make that possible. However, as we demonstrated above, even when respondents indicate they are not inferring a party, it is possible some still are, particularly in assuming negative information applies to out-party candidates.
Finally, when analyzing data from studies in which a control condition omits a party label, results should specify estimands comparing party conditions with each other, not with control quantities. In particular, results should be reported under each experimental condition, not effects “as compared with control.” And, mean differences for cells defined by key relevant identities, such as party identity, should be presented.9
Researchers often operationalize “control” conditions by omitting treatment labels in survey experimental vignettes. Using the motivating example of party labels, which are seemingly ubiquitous in U.S. political discourse, and therefore conspicuous in their absence, we show that holding such highly relevant and salient labels constant by simply excluding them from experimental stimuli results in a robust bias. In the case of political party, this ‘conjectural bias’ colors evaluations because partisans actively infer the party of the vignette’s subject through an interaction between the content of the stimuli and the respondent’s own party identity. The root cause of this phenomenon is a poorly defined counterfactual—respondents are engaging in this conjecture because the appropriate counterfactual for one party label is the other party label, not the absence of one. The pronounced bias we discuss here is clearly relevant to scholars producing or consuming work on the experimental effects of party labels, and more broadly, studies of any political subject using vignettes but not specifying political party.
Furthermore, while we focus on the partisan dimension here, conjectural bias likely represents a threat to inference in other areas of study where conspicuously absent labels and respondent identity might interact. Thus, we suggest that scholars designing experimental stimuli on a broad range of topics should be attuned to the presence of the three conditions that might conspire to foster conjectural bias: 1) an omitted descriptor that one might normally expect given the study’s context; 2) a vignette that paints the target in a positive or negative light; and 3) the likelihood that a respondent’s own identity might motivate them to succumb to in-group-favorable conjecture. It is easy to imagine these conditions being met in studies that directly or tangentially touch on highly salient identities such as race, sex, nationality, religion, ethnicity, and sexual orientation, among others. In all of these cases, it is worth considering that the relevant counterfactual for one label (e.g., Black, female, Muslim, LGBTQ) may be another label, not the absence of one.
Replication data and code are available on the Harvard Dataverse here: https://doi.org/10.7910/DVN/ISSAYP.
Anderson, C. A., & Sedikides, C. (1991). Thinking about people: contributions of a typological alternative to associationistic and dimensional models of person perception. Journal of Personality and Social Psychology, 60(2), 203–217. →
Arceneaux, K., & Nicholson, S. P. (2024). Anchoring political preferences: the psychological foundations of status quo bias and the boundaries of elite manipulation. Political Behavior, 46(2), 751–775. →
Barabas, J., & Jerit, J. (2010). Are survey experiments externally valid? American Political Science Review, 104(2), 226–242. →
Butler, D. M., & Broockman, D. E. (2011). Do politicians racially discriminate against constituents? A field experiment on state legislators. American Journal of Political Science, 55(3), 463–477. →
Campbell, D. E., Green, J. C., & Layman, G. C. (2011). The party faithful: partisan images, candidate religion, and the electoral impact of party identification. American Journal of Political Science, 55(1), 42–58. →
Dafoe, A., Zhang, B., & Caughey, D. (2018). Information equivalence in survey experiments. Political Analysis, 26(4), 399–416. a, b, c, d
Druckman, J. N., Green, D. P., Kuklinski, J. H., & Lupia, A. (Eds.) (2011). Cambridge handbook of experimental political science. Cambridge University Press. →
Druckman, J. N., Peterson, E., & Slothuus, R. (2013). How elite partisan polarization affects public opinion formation. American Political Science Review, 107(01), 57–79. →
Egan, P. J. (2013). Partisan Priorities: How Issue Ownership Drives and Distorts American Politics. Cambridge University Press. →
Goggin, S. N., & Theodoridis, A. G. (2017). Disputed ownership: parties, issues, and traits in the minds of voters. Political Behavior, 39, 675–702. →
Goggin, S. N., Henderson, J. A., & Theodoridis, A. G. (2020). What goes with red and blue? Mapping partisan and ideological associations in the minds of voters. Political Behavior, 42, 985–1013. a, b
Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014). Causal inference in conjoint analysis: understanding multidimensional choices via stated preference experiments. Political Analysis, 22(1), 1–30. →
Hayes, D. (2005). Candidate qualities through a partisan lens: a theory of trait ownership. American Journal of Political Science, 49(4), 908–923. →
Heit, E., & Nicholson, S. P. (2016). Missing the party: political categorization and reasoning in the absence of party label cues. Topics in Cognitive Science, 8(3), 697–714. →
Kahn, K. F. (1992). Does being male help? An investigation of the effects of candidate gender and campaign coverage on evaluations of us senate candidates. Journal of Politics, 54(2), 497–517. →
McGraw, K. (2011). Candidate impressions and evaluations. In: Cambridge handbook of experimental political science. Cambridge University Press (pp. 187–200). a, b, c
McGraw, K., Hasecke, E., & Conger, K. (2003). Ambivalence, uncertainty, and processes of candidate evaluation. Political Psychology, 24(3), 421–448. →
Miller, B. (2010). The effects of scandalous information on recall of policy-related information. Political Psychology, 31(6), 887–914. →
Mutz, D. C. (2011). Population-based survey experiments. Princeton University Press. a, b
Nicholson, S. P. (2011). Dominating cues and the limits of elite influence. The Journal of Politics, 73(4), 1165–1177. →
Nicholson, S. P. (2012). Polarizing cues. American Journal of Political Science, 56(1), 52–66. →
Riggle, E. D., Ottati, V. C., Wyer Jr, R. S., Kuklinski, J., & Schwarz, N. (1992). Bases of political judgments: the role of stereotypic and nonstereotypic information. Political Behavior, 14(1), 67–87. →
Sher, S., & McKenzie, C. R. (2006). Information leakage from logically equivalent frames. Cognition, 101(3), 467–494. →
Sigelman, C. K., Sigelman, L., Walkosz, B. J., and Nitz, M. (1995). Black candidates, white voters: understanding racial bias in political perceptions. American Journal of Political Science, 39(1), 243–265. →
Sniderman, P. M., & Grob, D. B. (1996). Innovations in experimental design in attitude surveys. Annual Review of Sociology, 22(1), 377–399. →
Theodoridis, A. G. (2017). Me, myself, and (i), (d), or (r)? partisanship and political cognition through the lens of implicit identity. The Journal of Politics, 79(4), 1253–1267. →
Theodoridis, A. G., & Goggin, S. N. (2026). Replication Data for: losing control…of the party: conjectural bias in survey experiments. Harvard Dataverse. https://doi.org/10.7910/DVN/ISSAYP
Theodoridis, A. G., Goggin, S. N., & Deichert, M. (2023). Separated by politics? Disentangling the dimensions of discrimination. Political Behavior, 45(4), 2025–2051. a, b
Tomz, M., & Van Houweling, R. (2009). The electoral implications of candidate ambiguity. American Political Science Review, 103(01), 83–98. →
Westwood, S. J., & Peterson, E. (2022). The inseparability of race and partisanship in the united states. Political Behavior, 44(3), 1125–1147. →
