Using Experimental Vignettes to Study how Survey Methods and Findings Affect the Public's Evaluation of Public Opinion Polls: Considering a Dual Process Approach

Abstract

Understanding how the public thinks about and understands survey findings is an important part of understanding the role of surveys in policymaking and democracy more broadly. In this paper, we examine the results of three vignette experiments conducted within representative sample surveys in which the methodology and results of a hypothetical public opinion survey assessing support for specific public policy proposals was manipulated. Respondents’ beliefs about the accuracy of the survey described in the vignette experiment, and their beliefs about whether or not it should be considered by policymakers were measured. We used both manipulated (e.g., the methodological rigor of the survey described in the vignette) and measured (e.g., respondents’ opinions on the proposed policy about which public opinion was measured in the vignette) to test four different models that could be used to explain public evaluations of public opinion surveys: (1) the rational actor model, which suggests that people will evaluate more methodologically rigorous surveys more positively; (2) the science literacy model, which suggests that people high in science literacy will evaluate more methodologically rigorous surveys more positively than will people low in science literacy; (3) the motivated reasoning model, which suggests that people will evaluate surveys more positively when the survey results are consistent with their prior opinions than when they are inconsistent; and (4) a dual process model approach, which suggests that people will evaluate more methodologically rigorous surveys more positively only when they are both able and motivated to do so.   We found some support for the scientific literacy and motivated reasoning models, but these findings were qualified by an interaction between factors associated with respondent motivation, ability, and survey methodology rigor that strongly supports the dual process model perspective.