Age and Sex Effects in Anchoring Vignette Studies: Methodological and Empirical Contributions

Abstract

Anchoring vignettes are an increasingly popular tool for identifying and correcting for group differences in use of subjective ordered response categories. However, existing techniques to maximize response consistency (use of the same standards for self-ratings as for vignette-ratings), which center on matching vignette characters’ demographic characteristics to respondents’ own characteristics, appear at times to be ineffective or to pose interpretive difficulties. Specifically, respondents often appear to neglect instructions to treat vignette characters as age peers. Furthermore, when vignette characters’ sex is matched to respondents’ sex, interpretation of sex differences in rating style is rendered problematic. This study applies two experimental manipulations to a national American sample (n=1,765) to clarify best practices for enhancing response consistency. First, an analysis of two methods of highlighting vignette characters’ age suggests that both yield better response consistency than previous, less prominent means. Second, a comparison of ratings of same- and opposite-sex vignette characters suggests that, with avoidable exceptions, the sex of the respondent rather than of the vignette character drives observed sex differences in rating style. Implications for interpretation and design of anchoring vignette studies are discussed. Findings also show significant sex, educational, and racial/ethnic differences in styles of rating health, and racial/ethnic differences in styles of rating political efficacy. These findings underscore the incomparability of unadjusted subjective self-ratings across demographic groups.