Question order and panel conditioning analysing self-reported data

Abstract

Self-ratings might not be directly comparable in socio-economic groups because of the presence of Differential Item Functioning (DIF) across individuals: survey respondents might interpret, understand or use the response categories for the same question in different ways, due to different perceptions, attitude and so on. The instrument of the anchoring vignettes has been introduced in the literature to provide DIF-free individual assessments. To take the DIF problem into account, in this paper we apply the anchoring vignette approach to investigate the extent of longitudinal priming effects for measuring the customer satisfaction of some online banking services in Italy, when the self-evaluation question is asked after (instead of before) the vignette questions. To this aim, a sample composed by a panel and a cross-sectional component of respondents is used: the former experienced the anchoring vignettes in the previous wave of the survey, the latter did not. We show that only in the longitudinal component priming effects due to the question order are not observed. Moreover, the knowledge of the anchoring vignette tool through participation in previous waves (that may lead to a better understanding of the meaning of the questions providing self-evaluations) leads to some changes in the response styles and in the perceived level of satisfaction with respect to the answers of the respondents who experienced the anchoring vignettes for the first time. The introduction of the anchoring vignettes induces some priming effects, that occur in different ways both in the perceived level of satisfaction and in the individual response style. These question order effects move in two opposite directions that, in some cases, might lead to some forms of compensation.