Measuring the Effect of Questionnaire Design on Rating Responses With the Class of CUB Models

Abstract

Responses to questionnaire items can be influenced by various factors including sample design, interview mode and/or how questions are phrased. To analyse these aspects, this paper draws on the Bank of Italy's surveys of households and firms, which employ different survey modes or questions with different phrasings, response options, or graphical features for sub-samples of respondents. We exploit the potential of CUB (Combination of Uniform Discrete and shifted Binomial random variables) modelling for the analysis of ordinal data. CUB models are able to capture and identify the different components of the cognitive process behind the responses and to study how these are related to the relevant covariates (such as respondents' characteristics). The results show that in general, although diverse survey modes and a different phrasing or graphical representation of questions may yield somewhat different findings in terms of uncertainty, responses to relevant questions such as those on reported satisfaction or expectations did not produce pronounced differences in data reliability.