Investigating Direction Effects in Rating Scales With Five and Seven Points in a Probability-Based Online Panel

Abstract

Survey questions with rating scales are a common method in attitude measurement. Similar to other scale characteristics, scale direction and its effects on answer behavior and data quality deserves special attention. This particularly applies to scale direction effects across different scale lengths. In order to contribute to the current state of research, we investigate scale direction effects across scales with five and seven points by analyzing observed and latent answer distributions including composite reliabilities. We conducted an experiment in the probability-based German Internet Panel (N = 4,676) using questions on achievement and job motivation that vary scale direction (i.e., decremental and incremental) and scale length (i.e., five and seven points). The results reveal differences between scales with five and seven points. Five-point scales are more robust against scale direction effects than their seven-point counterparts. In addition, decremental and incremental scales with five points are invariant. This does not apply to decremental and incremental scales with seven points. However, composite reliabilities are higher for scales with seven points than for scales with five points. This is irrespective of the scale direction.