Valid vs. Invalid Straightlining: The Complex Relationship Between Straightlining and Data Quality

Abstract

Straightlining – the tendency to give the same response to a series of grouped questions – can be the result of satisficing respondents. As a result, many survey practitioners use straightlining as one, and sometimes the only, indicator of data quality. Respondents identified as straightliners are often removed from the data set on the assumption that their answers are meaningless. In this paper we show that these practices are based on a logical fallacy and demonstrate that in many common survey formats, the incidence of straightlining can be increased by improving the validity and the reliability of survey questions. We take initial steps in investigating the complexities and challenges of data analysis by providing a formal definition of valid straightlining and leverage that definition in a series of Monte Carlo simulations to better understand the conditions that give rise to valid straightlining. Although it remains for future work to distinguish valid from invalid straightliners, our formal definition of the concept and our simulation methods augment the tools survey analysts employ in assessing the prevalence of low effort respondents in survey data sets. The paper thereby takes initial steps toward sounder methods of classifying straightliners as optimizers or satisficers.