Sequence Matters in Web Probing: the Impact of the Order of Probes on Response Quality, Motivation of Respondents, and Answer Content

Abstract

Due to the growing significance of international studies, the need for tools to assess the equivalence of items in international surveys is pressing. Online probing is a powerful tool for identifying the causes of non-equivalence; it incorporates probing techniques from cognitive interviewing in cross-national web surveys. So far, our online probing approach has applied three different probe types – category-selection probes, specific probes, and comprehension probes – to inquire about different aspects of an item. Previous research has mostly asked one probe type per item but in some situations it might be preferable to test potentially troublesome items with multiple probe types. However, empirical evidence is missing on whether the sequence of probe types has an impact on response quality, respondents’ motivation, and answer content. In this study, we report evidence from a web experiment that was conducted with 1,354 respondents from Germany, Great Britain, the U.S., Spain, and Mexico in June 2014. In this experiment, we asked respondents three different probes for one item and we manipulated the sequence of probes in each experimental condition. Our research indicates that the sequence in which different probe types are asked has an impact on response quality, the respondents’ motivation, and probe answer content. However, the respondents in the five countries react differently on the variation in the probe sequence suggesting that response behavior on probes is partly culturally driven.