Response Burden and Response Quality in Web Probing: An Experiment on the Effects of Probe Placement and Format

Abstract

Probes are follow-up questions to survey questions used for question pretesting and evaluation. A probe can be placed concurrently - that is directly following the survey question it relates to - or retrospectively later or at the end of a survey. Retrospective probing ensures that the natural flow of the survey questions is not interrupted. In contrast, concurrent probing is argued to prioritize the quality of probe responses, as respondents can still access their short-term memory. To date, concerns that retrospective probe placement negatively impacts how probes are perceived and answered have not been tested experimentally. Recently, it has been suggested to employ "closed" probes with predefined response options, for instance using a check-all-that-apply format, also with the rationale of improving probe response quality. Closed probe formats suffer from fewer nonresponse and more themes being chosen than being typed in response to open-ended probes. The present research reports a study on the impact of probe placement on response burden and response quality of web probes. Moreover, it examines whether the effects of probe placement are moderated by probe format. Results show that respondents needed longer to recapitulate the survey question when probes are asked retrospectively and were less likely to offer interpretable probe response content for one of three examined probes. These effects were not moderated by probe format. However, respondents who answered open-ended probes retrospectively were more likely to rely on contextual cues from intermittent survey questions. This effect did not occur when respondents were presented probes with predefined response options. Implications for web probing design are discussed.