The Three-Step Test-Interview (TSTI): An observation-based method for pretesting self-completion questionnaires

Abstract

Three-Step Test-Interview (TSTI) is a method for pretesting a self-completion questionnaire by first observing actual instances of interaction between the instrument and respondents (the response process) before exploring the reasons for this behavior. The TSTI consists of the following three steps: 1. (Respondent-driven) observation of response behavior. 2. (Interviewer-driven) follow-up probing aimed at remedying gaps in observational data. 3. (Interviewer-driven) debriefing aimed at eliciting experiences and opinions. We describe the aims and the techniques of these three steps, and then discuss pilot studies in which we tested the feasibility and the productivity of the TSTI by applying it in testing three rather different types of questionnaires. In the first study, the quality of a set of questions about alcohol consumption was assessed. The TSTI proved to be productive in identifying problems that resulted from a mismatch between the ‘theory’ underlying the questions on the one hand, and features of a respondent’s actual behavior and biography on the other hand. In the second pilot study, Dutch and Norwegian versions of an attitude scale, the 20-item Illegal Aliens Scale, were tested. The TSTI appeared to be productive in identifying problems that resulted from different ‘response strategies’. In the third pilot, a two-year longitudinal study, the TSTI appeared to be an effective method for documenting processes of ‘response shift’ in repeated measurements of health-related Quality of Life (QoL).