Observing Interviewer Performance in Slices or by Traces: A Comparison of Methods to Predict Interviewers’ Individual Contributions to Interviewer Variance

Abstract

The interviewing practice of survey interviewers has long been recognized as an important contributor to measurement error in survey data. In the current article, we compare two approaches that can be used to identify interviewers whose task performance might be inadequate and damaging to data quality. The first approach assesses interviewing behavior through the use of audio-recorded interviews. Behavioral assessments capture actual behavior in an interview, but typically rely only on “slices” of observed behavior. The second approach is based on interview time paradata, a type of “trace” data that can easily be aggregated in summary measurements such as average interview speed at the interviewer level. In the current study, we use data from the Dutch-speaking subsample of interviewers employed in two survey rounds of the European Social Survey in Belgium to evaluate how successful the two above approaches are for predicting interviewers’ contributions to interviewer variance. The results show that interviewers who deviate from a larger number of standardized interviewing practices in one (early) audio-recorded interview, as well as those who tend to accelerate their interviewing speed over the course of an interview, tend to contribute more to interviewer variance. The two types of performance assessments appear to be independent, additive predictors of interviewers’ variance contributions. While statistically significant, the effects are nevertheless modest in size. The implication for practice is that interviewer monitoring would benefit from well-considered combinations of both behavioral and paradata-based assessments.