The Role of Interviewer Observations in Obtaining Representative Data in Repeated Cross-National Studies

Abstract

Collecting representative data from surveys can be challenging in a survey climate where response rates are declining and costs are rising, but such data are essential for making valid inferences in social research. These issues can be compounded when conducting cross-national surveys that are repeated over time. This paper utilises the European Social Survey (ESS) contact form data to explore cross-national associations with nonresponse over time. Using sample frame data and interviewer observations, a form of paradata where interviewers record characteristics of the household and its neighbourhood, we construct representativity indicators (R-indicators) for each country over nine survey rounds and identify variables that contribute to non-representativeness. We find that interviewer observations produce evidence of non-representativeness, even after controlling for frame information, consistently and without large cross-national variations. In addition, they are stable over time, despite the trend of declining response rates. Consistent associations with the propensity to respond can be leveraged in several ways in survey practice, and we discuss how interviewer observations may be utilised in tasks such as fieldwork monitoring, to inform case prioritisation schemes where units associated with underrepresented characteristics are targeted (including in adaptive and responsive survey designs), and in post-survey adjustment in repeated cross-national surveys. These methods may reduce the risk of differential survey errors occurring in such surveys.