Responding as Expected? The Effects of Survey Mode on Estimates of Sensitive Attitudes in Self-Completion and Face-To-Face Interviews of the European Social Survey

Abstract

Conducting face-to-face surveys is becoming increasingly challenging. The shifting technological and social environment of survey data collection is leading researchers to seek more sustainable solutions through new data collection techniques and rethinking of traditional survey methods. Recent examples are the Push-to-Web (PtW) survey experiments initiated by the European Social Survey (ESS) research consortium, which using probability sampling reach respondents through postal invitation and combines web-, and paper-based self-completion questionnaires. The literature indicates that the method of data collection may have a strong impact on who participates in a survey and how people answer the questions. These two phenomena are combined in the concept of the mode effect. This paper evaluates the PtW surveys in Hungary from the perspective of mode effects by pairing two PtW surveys with face-to-face ESS surveys closest in time. The evaluation is made along the sample compositions and the answers to two culturally and socially sensitive questions for which the different survey situation is seen as a highly relevant factor due to the specific political context in Hungary: attitudes towards (1) gay and lesbian, and (2) immigrants. Answers to these questions from both surveys are assessed in one-, two-, and multidimensional (GLM models) analyses. The results show consistent patterns in the analyses of both survey pairs. (1) PtW surveys yield similar response rates as the face-to-face surveys but tend to appeal to the more highly educated and older sections of the population. (2) Despite the correction of sample composition by post-stratification, the results reflect different attitudes towards gay and lesbian, and immigrants in the two survey modes. (3) The survey mode has an independent impact on sensitive measurements: in GLM models, the survey mode has a significant effect on how respondents report their attitudes, and overwrites the original demographic correlations.