Responding as Expected? The Effects of Survey Mode on Estimates of Sensitive Attitudes in Self-Completion and Face-to-Face Interviews of the European Social Survey

1 Introduction

In recent decades, deploying surveys that employ interviewer-assisted face-to-face data collection methods has become increasingly challenging due to declining response rates, the rising costs of maintaining and managing interviewers’ networks, and decreasing data quality (Groves et al., 2009). The proliferation of self-completion (SC) online surveys is one of the main consequences of this process. Compared to face-to-face (F2F) surveys, online surveys are generally easier, faster, and cheaper to conduct; however, from a methodological perspective, online data collection has important weaknesses. These include coverage errors, high self-selection bias, and the difficulty of applying probability sampling (Hox et al., 2017; Klausch et al., 2017; Tourangeau et al., 2013). Combining different data collection methods in mixed-mode surveys (MMS) can produce more accurate results by increasing sampling coverage and reducing bias (Atkeson et al., 2014; Revilla, 2010; Wolf et al., 2021). MMS is becoming increasingly popular in the social sciences (De Leeuw, 2018), and a growing body of literature addresses the related methodological challenges and their potential solutions (for a comprehensive overview, see Shouten, 2022; or Coffey et al., 2024).

This study addresses the critical question of how different survey modes—interviewer-assisted face-to-face, and self-completion methods—impact the quality of data, particularly in surveys involving sensitive topics. Sensitive questions, especially those that inquire about personal or politically charged topics, are prone to mode effects whereby the presence or absence of an interviewer can influence respondents’ willingness to provide honest answers. Our research specifically examines the extent to which mode effects alter survey outcomes, aiming to contribute to the development of reliable methods for generating accurate reflections of public opinion.

A recent experiment with self-completion MMS was conducted by the European Social Survey (ESS). The Push-to-Web (PtW) survey experiment combined online self-completion with a paper-based self-completion questionnaire and applied probability-based sampling using postal services (Dillman, 2017). The first self-completion PtW survey experiment was inspired by the COVID-19 outbreak and initiated and funded by the ESS ERIC (European Research Infrastructure) in three countries: Austria, Hungary and Serbia in December 2020 (Messing et al., 2022). The main aim of the experiment was to assess the extent to which face-to-face surveys can be replaced by self-administered surveys (i.e., concerning response rates and costs). Later, as a result of the experiment, the PtW method was applied in several countries during the tenth round of the ESS in 2021, when face-to-face data collection was hindered by COVID-19 safety regulations or deemed too risky. A second self-completion PtW survey was conducted in Hungary in Autumn 2022 using the ESS Round 10 full questionnaire.

Even when using identical sampling strategies and questionnaires, self-completion (SC) and interviewer-administered face-to-face (F2F) surveys may introduce biases inherent to the survey mode. The mode effect comprises three types of biases caused or indirectly influenced by the mode of data collection: (1) coverage error, (2) non-response error, and (3) measurement error (Jans, 2008). Coverage error occurs when the population included in the sample does not accurately represent the target population of the survey (Mulry, 2008). Mixed-mode research, however, can intentionally incorporate the mode selection effect as part of its design, especially when using mixed modes to enhance response rates from difficult-to-reach social groups (Hox et al., 2017). Non-response error refers to the difference between the hypothetical estimate obtained from the entire attempted sample and that obtained from the successfully completed sample (Loosveldt, 2008). Measurement error is the observation gap between the ideal measurement and the responses actually obtained (Tourangeau, 2017; Groves et al., 2009). There is a pressing need to assess and quantify the impact of survey mode on data quality and improve the understanding of the role of each bias that collectively comprises the mode effect (Jäckle et al., 2010; Hope et al., 2022). One possible solution is to apply the “reference survey approach” by comparing data from self-completion MMS with data from a comparable single-mode interviewer-assisted survey. This comparison allows for the separation and assessment of self-selection bias (differences in the types of respondents choosing different modes) and measurement error (Vannieuwenhuyze, 2010). However, one problem with the reference survey approach is the assumption of representativeness, which may arise from treating the reference (interviewer-assisted single-mode) survey as identical to self-completion MMS with respect to the target population, sampling, selection, and other factors. Considering the issue of differential selection, multigroup analysis is frequently used to assess and correct for mode measurement error. Controlling for selection bias is possible using common background variables or propensity scores, together with advanced statistical models (e.g., multigroup confirmatory factor analysis—MCFA (Hox et al., 2017)). Another noteworthy method for evaluating mode effects is the counterfactual approach, which is typically employed in studies where two survey modes are applied in parallel to randomly assigned subsamples. The key idea is to predict how each respondent would have answered had they received the questionnaire in the other mode. Since each person is observed in only one mode, this approach uses modern imputation techniques to estimate the unobserved, counterfactual responses. The resulting dataset allows for the direct comparison of mode effects and supports both the diagnosis and adjustment of mode-related measurement error (Klausch et al., 2015; Kolenikov and Kennedy, 2014; Morgan and Winship, 2015).

What complicates the understanding of the mode effect further—especially in relation to the measurement of socially sensitive attitudes—is that these factors may manifest differently across social environments. In contexts where strong and stigmatising social norms prevail, both measurement error and non-response error tend to be higher than in open and tolerant societies. Additionally, technological development and coverage within a given social context significantly influence the extent of coverage error. Studies of mode effect are generally unable to directly distinguish non-response and measurement error in survey estimates because non-response and measurement error tend to reinforce each other in most survey modes (Roberts et al., 2006; Felderer et al., 2019). Survey mode comparison studies typically assess the effect of data collection mode on measurement either by testing (1) the comparability of data collected in different modes or (2) specific hypotheses about the potential causes of differences between modes (Jäckle et al., 2010; Vannieuwenhuyze, 2010). This study focuses on the former type of analysis, comparing measurements of sensitive attitudes across different survey modes.

Many surveys collect sensitive information, including data on illegal or embarrassing activities such as drug use and sexual behaviour, where the mode of data collection is decisive (Tourangeau and Smith, 1996). The literature argues that self-administered surveys support a more intimate, less disruptive survey situation and, thus, more honest responses (lower levels of measurement error), whereas interviewer-administered surveys risk respondents conforming to perceived social norms (Atkeson et al., 2014; Gordoni et al., 2012). For example, self-completion surveys reduced the discrepancy between the number of sexual partners men and women report and increased the proportion of respondents who admitted to having used illicit drugs compared to surveys where these questions were asked by an interviewer (Tourangeau and Smith, 1996). Further, the level of disclosure (the amount of personal information a person is willing to provide to others) is significantly higher for sensitive questions administered through self-completion surveys, while the level of disclosure depends on the degree of sensitivity (Dayan et al., 2009). At the same time, due to the lack of professional support, self-completion surveys may be associated with a higher “do not know” response rate and more item non-responses than face-to-face surveys (Heerwegh and Loosveldt, 2008). Recently, the Pew Research Center conducted studies about the mode effect as part of the American Trend Panel (Keeter, 2015). Respondents answering through a self-completion mode were more likely to give “very unfavourable” ratings to various political figures compared to when they responded to interviewers in person (Keeter, 2015). According to recent evidence on how the presence or absence of interviewers influences survey responses, Hopes et al. (2022) found that respondents performed more complicated tasks and were more motivated to fully consider their answers in interviewer-administered surveys, thus were less likely to select middle response categories. At the same time, they gave socially desirable responses more frequently than in self-administered surveys. A similar trend was confirmed by Atkeson et al. (2014), who provided strong evidence for mode effects due to social desirability and the presence of an interviewer regarding questions related to political attitudes and electoral activity. Other studies have demonstrated that not only the presence of interviewers but also their attitudes may affect the responses provided by respondents. For example, Stefkovics and Sik (2022) demonstrated that interviewers’ level of subjective happiness accounted for a significant portion of the variance in respondents’ reports of subjective happiness.

Based on a review of relevant literature, the following hypotheses were formulated:

• H1: Self-completion surveys and interviewer-assisted face-to-face surveys yield different estimates of sensitive attitudes.

• H2: Models based on self-completion and interviewer-assisted face-to-face survey data result in different predictions of sensitive attitudes.

• H3: The survey mode itself can be a key predictor of diverging responses associated with sensitive attitudes.

This paper aims to make several contributions to the mode effect literature. The analysis focuses on a distinct segment of sensitive questions related to attitudes towards gay and lesbian individuals and immigrants within a socio-political context characterised by democratic backsliding and pronounced polarisation. In this environment, the ruling populist party has openly adopted a hostile stance towards these groups, rendering anti-immigrant and anti-LGBTQ discourse socially acceptable (Bocskor, 2018).

We assess the effects of survey mode on responses to these attitudinal questions by comparing survey designs using data from two interviewer-assisted face-to-face data collection waves and two self-completion PtW data collection phases, which were aligned with the main ESS Round 9 and Round 10 fieldworks using identical sampling strategies and questionnaires. We remain aware that this research design is unsuitable for disentangling the three components of the mode effect (coverage error, non-response error, and measurement error); nonetheless, it is feasible to present the consequences of the combination of these three types of errors on the measurement outcomes.

This paper is structured as follows: Sect. 2 presents the data and methods, evaluating the sample composition of the two survey pairs and comparing it with the population microcensus data (Sect. 2.1). We also describe the model compositions used in the paper (Sect. 2.2). In Sect. 3, the motivation for the paper is explained, while Sect. 4 presents the results. Sect. 4.1, which addresses Hypothesis 1 (H1), displays one-dimensional distributions for both attitudes derived from two sets of face-to-face (F2F) and self-completion (SC) surveys. In Sect. 4.2, we introduce logistic regression models that identify the main drivers of attitude outcomes separately for interviewer-assisted F2F and self-completion survey modes in order to test Hypothesis 2 (H2). In addressing Hypothesis 3 (H3), Sect. 4.3 presents additional regression models that utilise merged data from the two data collection modes, incorporating the survey mode as an additional independent variable. Sect. 5 summarises the results, while Sects. 6 and 7 discuss the study’s limitations and include considerations about further research based on these results.

2 Data and Methods

We analysed two independent pairs of survey datasets that applied identical questionnaires and equivalent sampling designs¹ but different survey modes in Hungary. The European Social Survey (ESS) Rounds 9 and 10 (ESS, 2018, 2020) were conducted via interviewer-assisted F2F surveys in 2019 and 2021 (referred to as ESS Round 9 F2F and ESS Round 10 F2F in this paper), respectively. The two self-completion (SC) surveys were conducted in 2020 and 2022, respectively (referred to as SC 2020 and SC 2022) using the questionnaires from ESS R9 and R10. In the SC surveys, the original ESS questionnaires were adapted to facilitate self-completion mode by following the official ESS specifications. In designing the sample for the SC surveys, the same principles were followed as for the main ESS F2F survey: a multi-stage, stratified, probability-based sampling method was used for all data collection, firmly following the principles set out by the ESS. Individuals in all four surveys were selected by strict probability methods at every stage, and the substitution of non-responding households or individuals was not permitted at any stage (ESS 2022).

In both SC surveys, respondents were invited by letters sent by post to go online and complete a fillable questionnaire that was designed to work on both smaller (i.e., smartphone) and larger (i.e., laptop) screens. Obviously, a significant portion of the population may not have internet access or feel uncomfortable with online questionnaires. To capture this segment of the population, the paper version of the questionnaire was sent with a return envelope to those members of the sample who did not complete the questionnaire online, following two reminders. An unconditional incentive in the form of a general grocery voucher was provided to individuals invited to the survey (€2), and those who completed the survey also received a conditional incentive (€10) (Messing et al., 2022). Respondents to the F2F surveys were not incentivised.

2.1 Sample Compositions

The SC surveys and interviewer-assisted standard F2F ESS surveys produced similar response rates of about 40% ± 4%. Table 1 presents the achieved sample sizes for each phase of data collection.

For both SC surveys, the majority of respondents completed the questionnaire online: only 22 per cent (n = 95) of respondents in SC 2020, and 18 per cent (n = 198) in SC 2022 opted to use a paper questionnaire. Regarding the unweighted, basic demographic composition of the samples, the age group 60+ is over-represented for women in the F2F surveys (see Table A1 in the Appendix). We presume that this difference may be a consequence of interviewers’ presence: regarding F2F surveys, older women are easier to reach (more likely to be at home) and are more likely to be responsive to interviewers. Regarding the level of education, the differences in sample composition are even more pronounced (Table A2 in the Appendix). The SC survey—unsurprisingly—reached highly educated women and men to a greater extent, but performed weakly in reaching the less educated population. The difficulty in reaching out to low-educated respondents also holds true for the F2F survey, but to a lesser extent; this outcome aligns with previous literature (Booth et al., 2024; Holtom et al., 2022; Maslovskaya and Lugtig, 2022; Ortmanns and Schneider, 2016). It is evident that the presence (and support) of an interviewer appears to be important for this group of people. For the weighted analysis, differences in sample composition were adjusted using post-stratification weights based on gender, age, and education.

2.2 Model Compositions

For the models in Sect. 3.3, we included variables that the literature identifies as having explanatory value regarding attitudes toward immigrants and gay and lesbian individuals. The explanatory variables can be grouped into three thematic sets. The first set of variables includes demographic characteristics, such as gender, age, and level of education (Cowling et al., 2019). The second set of variables is used to understand political interests and preferences, including interest in politics, satisfaction with democracy, and political orientation. Numerous studies have found a significant relationship between these factors and attitudes toward minorities. (Cohrs and Stelzl, 2010; Graf and Sczesny, 2019; Lewis et al., 2017; Meuleman et al., 2019). Finally, the third set of variables indicates the respondent’s social embeddedness (frequency of meeting with others, interpersonal trust) and feeling of security (physical safety in their neighbourhood). Several studies have found that these variables impact attitudes toward minorities (Pellegrini et al., 2021; Harell et al., 2017; Messing and Ságvári, 2019, 2021). For a description of the explanatory variables, see Table A4 in the Appendix. In the SC data, item non-response values constituted 0%, while in the F2F sample, they ranged from 0% to 5% for the variables we used in the analysis. In an effort to eliminate the mode effect for the explanatory attitude variables, each variable was coded into two categories based on the given response relative to the median value within the corresponding survey mode: (1) median value and below, and (2) more positive opinion than the median value. In the models, the former (1) serves as the reference category.

3 When the Trend Breaks: Survey Mode and the Measurement Gap in Attitudes

Figs. 1 and 2 illustrate the time series for attitudes towards gay and lesbian people (Fig. 1) and towards immigrants from poorer countries outside Europe (Fig. 2) in the European Social Survey (ESS), and in the corresponding SC surveys. The interval between the two pairs of data collection waves (ESS R9 F2F/SC 2020 and ESS R10 F2F/SC 2022) was relatively brief, spanning barely a year. Thus, we consider these surveys to be directly comparable; however, we cannot completely disregard the possibility that attitudes in Hungary may have shifted slightly during this timeframe.

Fig. 1 Changes in attitude toward gays and lesbians in Hungary between 2002 and 2021 bi-yearly (%) [To what extent do you agree or disagree with each of the following statements: Gay men and lesbians should be free to live their own lives as they wish.]

Fig. 2 Changes in attitudes toward immigrants in Hungary between 2002 and 2020 (%) [To what extent do you think Hungary should allow people from the poorer countries outside Europe to come and live here?]

To assess attitudes toward immigrants, we used a single question that measured acceptance versus refusal of immigrants entering the country from poorer regions outside of Europe (see Fig. 2). Data collected through traditional face-to-face surveys reveal the clear trends in changing attitudes toward immigrants. There was a notable increase in rejections between 2002 and 2006, followed by a gradual decline over the next few years. This was followed by a sharp rise in rejection rates in 2014 and 2016, primarily due to the ongoing migration crisis and the Hungarian government’s fierce anti-migration campaign. Finally, there was a decline and consolidation of rejection rates in 2018 and 2020.

However, the attitudes measured in the Push-to-Web self-completion (SC 2020 and SC 2022) survey did not align with these trends. These surveys indicated more tolerant attitudes toward both groups, reflecting greater acceptance, which cannot be accounted for by time series trends. Since we cannot interpret the outlier values from the SC survey as mere fluctuations, we conclude that the difference is likely due to the method of survey administration. The motivation behind the analysis we describe was thus to analyse more carefully how the mode of the survey impacted responses to these sensitive attitudinal questions.

4 Results

First, Sect. 4.1 includes a one-dimensional analysis to test H1, which examines the differences in responses concerning sensitive attitudes. Sects. 4.2 and 4.3 introduce regression models to evaluate H2 and H3. We begin by predicting sensitive attitudes across different survey modes separately. Following this, we present models that help assess whether the survey mode itself is a significant predictor of responses, independent of other variables that have previously been shown to influence attitudes.

4.1 Differences in Estimates of Sensitive Attitudes as Measured by Self-Completion and Face-to-Face Survey Modes

The results of the one-dimensional analysis used to test H1 are presented in Tables 2 and 3. Regarding attitudes towards gay and lesbian people, the greatest differences are observed in the “agree strongly” and “agree” categories for both survey pairs. Particularly, for the “agree strongly” response, the difference between F2F and SC surveys for both survey pairs is particularly large (13% and 13%, respectively). It is important to note that item non-response (“don’t know” and refusal) was low (virtually non-existent) for both SC surveys, and also was not high for the F2F surveys.

The low p-values (p = 0.000, p = 0.012) suggest the significant influence of the mode of the survey on attitudes towards gay and lesbian individuals.

Table 3 presents the distributions of attitudes towards third-country national (TCN) immigrants, defined as people “from poorer countries outside Europe” in the ESS questionnaire. The most notable change occurs in the “allow none” category, where there is a significant difference between responses to ESS R9 F2F and SC 2020 (a −21% difference) and between ESS R10 F2F and SC 2022 (−23%). These results suggest a radical difference regarding having a more open stance on immigration between the two survey modes. Item non-response (“don’t know” and refusal to answer) was higher for the interviewer-assisted survey modes.

Again, low p-values (p = 0.000, p = 0.003) refer to the significant influence of the mode of the survey on attitudes towards immigrants.

For both survey pairs, we observe significant and similar differences that may be related to the mode of the survey: the attitudes measured by self-completion surveys are significantly more tolerant than those measured by interviewer-assisted surveys. While interviewer-assisted classical surveys present an image of an extremely hostile society (in European comparison), self-completion surveys reveal a higher level of tolerance in Hungary. Thus, we confirm H1. It is likely that differences in attitudes, as measured by the two survey modes, represent the phenomenon of adaptation to mainstream norms in the case of F2F versus SC surveys. In Hungary, the mainstream political discourse promotes intolerance towards these groups (Gera, 2023; Bocskor, 2018).

4.2 Predicting Sensitive Attitudes in Different Survey Modes

Table 4 presents two explanatory models of the most tolerant attitudes towards gays and lesbians in Hungary, based on two different survey data collection modes. (Model 1: ESS R10 F2F; Model 2: SC 2022). The table presents the odds ratios for various explanatory variables, including education, age, gender, satisfaction with democracy, interpersonal trust, interest in politics, social interactions, feeling of safety, and political orientation. The results highlight distinct patterns among the two data collection modes. In Model 1 (F2F), both interest in politics and right-leaning political orientation are associated with a lower probability of expressing positive attitudes toward gay and lesbian people, suggesting that political disengagement and conservative views contribute to less supportive attitudes. Conversely, maintaining an active social life appears to foster more open and accepting attitudes. In Model 2 (SC), dissatisfaction with democracy and right-wing political orientation are linked to less openness, while a positive feeling of safety has a reverse effect. It is interesting that gender and education do not seem to be associated with attitudes toward gay and lesbian people in any of the survey modes.

Table 5 presents two models using the same design as above to explain the negative attitudes towards immigrants in Hungary, again comparing the results from the face-to-face (ESS R10 F2F) and self-completion (SC 2022) surveys. Overall, these findings demonstrate that while some predictors, such as interpersonal trust and political orientation, are consistent across modes, others—like education, gender, and perceived safety—exhibit distinct patterns, underscoring the influence of survey methodology on the expression of social attitudes.

Interpersonal trust emerges as a strong and significant factor in both models, with higher trust levels consistently associated with a lower probability of expressing negative attitudes. Interestingly, the role of perceived safety diverges between the two modes. In the Model 1 (F2F), feeling safe significantly correlates with more negative attitudes towards immigrants, whereas in Model 2 (SC) it is linked to reduced hostility. This is not the case with political orientation, which consistently predicts negative attitudes in both models, with right-leaning individuals significantly more likely to oppose immigration.

Based on the results of the explanatory models, we observe both similarities and notable differences across the two survey modes. Several key predictors—such as education, age, and interpersonal trust—show effects in the same direction, although the strength and statistical significance of these vary between modes. This pattern suggests that while core attitudinal drivers may remain consistent, their salience or detectability may shift depending on how the data is collected. One notable divergence is the effect of the feeling of safety, which reverses direction and remains significant in both models. This could reflect genuine differences in how respondents process and report feelings of safety, depending on the mode; however, prior research suggests that this item may itself be sensitive to mode effects.

Overall, these findings provide partial support for our initial hypothesis (H2): while the survey mode may not drastically alter the direction of associations between most of the predictors and attitudes, it can affect the magnitude, statistical significance, and interpretation of these relationships. This justifies a closer examination of the direct effect of the data collection mode on the estimated outcomes. Given the inconsistency in relationships between attitudes and explanatory variables across survey modes, we proceed to examine the direct impact of data collection mode on the estimates.

4.3 The Effect of Survey Modes

In this section, in agreement with H3, we move on to identify the impact of the survey mode on the measurement of attitudes. To this end, the survey pair analysed in the previous section (ESS R10 F2F and SC 2022) are pooled into a single dataset. Table 6 shows the model results explaining attitudes toward gay and lesbian people, and the same models are presented in Table 7 for attitudes toward migration. In both tables, the first model (Model 1) includes the same socio-demographic and attitudinal explanatory variables as in the previous section, but for the pooled dataset. In Model 2, we include an additional dummy variable for survey mode: self-completion versus face-to-face. In Model 2, all other variables are the same as in Model 1.

Regarding Model 1 in Table 6, apart from the very minor effects of age and education, satisfaction with democracy and interest in politics are the most important explanatory variables. The direction of the relationship is the same as in the previous section. However, when survey mode is included (Model 2), it becomes the most powerful predictor (OR = 3.49, p < 0.001) in explaining positive attitudes toward gays and lesbians.

With regard to attitudes towards immigrants (Table 7), the two models yield broadly consistent results in terms of the direction of most explanatory variables. However, when the dummy variable for survey mode is added in Model 2, it emerges as the most powerful explanatory factor (OR = 0.24, p < 0.001), even when all other variables remain constant. This mirrors the findings for attitudes toward gay and lesbian people, where the inclusion of the mode variable (Table 6, Model 2) significantly improves model fit, showing a strong association in the expected direction.

Taken together, these results provide support for our third hypothesis (H3): survey mode is a strong and consistent predictor of responses to sensitive questions in Hungary.

5 Summary of Results and Conclusion

The distributions of attitudes towards gay and lesbian people and immigrants differ considerably, even in the one-dimensional tables. The estimates from the self-completion surveys suggest a more tolerant society compared to those from the face-to-face surveys (Sect. 4.1). The analysis of the mode effect across two independent survey pairs (Sect. 4.2) revealed that, depending on the survey mode, the binary regression models yielded somewhat different explanations. Nonetheless, the most important and robust finding emerged when the survey mode was incorporated into the explanatory model (Sect. 4.3). These models confirmed that the mode has an independent and particularly strong impact on both attitudes. After accounting for all socio-demographic and other explanatory variables, the deployment of the self-completion mode on its own appeared to increase the likelihood of expressing a positive attitude towards gay and lesbian people, while also decreasing the likelihood of exhibiting more intolerant attitudes towards immigrants.

6 Limitations

There are some important limitations to our conclusions. We have identified a mode effect related to questions that are significant in the public discourse in Hungary, a country marked by democratic regression and an illiberal populist government. The Hungarian government constantly prioritises polarising identity issues, such as immigration and LGBTQ+ rights, presenting the associated groups of individuals, along with NGOs that support them, as threats to the long-term integrity and survival of the ‘nation’. (Cianetti et al., 2018; Krekó and Enyedi, 2018; Shering and Szombati, 2020).

Due to this political framework, it is reasonable to assume that the interview situation influences respondents’ answers, particularly regarding these topics. Consequently, the likelihood of respondents adjusting to perceived social norms and/or the direction of the current dominant narratives is particularly pronounced. It is also possible that in other countries where the dominant narrative differs, we may find a reverse or negligible influence of survey mode on sensitive questions. Since there is no comparable data collection in other ESS countries, we could not analyse these aspects. However, several countries changed the mode of the survey to self-completion for the main ESS R10 due to COVID-related regulations: Austria, Cyprus, Germany, Latvia, Poland, Serbia, Spain, and Sweden (Figure A1 in the appendix). It is worthwhile, therefore, to look at the time series of their data. A quick glance at them reveals that the responses regarding attitudes do not align with the time series. Generally speaking, the self-completed ESS R10 registered more intolerant attitudes towards gays and lesbians, as well as immigrants. The difference is especially marked in countries where social norms concerning homosexuality and ethnic homogeneity are strong. This suggests that mode effects could also be present in other social contexts or concerning other, less politicised types of questions.

Another limitation of our analysis concerns the selection of questions: we intentionally selected highly sensitive ones that are most likely to be prone to the mode effect. A quick comparison of responses to other variables reveals that the effect of the mode on responses is observable for most questions, although not to the same extent. Table A3 (Appendix), which compares the distribution of responses to a larger set of questions in the ESS R9 F2F with SC 2020 and ESS R10 F2F with SC 2022, suggests that the survey mode affects the responses to a number of questions frequently deployed by ESS users, including those on subjective health, happiness, feeling of safety and interest in politics. More research is needed to disentangle the extent of the mode effect on the various variables.

Lastly, it should be noted that data collection for the two survey pairs was not conducted in parallel. While we do not believe that there were any significant changes in attitudes during the period that would have altered them in a meaningful way, we cannot rule out the possibility that this might have occurred.

Finally, survey sample composition was compared to that of the 2006 Microcensus data, which refer to the state of Hungarian society 4–6 years prior. This is undoubtedly a long time, but it is still the most reliable data on the demographic composition of the population.

7 Discussion

1 Supplementary Information

Funding

This research was supported by the European Union’s INFRA4NextGen programme (Grant Agreement ID: 101131118) and by the NRDI ADVANCED “OctoSense: A Multi-Modal Exploration of Social Behavior Using Smartphone Trace Data” project (Grant Agreement ID: 153009).

References

Atkeson, L. R., Adams, A. N., & Alvarez, R. M. (2014). Nonresponse and mode effects in self- and interviewer-administered surveys. Political Analysis, 22(3), 304–320. https://doi.org/10.1093/pan/mpt049. a, b, c

Bocskor, Á. (2018). Anti-immigration discourses in Hungary during the ‘crisis’ year: the Orbán government’s ‘national consultation’ campaign of 2015. Sociology, 52(3), 551–568. https://doi.org/10.1177/0038038518762081. a, b

Booth, C., Wong, E., Brown, M., & Fitzsimons, E. (2024). Evaluating the effect of monetary incentives on web survey response rates in the UK millennium cohort study. Survey Research Methods, 18(1), 47–58. https://doi.org/10.18148/srm/2024.v18i1.8210. →

Cianetti, L., Dawson, J., & Hanley, S. (2018). Rethinking ‘Democratic Backsliding’ in Central and Eastern Europe – Looking beyond Hungary and Poland. East European Politics 34(3):243–56. https://doi.org/10.1080/21599165.2018.1491401. →

Coffey, S., Maslovskaya, O., & McPhee, C. (2024). Recent innovations and advances in mixed-mode surveys. Journal of Survey Statistics and Methodology, 12(3), 507–531. https://doi.org/10.1093/jssam/smae025. →

Cohrs, J. C., & Stelzl, M. (2010). How ideological attitudes predict host society members’ attitudes toward immigrants: exploring cross-national differences: RWA, SDO, and attitudes toward immigrants. Journal of Social Issues, 66(4), 673–694. https://doi.org/10.1111/j.1540-4560.2010.01670.x. →

Cowling, M. M., Anderson, J. R., & Ferguson, R. (2019). Prejudice-relevant correlates of attitudes towards refugees: a meta-analysis. Journal of Refugee Studies, 32(3), 502–524. https://doi.org/10.1093/jrs/fey062. →

Dayan, Y., Paine, C. S., & Johnson, A. J. (2009). Responding to sensitive questions in surveys: a comparison of results from online panels, face to face and self-completion interviews. World Association for Public Opinion Research (WAPOR). http://www.ipsos-mori.com/Assets/Docs/Publications/Ops_RMC_Responding_Sensitive_Questions_08_03_10.pdf  →

DeLeeuw, E. D. (2018). Mixed-mode: past, present, and future. Survey Research Methods, 12, 75–89. https://doi.org/10.18148/SRM/2018.V12I2.7402. →

Dillman, D. A. (2017) The promise and challenge of pushing respondents to the Web in mixed-mode surveys. Survey Methodology, 43(1), 3–30. http://www.statcan.gc.ca/pub/12-001-x/2017001/article/14836-eng.htm. →

ESS (2018). European Social Survey (ESS) Round 9 data Data file edition 3.1 [Dataset]. Sikt—Norwegian Agency for Shared Services in Education and Research Data Archive and distributor of ESS data for ESS ERIC. https://doi.org/10.21338/NSD-ESS10-2020. →

ESS (2020). European Social Survey Round 10 Data Data file edition 1.2. [Dataset]. Sikt—Norwegian Agency for Shared Services in Education and Research Data Archive and distributor of ESS data for ESS ERIC. https://doi.org/10.21338/NSD-ESS10-2020. →

European Social Survey (ESS) (2022). European social surveyround 11 sampling guidelines: principles and implementation. https://www.europeansocialsurvey.org/sites/default/files/2023-06/Round-11-ESS-sampling-guidelines.pdf →

European Social Survey European Research Infrastructure (ESS ERIC) (2023). ESS10—Integrated file, edition 3.2 [Dataset]. Sikt—Norwegian Agency for Shared Services in Education and Research. https://doi.org/10.21338/ESS10E03_2. 

Felderer, B., Kirchner, A., & Kreuter, F. (2019). The effect of survey mode on data quality: disentangling nonresponse and measurement error bias. Journal of Official Statistics, 35(1), 93–115. https://doi.org/10.2478/jos-2019-0005. →

Gera, M. (2023). “Here, the Hungarian people will decide how to raise our children”: Populist rhetoric and social categorization in Viktor Orbán’s anti-LGBTQ campaign in Hungary. New Perspectives, 31(2), 104–129. https://doi.org/10.1177/2336825X231164311. →

Gordoni, G., Schmidt, P., & Gordoni, Y. (2012). Measurement Invariance across face-to-face and telephone modes: the case of minority-status collectivistic-oriented groups. International Journal of Public Opinion Research, 24(2), 185–207. https://doi.org/10.1093/ijpor/edq054. →

Graf, S., & Sczesny, S. (2019). Intergroup contact with migrants is linked to support for migrants through attitudes, especially in people who are politically right wing. International Journal of Intercultural Relations, 73, 102–106. https://doi.org/10.1016/j.ijintrel.2019.09.001. →

Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer E., & Tourangeau, R. (2009). Survey Methodology. Hoboken, NJ: Wiley. a, b

Harell, A., Soroka, S., & Iyengar, S. (2017). Locus of Control and Anti-Immigrant Sentiment in Canada, the United States, and the United Kingdom: Locus of Control and Anti-Immigrant Sentiment. Political Psychology 38(2):245–60. https://doi.org/10.1111/pops.12338. →

Heerwegh, D., & Loosveldt, G. (2008). Face-to-face versus web surveying in a high-Internet-coverage population: differences in response quality. Public Opinion Quarterly, 72(5), 836–846. https://doi.org/10.1093/poq/nfn045. →

Holtom, B., Baruch, Y., Aguinis, H., & Ballinger, A. G. (2022). Survey response rates: trends and a validity assessment framework. Human Relations, 75(8), 1560–1584. https://doi.org/10.1177/00187267211070769. →

Hope, S., Campanelli, P., Nicolaas, G., Lynn, P., & Jäckle, A. (2022). The role of the interviewer in producing mode effects: results from a mixed modes experiment comparing face-to-face, telephone and web administration. In Survey research methods (pp. 207–226). https://doi.org/10.18148/SRM/2022.V16I2.7771. a, b

Hox, J., De Leeuw, E., & Klausch, T. (2017). Mixed-mode research: issues in design and analysis. In P. P. Biemer, E. Leeuw, S. Eckman, B. Edwards, F. Kreuter, L. E. Lyberg, N. C. Tucker & B. T. West (Eds.), Total survey error in practice (1st edn., pp. 511–530). Wiley. https://doi.org/10.1002/9781119041702.ch23. a, b, c

Jäckle, A., Roberts, C., & Lynn, P. (2010). Assessing the effect of data collection mode on measurement. International Statistical Review, 78(1), 3–20. https://doi.org/10.1111/j.1751-5823.2010.00102.x. a, b

Jans, M. (2008). Mode effect. In Encyclopedia of survey research methods (pp. 476–480). SAGE. https://doi.org/10.4135/9781412963947. →

Keeter, S. From telephone to the web: The challenge of mode of interview effects in public opinion polls. PEW RESEARCH CENTER. https://www.pewresearch.org/methods/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/  a, b

Klausch, T., Hox, J., & Schouten, B. (2015). Selection Error in Single- and Mixed Mode Surveys of the Dutch General Population. Journal of the Royal Statistical Society Series A: Statistics in Society 178(4):945–61. https://doi.org/10.1111/rssa.12102. →

Klausch, T., Schouten, B., & Hox, J. J. (2017). Evaluating bias of sequential mixed-mode designs against benchmark surveys. Sociological Methods & Research, 46(3), 456–489. https://doi.org/10.1177/0049124115585362. →

Kolenikov, S., & Kennedy, C. (2014). Evaluating three approaches to statistically adjust for mode effects. Journal of Survey Statistics and Methodology, 2(2), 126–158. https://doi.org/10.1093/jssam/smu004. →

Krekó, P., & Enyedi, Z. (2018). Orbán’s Laboratory of Illiberalism. Journal of Democracy 29(3):39–51. https://doi.org/10.1353/jod.2018.0043. →

Lewis, D. C., Flores, A. R., Haider-Markel, D. P., Miller, P. R., Tadlock, B. L., & Taylor, J. K. (2017). Degrees of acceptance: variation in public attitudes toward segments of the LGBT community. Political Research Quarterly, 70(4), 861–875. https://doi.org/10.1177/1065912917717352. →

Loosveldt, G. (2008). Nonresponse error. In Encyclopedia of survey research methods (pp. 534–536). SAGE. https://doi.org/10.4135/9781412963947. →

Maslovskaya, O., & Lugtig, P. (2022). Representativeness in six waves of cross-national online survey (CRONOS) panel. Journal of the Royal Statistical Society Series A: Statistics in Society, 185(3), 851–871. https://doi.org/10.1111/rssa.12801. →

Messing, V., & Ságvári, B. (2019). Still divided but more open: mapping European attitudes towards migration before and after the migration crisis. Friedrich-Ebert-Stiftung Regional Project “Flight, Migration, Integration in Europe.”. →

Messing, V., & Ságvári, B. (2021). Are anti-immigrant attitudes the Holy Grail of populists? A comparative analysis of attitudes towards immigrants, values, and political populism in Europe. Intersections. East European Journal of Society and Politics, 7(2), 100–127. https://doi.org/10.17356/ieejsp.v7i2.750. →

Messing, V., Ságvári, B., & Szeitl, B. (2022). Webre terelés a személyes lekérdezés alternatívája? : Egy ”push-to-web“ hibrid survey tapasztalatai. Statisztikai Szemle, 100(3), 213–233. https://doi.org/10.20311/stat2022.3.hu0213. a, b

Meuleman, B., Abts, K., Slootmaeckers, K., & Meeusen, C. (2019). Differentiated threat and the genesis of prejudice: group-specific antecedents of homonegativity, islamophobia, anti-semitism, and anti-immigrant attitudes. Social Problems, 66(2), 222–244. https://doi.org/10.1093/socpro/spy002. →

Morgan, S. L., & Winship, C. (2015). Counterfactuals and causal inference: methods and principles for social research (2nd ed.). Cambridge University Press. →

Mulry, M. (2008). Coverage error. In Encyclopedia of survey research methods (pp. 162–166). SAGE. https://doi.org/10.4135/9781412963947. →

Ortmanns, V., & Schneider, S. L. (2016). Can we assess representativeness of cross-national surveys using the education variable? Survey Research Methods, 10(3), 189–210. https://doi.org/10.18148/srm/2016.v10i3.6608. →

Pellegrini, V., De Cristofaro, V., Salvati, M., Giacomantonio, M., & Leone, L. (2021). Social exclusion and anti-immigration attitudes in Europe: the mediating role of interpersonal trust. Social Indicators Research, 155(2), 697–724. https://doi.org/10.1007/s11205-021-02618-6. →

Revilla, M. (2010). Quality in Unimode and mixed-mode designs: a multitrait-multimethod approach. Survey Research Methods, 4, 151–164. https://doi.org/10.18148/SRM/2010.V4I3.4278. →

Roberts, C., Jackle, A. E., & Lynn, P. (2006). Causes of mode effects: separating out interviewer and stimulus effects in comparisons of face-to-face and telephone surveys. AAPOR—ASA Section on Survey Research Methods →

Scheiring, G., & Szombati, K. (2020). From Neoliberal Disembedding to Authoritarian Re-Embedding: The Making of Illiberal Hegemony in Hungary. International Sociology 35(6):721–38. https://doi.org/10.1177/0268580920930591. →

Schouten, B. (2022). Mixed-mode official surveys: design and analysis (1st edn.). Chapman & Hall/CRC. →

Stefkovics, Á., & Sik, E. (2022). What drives happiness? The interviewer’s happiness. Journal of Happiness Studies, 23(6), 2745–2762. https://doi.org/10.1007/s10902-022-00527-0. →

Tourangeau, R., & Smith, T. W. (1996). Asking sensitive questions: the impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60(2), 275. https://doi.org/10.1086/297751. a, b

Tourangeau, R., Conrad, F. G., & Couper, M. (2013). The science of web surveys. Oxford University Press. →

Tourangeau, R. (2017). Mixing Modes: Tradeoffs Among Coverage, Nonresponse, and Measurement Error. Pp. 115–32 in Total Survey Error in Practice, edited by P. P. Biemer, E. Leeuw, S. Eckman, B. Edwards, F. Kreuter, L. E. Lyberg, N. C. Tucker, and B. T. West. Wiley. →

Vannieuwenhuyze, J., Loosveldt, G., & Molenberghs, G. (2010). A method for evaluating mode effects in mixed-mode surveys. Public Opinion Quarterly, 74, 1027–1045. https://doi.org/10.2307/40985415. a, b

Wolf, C., Christmann, P., Gummer, T., Schnaudt, C., & Verhoeven, S. (2021). Conducting general social surveys as self-administered mixed-mode surveys. Public Opinion Quarterly, 85(2), 623–648. https://doi.org/10.1093/poq/nfab039. →