Memory Effects in Repeated Survey Questions: Reviving the Empirical Investigation of the Independent Measurements Assumption

Abstract

It is common to repeat survey questions in the social sciences, for example to estimate test-retest reliability or in pretest-posttest experimental designs. An underlying assumption is that the repetition of questions leads to independent measurements. Critics point to respondents’ memory as a source of bias for the resulting estimates. Yet there is little empirical evidence showing how large memory effects are within the same survey and none showing whether memory effects can be decreased through purposeful intervention during a survey. We aim to address both of these points based on data from a lab-based web survey containing an experiment. We repeated one of the initial questions at the end of the survey (on average 127 questions later) and asked respondents if they recall their previous answer and to reproduce it. Furthermore, we compared respondents’ memory of previously given responses between two experimental groups: A control group, where regular survey questions were asked in between repetitions and a treatment group which, additionally, received a memory interference task aimed at decreasing memory. We found that, after an average 20-minute interval, 60% of the respondents were able to correctly reproduce their previous answer, of which we estimated 17% to do so due to memory. We did not observe a decrease in memory as time intervals between repetitions become longer. This indicates a serious challenge to using repeated questions within the same survey. Moreover, the tested memory interference task did not reduce respondents’ recall of their previously given answer or the memory effect.