Bias and efficiency loss in regression estimates due to duplicated observations: a Monte Carlo simulation

  • Francesco Sarracino National Institute of Statistics of Luxembourg (STATEC) and National Research University Higher School of Economics
  • Malgorzata Mikucka Universit√© Catholique de Louvain and National Research University Higher School of Economics http://orcid.org/0000-0002-9648-0939
Keywords: duplicated observations, estimation bias, Monte Carlo simulation, inference

Abstract

Recent studies documented that survey data contain duplicate records. We assess how duplicate records affect regression estimates, and we evaluate the effectiveness of solutions to deal with duplicate records. Results show that the chances of obtaining unbiased estimates when data contain 40 doublets (about 5% of the sample) range between 3.5% and 11.5% depending on the distribution of duplicates. If 7 quintuplets are present in the data (2% of the sample), then the probability of obtaining biased estimates ranges between 11% and 20%. Weighting the duplicate records by the inverse of their multiplicity, or dropping superfluous duplicates outperform other solutions in all considered scenarios. Our results illustrate the risk of using data in presence of duplicate records and call for further research on strategies to analyze affected data.

Author Biographies

Francesco Sarracino, National Institute of Statistics of Luxembourg (STATEC) and National Research University Higher School of Economics
Research division
Malgorzata Mikucka, Université Catholique de Louvain and National Research University Higher School of Economics
Researcher at Centre for Demographic Research
Published
2017-04-10
How to Cite
Sarracino, F., & Mikucka, M. (2017). Bias and efficiency loss in regression estimates due to duplicated observations: a Monte Carlo simulation. Survey Research Methods, 11(1), 17-44. https://doi.org/10.18148/srm/2017.v11i1.7149
Section
Articles