Bias and efficiency loss in regression estimates due to duplicated observations: a Monte Carlo simulation

Authors

  • Francesco Sarracino National Institute of Statistics of Luxembourg (STATEC) and National Research University Higher School of Economics
  • Malgorzata Mikucka Université Catholique de Louvain and National Research University Higher School of Economics http://orcid.org/0000-0002-9648-0939

DOI:

https://doi.org/10.18148/srm/2017.v11i1.7149

Keywords:

duplicated observations, estimation bias, Monte Carlo simulation, inference

Abstract

Recent studies documented that survey data contain duplicate records. We assess how duplicate records affect regression estimates, and we evaluate the effectiveness of solutions to deal with duplicate records. Results show that the chances of obtaining unbiased estimates when data contain 40 doublets (about 5% of the sample) range between 3.5% and 11.5% depending on the distribution of duplicates. If 7 quintuplets are present in the data (2% of the sample), then the probability of obtaining biased estimates ranges between 11% and 20%. Weighting the duplicate records by the inverse of their multiplicity, or dropping superfluous duplicates outperform other solutions in all considered scenarios. Our results illustrate the risk of using data in presence of duplicate records and call for further research on strategies to analyze affected data.

Author Biographies

Francesco Sarracino, National Institute of Statistics of Luxembourg (STATEC) and National Research University Higher School of Economics

Research division

Malgorzata Mikucka, Université Catholique de Louvain and National Research University Higher School of Economics

Researcher at Centre for Demographic Research

Downloads

Additional Files

Published

2017-04-10

How to Cite

Sarracino, F., & Mikucka, M. (2017). Bias and efficiency loss in regression estimates due to duplicated observations: a Monte Carlo simulation. Survey Research Methods, 11(1), 17–44. https://doi.org/10.18148/srm/2017.v11i1.7149

Issue

Section

Articles