Non-unique Records in International Survey Projects: The Need for Extending Data Quality Control

  • Kazimierz Maciek Slomczynski
  • Przemek Powalko Polish Academy of Sciences
  • Tadeusz Krauze Hofstra University
Keywords: Survey Data Quality, Duplicate Records, Rare Events, Non-Random Errors in Survey Data

Abstract

For a given survey data file we define a non-unique record, NUR, as a sequence of all values in a given case (record), which is identical to that of another case in the same dataset. We analyzed 1,721 national surveys in 22 international projects, covering 142 countries and 2.3 million respondents, and found a total of 5,893 NURs concentrated in 162 national surveys, in 17 projects and 80 countries. We show that the probability of the occurrence of any NUR in an average survey sample is exceedingly small, and although NURs constitute a minor fraction of all records, it is unlikely that they are solely the result of random chance. We describe how NURs are distributed across projects, countries, time, modes of data collection, and sampling methods. We demonstrate that NURs diminish data quality and potentially have undesirable effects on the results of statistical analyses. Identifying NURs allows researchers to examine the consequences of their existence in data files. We argue that such records should be flagged in all publically available data archives. We provide a complete list of NURs for all analyzed national surveys.

Author Biographies

Kazimierz Maciek Slomczynski
Professor Emeritus, Department of Sociology OSU, Professor PAN
Przemek Powalko, Polish Academy of Sciences
Computer specialist, Institute of Philosophy and Sociology, Polish Academy of Sciences
Tadeusz Krauze, Hofstra University
Professor, Department of Sociology
Published
2017-04-10
How to Cite
Slomczynski, K. M., Powalko, P., & Krauze, T. (2017). Non-unique Records in International Survey Projects: The Need for Extending Data Quality Control. Survey Research Methods, 11(1), 1-16. https://doi.org/10.18148/srm/2017.v11i1.6557
Section
Articles