Non-unique Records in International Survey Projects: The Need for Extending Data Quality Control

Kazimierz Maciek Slomczynski, Przemek Powalko, Tadeusz Krauze

Abstract


For a given survey data file we define a non-unique record, NUR, as a sequence of all values
in a given case (record), which is identical to that of another case in the same dataset. We
analyzed 1,721 national surveys in 22 international projects, covering 142 countries and 2.3
million respondents, and found a total of 5,893 NURs concentrated in 162 national surveys, in
17 projects and 80 countries. We show that the probability of the occurrence of any NUR in
an average survey sample is exceedingly small, and although NURs constitute a minor fraction
of all records, it is unlikely that they are solely the result of random chance. We describe how
NURs are distributed across projects, countries, time, modes of data collection, and sampling
methods. We demonstrate that NURs diminish data quality and potentially have undesirable
effects on the results of statistical analyses. Identifying NURs allows researchers to examine
the consequences of their existence in data files. We argue that such records should be flagged
in all publically available data archives. We provide a complete list of NURs for all analyzed
national surveys.

Keywords


Survey Data Quality, Duplicate Records, Rare Events, Non-Random Errors in Survey Data

Full Text:

PDF


DOI: http://dx.doi.org/10.18148/srm/2017.v11i1.6557

Copyright (c) 2017 Kazimierz Maciek Slomczynski, Przemek Powalko, Tadeusz Krauze

Hosted by the Library of the University of Konstanz