Conflict Deaths in Iraq: A Methodological Critique of the ORB Survey Estimate

Abstract

In September of 2007 ORB, a British opinion polling firm, released an estimate that 1.2 million Iraqis had been killed in the conflict, subsequently lowering its estimate to 1 million. We compare three ORB polls and find important irregularities in ORB's mortality data in four central governorates of Iraq that account for more than 80% of the estimated deaths. These internal validity checks indicate that the ORB mortality data are not credible and would suggest a much lower estimate than ORB has published. We also analyze a number of specific error sources in the poll. Systematic errors, which include non-coverage and measurement errors, mostly point toward overestimation. Variable errors are also substantial but they are difficult to quantify in part due to incomplete disclosure of methodological details by ORB. External validity checks, including comparisons with two much larger and higher quality surveys, reinforce the conclusion that ORB has overestimated the number killed in Iraq by a wide margin. Thus, our paper answers a challenge facing the field of survey methodology, to explain how different surveys have produced such divergent mortality estimates for Iraq.