Automatic Coding of Open-ended Questions into Multiple Classes: Whether and How to Use Double Coded Data

  • Zhoushanyue He University of Waterloo
  • Matthias Schonlau University of Waterloo
Keywords: Open-ended question, Double coding, Text coding, Text classification, Statistical learning, Machine learning

Abstract

Responses to open-ended questions in surveys are usually coded into pre-specified classes, manually or automatically using a statistical learning algorithm. Automatic coding of open-ended responses relies on a set of manually coded responses, based on which a statistical learning model is fitted. In this paper, we investigate whether and how double coding can help improve the automatic classification of open-ended responses. We evaluate four strategies for training the statistical algorithm on double coded data, using experiments on simulated and real data. We find that, when the data are already double-coded (i.e. double coding does not incur additional costs), double coding where an expert resolves intercoder disagreement leads to the greatest classification accuracy. However, when we have a fixed budget for manually coding, single coding is preferable if the coding error rate is anticipated to be less than about 35% to 45%.
Published
2020-08-10
How to Cite
He, Z., & Schonlau, M. (2020). Automatic Coding of Open-ended Questions into Multiple Classes: Whether and How to Use Double Coded Data. Survey Research Methods, 14(3), 267-287. https://doi.org/10.18148/srm/2020.v14i3.7639
Section
Articles