Automatic Coding of Open-ended Questions into Multiple Classes: Whether and How to Use Double Coded Data
Keywords: Open-ended question, Double coding, Text coding, Text classification, Statistical learning, Machine learning
AbstractResponses to open-ended questions in surveys are usually coded into pre-specified classes, manually or automatically using a statistical learning algorithm. Automatic coding of open-ended responses relies on a set of manually coded responses, based on which a statistical learning model is fitted. In this paper, we investigate whether and how double coding can help improve the automatic classification of open-ended responses. We evaluate four strategies for training the statistical algorithm on double coded data, using experiments on simulated and real data. We find that, when the data are already double-coded (i.e. double coding does not incur additional costs), double coding where an expert resolves intercoder disagreement leads to the greatest classification accuracy. However, when we have a fixed budget for manually coding, single coding is preferable if the coding error rate is anticipated to be less than about 35% to 45%.
How to Cite
He, Z., & Schonlau, M. (2020). Automatic Coding of Open-ended Questions into Multiple Classes: Whether and How to Use Double Coded Data. Survey Research Methods, 14(3), 267-287. https://doi.org/10.18148/srm/2020.v14i3.7639
Copyright for articles published in this journal is retained by the authors, with first publication rights granted to the journal. By virtue of their appearance in this open access journal, users can use, reuse and build upon the material published in the journal but only for non-commercial purposes and with proper attribution.