Semi-automated categorization of open-ended questions

Authors

  • Matthias Schonlau University of Waterloo
  • Mick P. Couper University of Michigan

DOI:

https://doi.org/10.18148/srm/2016.v10i2.6213

Keywords:

multinomial boosting, qualitative data, open-ended questions, text mining, uncertainty sampling

Abstract

Text data from open-ended questions in surveys are difficult to analyze and are frequently ignored. Yet open-ended questions are important because they do not constrain respondents’ answer choices. Where open-ended questions are necessary, sometimes multiple human coders hand-code answers into one of several categories. At the same time, computer scientists have made impressive advances in text mining that may allow automation of such coding. Automated algorithms do not achieve an overall accuracy high enough to entirely replace humans. We categorize open-ended questions soliciting narrative responses using text mining for easy-to-categorize answers and humans for the remainder using expected accuracies to guide the choice of the threshold delineating between “easy” and “hard”. Employing multinomial boosting avoids the common practice of converting machine learning “confidence scores” into pseudo-probabilities. This approach is illustrated with examples from open-ended questions related to respondents’ advice to a patient in a hypothetical dilemma, a follow-up probe related to respondents’ perception of disclosure/privacy risk, and from a question on reasons for quitting smoking from a follow-up survey from the Ontario Smoker’s Helpline. Targeting 80% combined accuracy, we found that 54%-80% of the data could be categorized automatically in research surveys.

Author Biographies

Matthias Schonlau, University of Waterloo

Professor , Department of Statistics and Actuarial Science

Mick P. Couper, University of Michigan

Mick P. Couper is a research professor in the Survey Research Center at the Institute for Social Research at the University of Michigan, and in the Joint Program in Survey Methodology at the University of Maryland.

Downloads

Additional Files

Published

2016-08-15

How to Cite

Schonlau, M., & Couper, M. P. (2016). Semi-automated categorization of open-ended questions. Survey Research Methods, 10(2), 143–152. https://doi.org/10.18148/srm/2016.v10i2.6213

Issue

Section

Articles

Most read articles by the same author(s)

1 2 > >> 

Similar Articles

1 2 3 4 5 6 7 8 9 10 > >> 

You may also start an advanced similarity search for this article.