Testing Bayesian measures of relevance in discourse

Abstract

Despite their popularity among linguists, categorical notions of relevance based on partial answerhood have well-known problems. Gradient information-theoretic measures of utility adopted in cognitive modeling, on the other hand, have not been tested as measures of relevance in discourse. We begin to fill this gap by experimentally evaluating two gradient measures of question under discussion (QUD) relevance in question-answer pairs in comparison to the categorical theory: entropy reduction, which measures the degree to which an answer decreases uncertainty about the resolution of the QUD; and KL divergence, which measures the degree to which an answer changes the probability distribution over the alternatives. Our experiments provide decisive evidence against the categorical theory of relevance, but do not give strong support to any one gradient measure. Both KL divergence and entropy reduction have systematic failure modes, and are less predictive of relevance judgments than comparatively unmotivated measures like the difference between prior and posterior. We outline several criteria for an adequate gradient theory of relevance, and identify candidate measures for future investigation.