Shallowly accurate but deeply confused – how language models deal with antonyms

Abstract

Antonymic adjectives are subject to a variety of asymmetries regarding pragmatic inferences. The Inference Towards the Antonym (Horn, 1989; Krifka, 2007; Ruytenbeek et al., 2017; Gotzner et al., 2018) in particular, consists in deriving the antonym of an adjective A when encountering its negation (not A). Within a given antonymic pair, this inference is sup-posed to apply to a greater extent to negated positive adjective, as opposed to negated negative adjectives. This is especially true when the latter is morphologically transparent. In this pa-per, we test if recent Large Language Models capture this contrast using different probing methods. We conclude that some but not all models exhibit a contrast between positive and negative adjectives regarding the target inference, although (i) the observed contrasts are not readily interpretable at the level of word processing (ii) part of it may be explained by frequency differences (iii) more general expectations about the models’ behavior regarding antonymic ad-jectives (parsing, reversing effect of negation) are not met. This casts doubt on the ability of such models to abstractly encode the concept of antonymy.