Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network Model

Abstract

Multi-item scales are widely used in social research. The psychometric characteristics of a scale and the successful use of a scale in research depend in part on item wording. This article demonstrates a method for using natural language processing (NLP) tools to assist with the item development process, by showing that numeric embedding representations of items are useful in predicting the characteristics of a scale. NLP comprises a set of algorithmic techniques for analysing words, phrases, and larger units of written language. We used NLP tools to create and analyse semantic summaries of the item texts for n=386 previously published multi-item scales. Results showed that semantic representations of items connect to scale characteristics such as Cronbach's alpha internal consistency.