Dealing With Inconclusive Findings in Survey Research: The Case of Plain-Language Question Comprehensibility

Abstract

This paper investigates whether translating survey questions into plain language (PL) improvescomprehensibility and response quality in web surveys compared with standard language (SL).Contrary to theoretical expectations and prior research, results from Study I show that PL didnot enhance comprehensibility and was in some cases associated with lower response qualitythan SL. In Study II, we conducted supplementary analyses to explore whether inconclusivefindings from Study I were due to participant selection, the choice of measure, or the designof the experimental stimuli, with survey experts evaluating the items for comprehensibilityand comparability. The results indicate that the PL translations did not produce meaningfuldifferences in comprehensibility, suggesting that the translation process resulted in stimuli thatdid not function as intended. Therefore, we advocate establishing standards and guidelines fortranslating survey questions into PL to ensure more consistent and effective translation outcomesand performance. More broadly, the paper demonstrates the value of supplementary analyses indisentangling substantive and methodological factors behind inconclusive results.