Estimation of the effects of measurement characteristics on the quality of survey questions

Abstract

When designing questionnaires, many choices have to be made. Because the consequences of these choices for the quality of the questions are largely unknown, it has often been said that designing a questionnaire is an art. To make it a more scientific activity we need to know more about the consequences of these choices. In order to further such an approach we have: 1. made an inventory of the choices to be made when designing survey questions and created a code book to transform these question characteristics into the independent variables for explaining quality of survey questions; 2. assembled a large set of studies that use Multi-trait Multi-method experiments to estimate the reliability and validity of questions 3. carried out a meta-analysis that relates these question characteristics to the reliability and validity estimates of the questions. On the basis of the results of these efforts we have constructed a database. This data base contains at present 1023 measurement instruments based on 87 experiments conducted on random samples from sometimes regional but mostly national samples of 300 to 2000 respondents. The database contains information on studies of reliability and validity of survey questions formulated in three different languages: English, German and Dutch. The purpose of this study was to generate cross national generalizations of the findings published so far drawn from national studies. This analysis provides a quantitative estimate of the effects of the different choices on the reliability, validity and the method effects.