Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach

Abstract

Measurement errors can have profound effects on statistical relationships, and better information on the quality of measures seems needed. This study uses a new technology—structural modeling of data from special supplements to regular surveys—to generate estimates of construct validity, method effects (a major source of correlated error), and residual error (mainly random error) for a broad set of measures obtained from five national surveys and an organizational survey (total respondents = 7,706). Analysis of these estimates suggested that a typical survey item, when administered by a respected survey organization to a general population sample, can be expected to yield 50–83 percent valid variance, 0–7 percent method effects variance, and 14–48 percent residual variance. Multivariate analysis showed that over two-thirds of the variation in measurement quality could be explained by 13 survey design characteristics; characteristics of respondents explained a small additional portion. Results provide: (a) information on design conditions associated with better (or worse) measurement quality, (b) empirically based suggestions for improving measurement quality in future surveys, and (c) a set of coefficients for predicting the quality of measures not studied here.