Rethinking Traditional Methods of Survey Validation
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
It is commonly believed that self-report, survey-based instruments can be used to measure a wide range of psychological attributes, such as self-control, growth mindsets, and grit. Increasingly, such instruments are being used not only for basic research but also for supporting decisions regarding educational policy and accountability. The validity of such instruments is typically investigated using a classic set of methods, including the examination of reliability coefficients, factor or principal components analyses, and correlations between scores on the instrument and other variables. However, these techniques may fall short of providing the kinds of rigorous, potentially falsifying tests of relevant hypotheses commonly expected in scientific research. This point is illustrated via a series of studies in which respondents were presented with survey items deliberately constructed to be uninterpretable, but the application of the aforementioned validation procedures nonetheless returned favorable-appearing results. In part, this disconnect may be traceable to the way in which operationalist modes of thinking in the social sciences have reinforced the perception that attributes do not need to be defined independently of particular sets of testing operations. It is argued that affairs might be improved via greater attention to the manner in which definitions of psychological attributes are articulated and greater openness to treating beliefs about the existence and measurability of psychological attributes as hypotheses rather than assumptions—in other words, as beliefs potentially subject to revision.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Mindset: The New Psychology of Success
(2006) · Choice Reviews Online · doi:10.5860/choice.44-2397 · Cited by 7,804
Grit: perseverance and passion for long-term goals
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.6.1087 · Cited by 6,911 · cited 1×
“This is a common result in the presence of positively and negatively worded items; as a notable example, this finding mirrors the results of the study in which the Grit Scale was introduced (Duckworth et al., 2007), in which two factors composed exclusively of positively coded (e.”
ResultOn the Theory of Scales of Measurement
Stevens, S. S. (1946) · Science · doi:10.1126/science.103.2684.677 · Cited by 4,761
Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007) · Child Development · doi:10.1111/j.1467-8624.2007.00995.x · Cited by 3,784
The theoretical status of latent variables.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003) · Psychological Review · doi:10.1037/0033-295x.110.2.203 · Cited by 1,566
Duckworth, A. L., & Yeager, D. S. (2015) · Educational Researcher · doi:10.3102/0013189x15584327 · Cited by 993 · cited 1×
“Duckworth and Yeager (2015) define self-control as “the regulation of attention, emotion, and behavior when enduringly valued goals conflict with more immediately pleasurable temptations” (p. 239).10 Although this definition is articulate and may be useful for many purposes, one can immediately…”
BackgroundInventing Temperature: Measurement and Scientific Progress
Chang, H. (2004) · doi:10.1093/0195171276.001.0001 · Cited by 883
Wilson, M. (2004) · doi:10.4324/9781410611697 · Cited by 277
Frontiers of Test Validity Theory
Markus, K. A., & Borsboom, D. (2013) · doi:10.4324/9780203501207 · Cited by 248
Validity in Educational & Psychological Assessment
Newton, P. E., & Shaw, S. D. (2014) · doi:10.4135/9781446288856 · Cited by 189
Maul, A. (2012) · Emotion Review · doi:10.1177/1754073912445811 · Cited by 122
Self-citation
Cited by
Thinking twice about sum scores
McNeish, D., & Wolf, M. G. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01398-0 · Cited by 592
Matthews, R. A., Pineault, L., & Hong, Y.-H. (2022) · Journal of Business and Psychology · doi:10.1007/s10869-022-09813-3 · Cited by 462
Worse than measurement error: Consequences of inappropriate latent variable measurement models.
Rhemtulla, M., van Bork, R., & Borsboom, D. (2020) · Psychological Methods · doi:10.1037/met0000220 · Cited by 257
Back to Basics: The Importance of Conceptual Clarification in Psychological Science
Bringmann, L. F., Elmer, T., & Eronen, M. I. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214221096485 · Cited by 221
“However, as Maul (2017) showed, even completely made-up nonsense data can result in a neat one-factor (or two-factor) solution with high loadings and correlations with other variables.”
BackgroundTaves, A. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691619895047 · Cited by 140
“Whatever other evidence of validation is offered, itemlevel validation is necessary in order to ensure that items are understood as researchers intend by the populations to which the instrument is administered (Maul, 2017).”
Background
Metadata
- Authors
- Maul, A.
- Venue
- Measurement: Interdisciplinary Research and Perspectives
- Publisher
- Informa UK Limited
- Published
- 2017-04-03
- Volume
- 15
- Issue
- 2
- Pages
- 51-69
- ISSN
- 1536-6359 (ISSN-L)
- 1536-6367
- OpenAlex
- W2742736991
- Cited by
- 142
- References
- 12
- FWCI
- 12.82
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.