Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs.
Abstract
On the basis of an empirical study of measures of constructs from the cognitive domain, the personality domain, and the domain of affective traits, the authors of this study examine the implications of transient measurement error for the measurement of frequently studied individual differences variables. The authors clarify relevant reliability concepts as they relate to transient error and present a procedure for estimating the coefficient of equivalence and stability (L. J. Cronbach, 1947), the only classical reliability coefficient that assesses all 3 major sources of measurement error (random response, transient, and specific factor errors). The authors conclude that transient error exists in all 3 trait domains and is especially large in the domain of affective traits. Their findings indicate that the nearly universal use of the coefficient of equivalence (Cronbach's alpha; L. J. Cronbach, 1951), which fails to assess transient error, leads to overestimates of reliability and undercorrections for biases due to measurement error.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Single-item scales tend to be unstable and susceptible to random response error.
“A further disadvantage of one-item scales is a tendency to be unstable and vulnerable to random response error (e.g., Nunnally & Bernstein, 1994; Schmidt, Le, & Ilies, 2003).”
From Dissociation and memory fragmentation in post-traumatic stress disorder: An evaluation of the dissociative encoding hypothesis (2012) · cites this paper as Schmidt, Le, & Ilies, 2003
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · see citing works
Development and validation of brief measures of positive and negative affect: The PANAS scales.
Watson, D., Clark, L. A., & Tellegen, A. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.6.1063 · Cited by 32,433 · see citing works
Society and the Adolescent Self-Image
Rosenberg, M. (1965) · doi:10.1515/9781400876136 · Cited by 30,126 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
Self-citationThe development of markers for the Big-Five factor structure
Goldberg, L. R. (1992) · Psychological Assessment · doi:10.1037/1040-3590.4.1.26 · Cited by 5,530 · see citing works
Buss, A. H., & Perry, M. (1992) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.63.3.452 · Cited by 4,693 · see citing works
The Self-Efficacy Scale: Construction and Validation
Sherer, M., Maddux, J. E., Mercandante, B., Prentice-Dunn, S., Jacobs, B., & Rogers, R. W. (1982) · Psychological Reports · doi:10.2466/pr0.1982.51.2.663 · Cited by 4,028 · see citing works
Measurement error in psychological research: Lessons from 26 research scenarios.
Schmidt, F. L., & Hunter, J. E. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.199 · Cited by 567 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Kotov, R., Gamez, W., Schmidt, F., & Watson, D. (2010) · Psychological Bulletin · doi:10.1037/a0020327 · Cited by 2,677 · see citing works
Self-citationThe Mini-IPIP Scales: Tiny-yet-effective measures of the Big Five Factors of Personality.
Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006) · Psychological Assessment · doi:10.1037/1040-3590.18.2.192 · Cited by 2,442 · see citing works
Kirsch, I., Deacon, B. J., Huedo-Medina, T. B., Scoboria, A., Moore, T. J., & Johnson, B. T. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050045 · Cited by 2,442 · see citing works
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718 · see citing works
Hofmann, W., Gawronski, B., Gschwendner, T., Le, H., & Schmitt, M. (2005) · Personality and Social Psychology Bulletin · doi:10.1177/0146167205275613 · Cited by 1,642 · see citing works
Self-citationConstructing validity: New developments in creating objective measuring instruments.
Clark, L. A., & Watson, D. (2019) · Psychological Assessment · doi:10.1037/pas0000626 · Cited by 1,470 · see citing works
An Updated Guideline for Assessing Discriminant Validity
Rönkkö, M., & Cho, E. (2022) · Organizational Research Methods · doi:10.1177/1094428120968614 · Cited by 1,309 · see citing works
Connelly, B. S., & Ones, D. S. (2010) · Psychological Bulletin · doi:10.1037/a0021212 · Cited by 1,059 · see citing works
Rethinking Internal Consistency in Cronbach’s Alpha
Vaske, J. J., Beaman, J., & Sponarski, C. C. (2017) · Leisure Sciences · doi:10.1080/01490400.2015.1127189 · Cited by 856 · see citing works
Sharma, R., Yetton, P., & Crawford, J. (2009) · MIS Quarterly · doi:10.2307/20650305 · Cited by 496 · see citing works
Beal, D. J. (2015) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-032414-111335 · Cited by 436 · see citing works
Shaffer, J. A., DeGeest, D., & Li, A. (2016) · Organizational Research Methods · doi:10.1177/1094428115598239 · Cited by 393 · see citing works
Making Reliability Reliable: A Systematic Approach to Reliability Coefficients
Cho, E. (2016) · Organizational Research Methods · doi:10.1177/1094428116656239 · Cited by 378 · see citing works
Differences among dark triad components: a meta-analytic investigation
Vize, C. E., Lynam, D. R., Collison, K. L., & Miller, J. D. (2018) · Personality Disorders: Theory, Research, and Treatment · doi:10.1037/per0000222 · Cited by 321 · see citing works
SCHMIDT, F. L., SHAFFER, J. A., & OH, I. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00132.x · Cited by 190 · see citing works
Self-citationObtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts
Wiernik, B. M., & Dahlke, J. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919885611 · Cited by 135 · see citing works
Bedard-Gilligan, M., & Zoellner, L. A. (2012) · Memory · doi:10.1080/09658211.2012.655747 · Cited by 116 · see citing works
“A further disadvantage of one-item scales is a tendency to be unstable and vulnerable to random response error (eg, Nunnally & Bernstein, 1994 ; Schmidt, Le, & Ilies, 2003 ).”
VAN IDDEKINGE, C. H., & PLOYHART, R. E. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00133.x · Cited by 101 · see citing works
Abusive supervision, thwarted belongingness, and workplace safety: a group engagement perspective
Yang, L.-Q., Zheng, X., Liu, X., Lu, C.-Q., & Schaubroeck, J. M. (2020) · Journal of Applied Psychology · doi:10.1037/apl0000436 · Cited by 69 · see citing works
Chahoud, M., Chahine, R., Salameh, P., & Sauleau, E. A. (2017) · eNeurologicalSci · doi:10.1016/j.ensci.2017.03.003 · Cited by 64 · see citing works
Aripiprazole monotherapy in the treatment of bipolar disorder: A meta-analysis
Fountoulakis, K. N., Vieta, E., & Schmidt, F. (2011) · Journal of Affective Disorders · doi:10.1016/j.jad.2010.10.018 · Cited by 52 · see citing works
Self-citationSchmidt, F. L. (2017) · Archives of Scientific Psychology · doi:10.1037/arc0000033 · Cited by 45 · see citing works
Self-citationWatson-Singleton, N. N., Walker, J. H., LoParo, D., Mack, S. A., & Kaslow, N. J. (2018) · Mindfulness · doi:10.1007/s12671-017-0776-0 · Cited by 37 · see citing works
Martínez, A., Salgado, J. F., & Lado, M. (2022) · Revista de Psicología del Trabajo y de las Organizaciones · doi:10.5093/jwop2022a16 · Cited by 11 · see citing works
- Authors
- Schmidt, F. L., Le, H., & Ilies, R.
- Affiliations
- University of Iowa (US)
- Venue
- Psychological Methods
- Published
- 2003
- Volume
- 8
- Issue
- 2
- Pages
- 206-224
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2049058583
- Cited by
- 362
- References
- 8
- Citations by year
- 2026: 12 · 2025: 12 · 2024: 14 · 2023: 21 · 2022: 15 · 2021: 23 · 2020: 22 · 2019: 22 · 2018: 28 · 2017: 22 · 2016: 22 · 2015: 19 · 2014: 25 · 2013: 12 · 2012: 13
- FWCI
- 12.53
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Keywords
- measurement error, reliability statistics, Cronbach's alpha, individual differences, personality traits, emotional characteristics, cognitive characteristics
- MeSH terms
- Humans, Individuality, Mathematics, Psychometrics, Research Design, Weights and Measures, Reproducibility of Results, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
