The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Statistically Controlling for Confounding Constructs Is Harder than You Think
Open accessAbstract
Social scientists often seek to demonstrate that a construct has incremental validity over and above other related constructs. However, these claims are typically supported by measurement-level models that fail to consider the effects of measurement (un)reliability. We use intuitive examples, Monte Carlo simulations, and a novel analytical framework to demonstrate that common strategies for establishing incremental construct validity using multiple regression analysis exhibit extremely high Type I error rates under parameter regimes common in many psychological domains. Counterintuitively, we find that error rates are highest--in some cases approaching 100%--when sample sizes are large and reliability is moderate. Our findings suggest that a potentially large proportion of incremental validity claims made in the literature are spurious. We present a web application (http://jakewestfall.org/ivy/) that readers can use to explore the statistical properties of these and other incremental validity arguments. We conclude by reviewing SEM-based statistical approaches that appropriately control the Type I error rate when attempting to establish incremental validity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“meaning that the association between loneliness and mental health outcomes in these studies is very likely to be inflated.16”
From Rapid Systematic Review: The Impact of Social Isolation and Loneliness on the Mental Health of Children and Adolescents in the Context of COVID-19 (2020) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1998) · Cited by 36,049
Structural Equations with Latent Variables.
Clogg, C. C., & Bollen, K. A. (1991) · Contemporary Sociology · doi:10.2307/2072165 · Cited by 17,442
Structural Equations with Latent Variables
Bollen, K. A. (1989) · Wiley series in probability and statistics · doi:10.1002/9781118619179 · Cited by 17,277
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195 · cited 2×
“After all, in any given analysis, there is a simple fact of the matter as to whether or not the unique contribution of one or more variables in a regression is statistically significant when controlling for other variables; what room is there for inferential error? Trouble arises, however, when researchers behave as i…”
BackgroundTHE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META‐ANALYSIS
BARRICK, M. R., & MOUNT, M. K. (1991) · Personnel Psychology · doi:10.1111/j.1744-6570.1991.tb00688.x · Cited by 8,689
Handbook of Personality: Theory and Research
Krueger, R. F., & Johnson, W. (2008) · Cited by 7,884
Implicit social cognition: Attitudes, self-esteem, and stereotypes
Greenwald, A. G., & Banaji, M. R. (1995) · Psychological Review · doi:10.1037/0033-295x.102.1.4 · Cited by 6,682
Handbook of Personality : Theory and Research
Pervin, L. A. (1992) · Cited by 4,543
Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015575 · Cited by 3,257
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200 · cited 1×
“For example, suppose one takes the view that the null hypothesis is almost never exactly true, and that researchers should instead focus on parameter estimation [34,35].”
Background
Research works printed but not linked: 1
- LR Goldberg, 2008, The Eugene-Springfield community sample: Information available from the research participants, ORI Tech Rep, 45
Cited by
Advanced Bayesian Multilevel Modeling with the R Package brms
Bürkner, P.-C. (2018) · The R Journal · doi:10.32614/rj-2018-017 · Cited by 3,015
Loades, M. E., Chatburn, E., Higson-Sweeney, N., Reynolds, S., Shafran, R., Brigden, A., Linney, C., McManus, M. N., Borwick, C., & Crawley, E. (2020) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2020.05.009 · Cited by 2,906
“In cross-sectional studies, likely confounders (eg, adversity, socioeconomic status [SES]) were rarely controlled for, meaning that the association between loneliness and mental health outcomes in these studies is very likely to be inflated.(16) Four longitudinal studies used multi-informant approaches, including self…”
BackgroundChoosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,916
“For example, we and others have demonstrated that under typical measurement conditions, statistical significance patterns observed in fitted regression models are often very unstable and can produce highly misleading conclusions (Shear & Zumbo, 2013; Westfall & Yarkoni, 2016).”
BackgroundSelf-citationThinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data
Rohrer, J. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917745629 · Cited by 1,002
“Westfall and Yarkoni (2016) assessed how the false positive rate for an effect is affected by measurement error of covariates that are being controlled for.”
BackgroundInfluentialShort and extra-short forms of the Big Five Inventory–2: The BFI-2-S and BFI-2-XS
Soto, C. J., & John, O. P. (2017) · Journal of Research in Personality · doi:10.1016/j.jrp.2017.02.004 · Cited by 927
Metadata
- Authors
- Westfall, J., & Yarkoni, T.
- Affiliations
- The University of Texas at Austin (US)
- Corresponding author
- Westfall, Jacob
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-03-31
- Volume
- 11
- Issue
- 3
- Pages
- e0152719
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4816570
- OpenAlex
- W2333692357
- Cited by
- 547
- References
- 57
- Citations by year
- 2026: 37 · 2025: 44 · 2024: 43 · 2023: 51 · 2022: 39 · 2021: 67 · 2020: 60 · 2019: 72 · 2018: 60 · 2017: 57 · 2016: 16
- FWCI
- 74.91
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of Mental Health (R01MH096906)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- type I error rate, measurement error, multiple linear regression, structural equation modeling, statistical power
- MeSH terms
- Humans, Monte Carlo Method, Probability, Psychometrics, Social Sciences, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistically Controlling for Confounding Constructs Is Harder than You Think. https://app.cochat.ai/papers/10.1371/journal.pone.0152719
- The paper
- Westfall, J., & Yarkoni, T. (2016). Statistically Controlling for Confounding Constructs Is Harder than You Think. PLOS ONE. https://doi.org/10.1371/journal.pone.0152719
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.