Measurement error and the replication crisis
Abstract
The assumption that measurement error always reduces effect sizes is false
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Low replication rates have been attributed to more than just questionable research practices.
“low replication rates have been attributed to more than just questionable research practices.”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Loken & Gelman, 2017
Low replication rates have been attributed to insufficiently stringent evidence thresholds, insufficiently lenient evidence thresholds, poor measurement, model misspecification, low power, poor theory, an underappreciation of the influence of hidden moderators, and errors in substantive inference.
“low replication rates have been attributed to (a) insufficiently stringent evidence thresholds (Benjamin et al., 2018), (b) insufficiently lenient evidence thresholds (Devezer et al., 2020), (c) poor measurement (e.g., Loken & Gelman, 2017), (d) model misspecification (Devezer et al., 2020), (e) low power (e.g., Rossi, 1990), (f) poor theory (e.g., Oberauer & Lewandowsky, 2019; Szollosi &a”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Loken & Gelman, 2017
Increased error in a population, whether systematic or random, makes measures less capable at capturing individual differences in the trait of interest in that population.
“Increased error in some population (whether systematic or random) makes measures less capable at capturing individual differences in the trait of interest in that population (see e.g., refs. 20–22; see also ref. 23).”
From Simpson’s gender-equality paradox (2025) · cites this paper as [23]
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457 · see citing works
Power Analysis and Effect Size in Mixed Effects Models: A Tutorial
Brysbaert, M., & Stevens, M. (2018) · Journal of Cognition · doi:10.5334/joc.10 · Cited by 1,420 · see citing works
“The consequences of underpowered studies are becoming increasingly clear (Loken & Gelman, 2017).”
BackgroundWhy Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382 · see citing works
“Further, measures with low reliability compromise the inferences drawn from hypothesis tests because noise factors obscure causal effects on the dependent variable (Loken & Gelman, 2017; Shadish et al., 2001).”
BackgroundSearching for the backfire effect: Measurement and design considerations.
Swire-Thompson, B., DeGutis, J., & Lazer, D. (2020) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2020.06.006 · Cited by 360 · see citing works
Reflection on modern methods: five myths about measurement error in epidemiological research
van Smeden, M., Lash, T. L., & Groenwold, R. H. H. (2020) · International Journal of Epidemiology · doi:10.1093/ije/dyz251 · Cited by 173 · see citing works
Obtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts
Wiernik, B. M., & Dahlke, J. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919885611 · Cited by 135 · see citing works
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137 · see citing works
Measuring behavior in the home cage: study design, applications, challenges, and perspectives
Grieco, F., Bernstein, B. J., Biemans, B., Bikovski, L., Burnett, C. J., Cushman, J. D., van Dam, E. A., Fry, S. A., Richmond-Hacham, B., Homberg, J. R., et al. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.735387 · Cited by 109 · see citing works
Irving, J., Patel, R., Oliver, D., Colling, C., Pritchard, M., Broadbent, M., Baldwin, H., Stahl, D., Stewart, R., & Fusar-Poli, P. (2021) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaa126 · Cited by 92 · see citing works
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61 · see citing works
“…evidence thresholds (Benjamin et al., 2018), (b) insufficiently lenient evidence thresholds (Devezer et al., 2020), (c) poor measurement (e.g., Loken & Gelman, 2017), (d) model misspecification (Devezer et al., 2020), (e) low power (e.g., Rossi, 1990), (f) poor theory (e.g., Oberauer &…”
BackgroundFlake, J. K. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1898962 · Cited by 58 · see citing works
Random measurement error: Why worry? An example of cardiovascular risk factors
Brakenhoff, T. B., van Smeden, M., Visseren, F. L. J., & Groenwold, R. H. H. (2018) · PLOS ONE · doi:10.1371/journal.pone.0192298 · Cited by 52 · see citing works
Oliver, D., Spada, G., Colling, C., Broadbent, M., Baldwin, H., Patel, R., Stewart, R., Stahl, D., Dobson, R., McGuire, P., et al. (2021) · Schizophrenia Research · doi:10.1016/j.schres.2020.05.007 · Cited by 49 · see citing works
Jaric, I., Voelkl, B., Clerc, M., Schmid, M. W., Novak, J., Rosso, M., Rufener, R., von Kortzfleisch, V. T., Richter, S. H., Buettner, M., et al. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001837 · Cited by 32 · see citing works
Simpson’s gender-equality paradox
Berggren, M., & Bergh, R. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2422247122 · Cited by 17 · see citing works
“23 ).”
Online repression and tactical evasion: evidence from the 2020 Day of Anger protests in Egypt
Lutscher, P. M., & Ketchley, N. (2023) · Democratization · doi:10.1080/13510347.2022.2140798 · Cited by 13 · see citing works
- Authors
- Loken, E., & Gelman, A.
- Affiliations
- University of Connecticut (US)
- Columbia University (US)
- ORCID
- Gelman, Andrew
- Venue
- Science
- Published
- 2017-02-10
- Volume
- 355
- Issue
- 6325
- Pages
- 584-585
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2589849966
- Cited by
- 628
- References
- 1
- Citations by year
- 2026: 37 · 2025: 63 · 2024: 55 · 2023: 68 · 2022: 79 · 2021: 84 · 2020: 90 · 2019: 75 · 2018: 46 · 2017: 29 · 2013: 1
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Office of Naval Research (N00014-15-1-2541)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- measurement error, replication crisis, effect size, measurement error bias, replication study, research reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
