Statistical Reform in Psychology: Is Anything Changing?
Open accessAbstract
We investigated whether statistical practices in psychology have been changing since 1998. Early in this period, the American Psychological Association (APA) Task Force on Statistical Inference (TFSI; Wilkinson & TFSI, 1999) advocated improved statistical practices, including reporting effect sizes and confidence intervals (CIs). The APA (2001) Publication Manual included expanded advice on effect sizes and stated that CIs are ‘‘in general, the best reporting strategy’’ (p. 22). Any changes since 1998 may, of course, have had many causes other than those two sources of advice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST).
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Cumming et al., 2007
Null hypothesis significance testing (NHST) is one of the most widely used methods for testing hypotheses in psychological research.
“Null hypothesis significance testing (NHST) is one of the most widely used methods for testing hypotheses in psychological research (Cumming et al., 2007).”
From Misconceptions of the p-value among Chilean and Italian Academic Psychologists (2016) · cites this paper as Cumming et al., 2007
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791 · see citing works
Inference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
- Fidler, F., Cumming, G., Thomason, N., Pannuzzo, D., Smith, J., Fyffe, P., Edmonds, H., Harrington, C., & Schmitt, R. (2005). Evalu- ating the effectiveness of editorial policy to improve statistical practice: The case of the Journal of Consulting and Clinical Psychology. Journal of Consulting and Clinical Psychology, 73, 136-143.
- La Greca, A.M. (2005). Editorial. Journal of Consulting and Clinical Psychology, 73, 3-5.
- Meehl, P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.
Cited by
Sorted by citations
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461 · see citing works
Un sistema de clasificación de los diseños de investigación en psicología
Ato, M., López-García, J. J., & Benavente, A. (2013) · Anales de Psicología · doi:10.6018/analesps.29.3.178511 · Cited by 2,254 · see citing works
Modern robust statistical methods: An easy way to maximize the accuracy and power of your research.
Erceg-Hurn, D. M., & Mirosevich, V. M. (2008) · American Psychologist · doi:10.1037/0003-066x.63.7.591 · Cited by 959 · see citing works
Error bars in experimental biology
Cumming, G., Fidler, F., & Vaux, D. L. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200611141 · Cited by 940 · see citing works
Self-citationStandardized or simple effect size: what should be reported?
Baguley, T. (2009) · British Journal of Psychology · doi:10.1348/000712608x377117 · Cited by 619 · see citing works
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · see citing works
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451 · see citing works
Self-citationThe (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · see citing works
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336 · see citing works
A peculiar prevalence of <i>p</i> values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266 · see citing works
Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.102 · Cited by 79 · see citing works
Coulson, M. (2010) · Frontiers in Psychology · doi:10.3389/fpsyg.2010.00026 · Cited by 77 · see citing works
Nuijten, M. B., & Polanin, J. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1408 · Cited by 56 · see citing works
Misconceptions of the p-value among Chilean and Italian Academic Psychologists
Badenes-Ribera, L., Frias-Navarro, D., Iotti, B., Bonilla-Campos, A., & Longobardi, C. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01247 · Cited by 35 · see citing works
Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science
Sakaluk, J., Williams, A., & Biernat, M. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614549257 · Cited by 32 · see citing works
Consistency errors in p-values reported in Spanish psychology journals
Caperos, J., & Pardo, A. (2013) · Psicothema · doi:10.7334/psicothema2012.207 · Cited by 23 · see citing works
Shearer-Underhill, C., & Marker, C. (2010) · Clinical Psychology: Science and Practice · doi:10.1111/j.1468-2850.2009.01191.x · Cited by 16 · see citing works
Nuijten, M. B., & Wicherts, J. M. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241258945 · Cited by 6 · see citing works
“Many conclusions in psychological research are based on the results of null hypothesis significance tests (NHSTs; Cumming et al., 2007; Nuijten et al., 2016).”
- Authors
- Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Lo, J., McMenamin, N., & Wilson, S.
- Affiliations
- La Trobe University (AU)
- La Trobe University, Melbourne, Victoria, Australia
- Corresponding author
- Cumming, Geoff
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2007-03
- Volume
- 18
- Issue
- 3
- Pages
- 230-232
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2137782068
- Cited by
- 122
- References
- 2
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 3 · 2022: 1 · 2021: 2 · 2020: 7 · 2019: 7 · 2018: 9 · 2017: 8 · 2016: 13 · 2015: 2 · 2014: 11 · 2013: 10 · 2012: 8
- FWCI
- 11.99
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- statistical reform, effect size, confidence intervals, APA Publication Manual, reporting practices, change over time
- MeSH terms
- Data Interpretation, Statistical, Humans, Periodicals as Topic, Psychology, Experimental, Confidence Intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
