The New Statistics: Why and How
The DOI matches its Crossref record, with a small difference in title or year; 91 of the 91 references it cites were found in a registry or an index.
Abstract
We need to make substantial changes to how we conduct research. First, in response to heightened concern that our published research literature is incomplete and untrustworthy, we need new requirements to ensure research integrity. These include prespecification of studies whenever possible, avoidance of selection and other inappropriate data-analytic practices, complete reporting, and encouragement of replication. Second, in response to renewed recognition of the severe flaws of null-hypothesis significance testing (NHST), we need to shift from reliance on NHST to estimation and other preferred techniques. The new statistics refers to recommended practices, including estimation based on effect sizes, confidence intervals, and meta-analysis. The techniques are not new, but adopting them widely would be new for many researchers, as well as highly beneficial. This article explains why the new statistics are important and offers guidance for their use. It describes an eight-step new-statistics strategy for research with integrity, which starts with formulation of research questions in estimation terms, has no place for NHST, and is aimed at building a cumulative quantitative discipline.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Cumming (2013)
In the citing paper
“The bootstrap procedure was also used to produce better estimates of p values and confidence intervals (CIs). This method is often recommended because it does not assume normally distributed data (Cumming, 2014).”
From In an Absolute State: Elevated Use of Absolutist Words Is a Marker Specific to Anxiety, Depression, and Suicidal Ideation (2018) · cites this paper as Cumming, 2014
In the citing paper
“Research on procrastination, perfectionism and their correlates has burgeoned in recent years indicating that such an update is both timely and important to gain current insights into the nomological networks of each (Cumming, 2014).”
From A Meta–Analytic and Conceptual Update on the Associations between Procrastination and Multidimensional Perfectionism (2017) · cites this paper as Cumming, 2014
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,042
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
David, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
(2007) · Cited by 10,092
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Comparison of Beck Depression Inventories -IA and -II in psychiatric outpatients
Beck, A. T., Steer, R. A., Ball, R., & Ranieri, W. F. (1996) · Journal of Personality Assessment · doi:10.1207/s15327752jpa6703_13 · Cited by 5,965
Effect size estimates: Current use, calculations, and interpretation.
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024338 · Cited by 5,199 · cited 1×
“Fritz, Morris, and Richler (2012) discussed a range of ES measures, especially in the context of multiway designs, with an emphasis on practical use and interpretation.”
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
(2011) · Cited by 2,025
Self-citationThe statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Research Synthesis and Meta-Analysis: A Step-by-Step Approach
Cooper, H. (2017) · doi:10.4135/9781071878644 · Cited by 1,716
Publication Manual of the American Psychological Association: BOOK REVIEW
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,451
Inference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449 · cited 3×
“The best way to think of SE bars on a mean is usually to double the whole length—so the bars extend 2 SE above and 2 SE below the mean—and interpret them as being, approximately, the 95% CI, as Figure 2 illustrates (Cumming & Finch, 2005).”
Self-citationPashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Statistics with Confidence: Confidence Intervals and Statistical Guidelines.
Jolliffe, F. R., Gardner, M. J., & Altman, D. G. (1990) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982808 · Cited by 1,319
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
Modern robust statistical methods: An easy way to maximize the accuracy and power of your research.
Erceg-Hurn, D. M., & Mirosevich, V. M. (2008) · American Psychologist · doi:10.1037/0003-066x.63.7.591 · Cited by 959 · cited 1×
“Robust methods largely sidestep such problems and deserve to be more widely used (Erceg-Hurn & Mirosevich, 2008; Wilcox, 2011).”
Error bars in experimental biology
Cumming, G., Fidler, F., & Vaux, D. L. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200611141 · Cited by 940
Self-citationContrasts and Effect Sizes in Behavioral Research: A Correlational Approach
Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (1999) · Cited by 854
McCarthy, M. A. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511802454 · Cited by 756
Using confidence intervals for graphically based data interpretation.
Masson, M. E. J., & Loftus, G. R. (2003) · Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale · doi:10.1037/h0087426 · Cited by 730
Schmidt, F. L., Oh, I., & Hayes, T. L. (2009) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711007x255327 · Cited by 708
Inference by eye: Reading the overlap of independent confidence intervals
Cumming, G. (2009) · Statistics in Medicine · doi:10.1002/sim.3471 · Cited by 670
Self-citationWhat If There Were No Significance Tests?
(2013) · Psychology Press eBooks · doi:10.4324/9781315827353 · Cited by 667
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 654
Doing Bayesian Data Analysis: A Tutorial with R and BUGS
Barry, J. (2011) · Europe’s Journal of Psychology · doi:10.5964/ejop.v7i4.163 · Cited by 563 · cited 1×
“Bayesian approaches to estimation based on credible intervals, to model assessment and selection, and to meta-analysis are highly valuable (Kruschke, 2010).”
Steiger, J. H. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.164 · Cited by 547
Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542
What If There Were No Significance Tests?
(2016) · Routledge eBooks · doi:10.4324/9781315629049 · Cited by 542
Research synthesis and meta-analysis: A step-by-step approach (4th ed.).
Cooper, H. M. (2010) · Cited by 541
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528
Self-citationGrissom, R. J., & Kim, J. J. (2012) · doi:10.4324/9780203803233 · Cited by 470
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451
Self-citationThe epistemology of mathematical and statistical modeling: A quiet methodological revolution.
Rodgers, J. L. (2010) · American Psychologist · doi:10.1037/a0018326 · Cited by 402
Effect Sizes for Research: Univariate and Multivariate Applications
Grissom, R. J., & Kim, J. J. (2005) · Cited by 372
Cooper, H., & Patall, E. A. (2009) · Psychological Methods · doi:10.1037/a0015565 · Cited by 339 · cited 8×
“As mentioned earlier, extensions of most of the discussions in this estimation section, and information about ESCI, are available elsewhere (Cumming, 2012, 2013). Fidler and I (Fidler & Cumming, 2013) also discussed many of the topics of this article. Kline (2013b) discussed many new-statistics issues and provided gui…”
InfluentialBeyond significance testing: Statistics reform in the behavioral sciences (2nd ed.).
Kline, R. B. (2013) · doi:10.1037/14136-000 · Cited by 316
UNIFORM REQUIREMENTS FOR MANUSCRIPTS SUBMITTED TO BIOMEDICAL JOURNALS
Davidoff, F., Godlee, F., Hoey, J., Glass, R. M., Overbeke, J., Utiger, R. D., Nicholls, M. G., Horton, R. C., Nylenna, M., Højgaard, L., et al. (1988) · Indian Journal of Pharmacology · doi:10.4103/ijpha_2003355_330 · Cited by 303
BootES: An R package for bootstrap confidence intervals on effect sizes
Kirby, K. N., & Gerlanc, D. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0330-5 · Cited by 269
A peculiar prevalence of p values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266 · cited 1×
“Not using NHST would remove the pressure to find ways to tweak p to get past the sharp criterion for statistical significance (Masicampo & Lalande, 2012).”
Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach
Boik, R. J. (2001) · Journal of the American Statistical Association · doi:10.1198/jasa.2001.s432 · Cited by 257
THE FALLACY OF EMPLOYING STANDARDIZED REGRESSION COEFFICIENTS AND CORRELATIONS AS MEASURES OF EFFECT
GREENLAND, S., SCHLESSELMAN, J. J., & CRIQUI, M. H. (1986) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a114229 · Cited by 256
Methods for the Behavioral, Educational, and Social Sciences: An R package
Kelley, K. (2007) · Behavior Research Methods · doi:10.3758/bf03192993 · Cited by 255
Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014) · Personality and Social Psychology Review · doi:10.1177/1088868313507536 · Cited by 245
Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01502008.x · Cited by 244
Self-citationFiedler, K., Kutzner, F., & Krueger, J. I. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462587 · Cited by 229
Modern Statistics for the Social and Behavioral Sciences: A Practical Introduction
Wilcox, R. R. (2011) · Cited by 217
Serious stats: a guide to advanced statistics for the behavioral sciences
(2013) · Choice Reviews Online · doi:10.5860/choice.50-3917 · Cited by 197
Baguley, T. (2012) · doi:10.1007/978-0-230-36355-7 · Cited by 194
Not exactly: in praise of vagueness
(2010) · Choice Reviews Online · doi:10.5860/choice.48-0189 · Cited by 186
Statistics with Confidence. Confidence Intervals and Statistical Guidelines
Newcombe, R. (1990) · Journal of Medical Genetics · doi:10.1136/jmg.27.1.71 · Cited by 181
Smithson, M. (2003) · Sage eBooks · doi:10.4135/9781412983761 · Cited by 179
Replication and Researchers' Understanding of Confidence Intervals and Standard Error Bars
Cumming, G., Williams, J., & Fidler, F. (2004) · Understanding Statistics · doi:10.1207/s15328031us0304_5 · Cited by 172
Self-citationFisher, W. H. (2008) · Psychiatric Services · doi:10.1176/appi.ps.59.5.579 · Cited by 147
The Ancestor's Tale: A Pilgrimage to the Dawn of Life
Dawkins, R., & Wong, Y. (2016) · Cited by 140
The Ancestor's Tale: A Pilgrimage to the Dawn of Life
Mataré, H. F. (2005) · Mankind Quarterly · Cited by 135
Cumming, G., Fidler, F., Kalinowski, P., & Lai, J. (2012) · Australian Journal of Psychology · doi:10.1111/j.1742-9536.2011.00037.x · Cited by 135
Self-citationHarlow, L. L., Mulaik, S. A., & Steiger, J. H. (2013) · doi:10.4324/9781315629049-15 · Cited by 134
Meta-analytic interval estimation for standardized and unstandardized mean differences.
Bonett, D. G. (2009) · Psychological Methods · doi:10.1037/a0016619 · Cited by 125
Confidence intervals and replication: Where will the next mean fall?
Cumming, G., & Maillardet, R. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.3.217 · Cited by 122
Self-citationResearch methods in psychology, 2nd ed.
Breakwell, G. M., Hammond, S., & Fife‐Schaw, C. (2000) · Cited by 118
Handbook of Ethics in Quantitative Methodology
(2011) · Routledge eBooks · doi:10.4324/9780203840023 · Cited by 108
Rejecting the Ideal of Value‐Free Science
Douglas, H. (2007) · Value-Free Science? · doi:10.1093/acprof:oso/9780195308969.003.0009 · Cited by 106
The Importance of Effect Magnitude
Kirk, R. E. (2003) · Handbook of Research Methods in Experimental Psychology · doi:10.1002/9780470756973.ch5 · Cited by 94
Confidence Intervals: Better Answers to Better Questions
Cumming, G., & Fidler, F. (2009) · Zeitschrift für Psychologie / Journal of Psychology · doi:10.1027/0044-3409.217.1.15 · Cited by 83 · cited 3×
“Because d is the ratio of two estimated quantities, its distribution is complex, and it is not straightforward to calculate CIs for d. ESCI can provide CIs on d in a number of basic situations, or you can use good approximations (Cumming & Fidler, 2009).”
Self-citationCoulson, M. (2010) · Frontiers in Psychology · doi:10.3389/fpsyg.2010.00026 · Cited by 77
Fidler, F., & Loftus, G. R. (2009) · Zeitschrift für Psychologie / Journal of Psychology · doi:10.1027/0044-3409.217.1.27 · Cited by 72
Inference by Eye: Pictures of Confidence Intervals and Thinking About Levels of Confidence
Cumming, G. (2007) · Teaching Statistics · doi:10.1111/j.1467-9639.2007.00267.x · Cited by 62
Self-citationModern Statistics for the Social and Behavioral Sciences
Wilcox, R. (2011) · doi:10.1201/9781466503236 · Cited by 62
Handbook of Research Methods in Experimental Psychology
(2005) · Blackwell Publishing Ltd eBooks · doi:10.1111/b.9781405132800.2005.x · Cited by 59
Fidler, F. (2018) · doi:10.31237/osf.io/7xdpq · Cited by 49
Reporting Research in Psychology: How to Meet Journal Article Reporting Standards
Cooper, H. M. (2010) · Medical Entomology and Zoology · Cited by 48
The value of RCT evidence depends on the quality of statistical analysis
Faulkner, C., Fidler, F. M., & Cumming, G. (2007) · Behaviour Research and Therapy · doi:10.1016/j.brat.2007.12.001 · Cited by 47
Self-citationReform of statistical inference in psychology: The case ofMemory & Cognition
Finch, S. J., Cumming, G., Williams, J., Palmer, L., Griffith, E., Alders, C., Anderson, J. L., & Goodman, O. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195577 · Cited by 43
Self-citationTheory Testing Using Quantitative Predictions of Effect Size
Velicer, W. F., Cumming, G., Fava, J. L., Rossi, J. S., Prochaska, J. O., & Johnson, J. (2008) · Applied Psychology · doi:10.1111/j.1464-0597.2008.00348.x · Cited by 43
Self-citationPutting Research in Context: Understanding Confidence Intervals from One or More Studies
Finch, S., & Cumming, G. (2009) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsn118 · Cited by 43
Self-citationIntroduction to the Special Section on Research Practices
Spellman, B. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465075 · Cited by 24
META-ANALYSIS: PICTURES THAT EXPLAIN HOW EXPERIMENTAL FINDINGS CAN BE INTEGRATED
Cumming, G. (2006) · Cited by 23
Self-citationOn Confidence Intervals for Within-Subjects Designs.
Blouin, D. C., & Riopelle, A. J. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.397 · Cited by 19
Effect Size Estimation and Confidence Intervals
Fidler, F. M., & Cumming, G. (2012) · Wiley eBooks · doi:10.1002/9781118133880.hop202005 · Cited by 9
Self-citation
Research works printed but not linked: 2
- 10.7326/0003-4819-108-2-258 · doi:10.7326/0003-4819-108-2-258
- Kline R. B. (2013a). Beyond significance testing (2nd edition): Reader and instructor resource guide. Retrieved from tiny.cc/rklinesite
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Moving to a World Beyond “ p < 0.05”
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019) · The American Statistician · doi:10.1080/00031305.2019.1583913 · Cited by 2,976
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Bakdash, J. Z., & Marusich, L. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00456 · Cited by 2,018
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962
Metadata
- Authors
- Cumming, G.
- Affiliations
- La Trobe University (AU)
- Corresponding author
- Cumming, Geoff
- ORCID
- Cumming, Geoff
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2013-11-12
- Volume
- 25
- Issue
- 1
- Pages
- 7-29
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2127875809
- Semantic Scholar
- 8ad6d501eac936c9d9b780eca017d5d1d1e59fae
- Cited by
- 3,200
- References
- 91
- Citations by year
- 2026: 231 · 2025: 225 · 2024: 220 · 2023: 220 · 2022: 271 · 2021: 309 · 2020: 306 · 2019: 255 · 2018: 288 · 2017: 260 · 2016: 236 · 2015: 238 · 2014: 135 · 2013: 4
- FWCI
- 222.82
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- new statistics, null hypothesis significance testing, research integrity, replication, effect size, confidence intervals, meta-analysis, cumulative science
- MeSH terms
- Data Interpretation, Statistical, Humans, Psychology, Statistics as Topic, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.