From Statistical Significance to Effect Estimation: Statistical Reform in Psychology, Medicine and Ecology
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Compelling criticisms of statistical significance testing (or Null Hypothesis Significance Testing, NHST) can be found in virtually all areas of the social and life sciences—including economics, sociology, ecology, biology, education and psychology. Because it is the overwhelmingly dominant statistical method in these sciences, criticisms need to be taken seriously. Yet, after half a century of cogent arguments against NHST and calls to adopt alternative practices some disciplines show little sign of change. One obvious question is ‘why?’ Why are researchers so unwilling to abandon this flawed practice? In this thesis I attempt to answer this question, and compare practice across scientific disciplines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Schmidt, F. L., & Hunter, J. E. (1998) · Psychological Bulletin · doi:10.1037//0033-2909.124.2.262 · Cited by 4,180
Defining empirically supported therapies.
Chambless, D. L., & Hollon, S. D. (1998) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.66.1.7 · Cited by 3,067
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-0919-5_38 · Cited by 3,028
AIC model selection using Akaike weights
Wagenmakers, E.-J., & Farrell, S. (2004) · Psychonomic Bulletin & Review · doi:10.3758/bf03206482 · Cited by 2,684
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews
Rosenthal, R., & DiMatteo, M. R. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.59 · Cited by 1,780 · cited 1×
“, difficulties in conducting power and meta-analysis without adequate effect size information were discussed by Rossi (1997); amongst others Cohen (1988) and Rosenthal and Rubin (1982) have discussed the interpretation of effect sizes.”
MethodPractical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · cited 4×
“Journal editors, as has been suggested (Kirk, 1996; Sedlmeier & Gigerenzer, 1989), may be the key to change, but to date the policies in psychology journals(23) have not been effective in this capacity.”
BackgroundMethodInfluentialBayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
The specificity of the scientific field and the social conditions of the progress of reason
Bourdieu, P. (1975) · Social Science Information · doi:10.1177/053901847501400602 · Cited by 1,483
Inference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449
Statistical Power Analysis for the Behavioral Sciences
Wassertheil, S., & Cohen, J. (1970) · Biometrics · doi:10.2307/2529115 · Cited by 1,383
Eta-squared and partial eta-squared in fixed factor anova designs
Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300111 · Cited by 1,333
Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984) · Behavior Therapy · doi:10.1016/s0005-7894(84)80002-7 · Cited by 1,294
On judging the significance of differences by examining the overlap between confidence intervals.
Schenker, N., & Gentleman, J. F. (2001) · The American Statistician · doi:10.1198/000313001317097960 · Cited by 1,218
Cumulative Meta-Analysis of Therapeutic Trials for Myocardial Infarction
Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992) · New England Journal of Medicine · doi:10.1056/nejm199207233270406 · Cited by 1,140
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 4×
“Sterling (1959) surveyed articles in four leading psychology journals(10) published in 1955 and 1956.”
BackgroundInfluentialTheory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
The case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931
Generality of learned helplessness in man.
Hiroto, D. S., & Seligman, M. E. (1975) · Journal of Personality and Social Psychology · doi:10.1037/h0076270 · Cited by 896 · cited 3×
“The effects of helplessness on growth of cancerous tumours and death rates were first observed in rats, and later experiments demonstrated the links in human subjects (Hiroto & Seligman, 1975; Miller & Seligman, 1975). Seligman and his colleagues published at least 25 articles on the topic between 1969 and 1977, and a…”
InfluentialJacobson, N. S., Roberts, L. J., Berns, S. B., & McGlinchey, J. B. (1999) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.67.3.300 · Cited by 853
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564
Depression and learned helplessness in man
Miller, W. R., & Seligman, M. E. (1975) · Journal of Abnormal Psychology · doi:10.1037/h0076720 · Cited by 563
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528
Researchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390
Self-citationJacobson, N. S., & Truax, P. (1992) · Methodological issues & strategies in clinical research. · doi:10.1037/10109-042 · Cited by 384
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323 · cited 2×
“For example, Chase and Chase (1976) found that the average statistical power in JAP was considerably higher than what Cohen (1962) and follow up surveys (Rossi, 1990; Sedlemier & Gigerenzer, 1989) found in the Journal of Abnormal Psychology.”
BackgroundNeeded: A Ban on the Significance Test
Hunter, J. E. (1997) · Psychological Science · doi:10.1111/j.1467-9280.1997.tb00534.x · Cited by 314
When shall we use our heads instead of the formula?
Meehl, P. E. (1957) · Journal of Counseling Psychology · doi:10.1037/h0047554 · Cited by 310 · cited 1×
“An often cited example of this is Melton’s (1962) policy at the Journal of Experimental Psychology.”
BackgroundStatistical Evaluation of Medical Journal Manuscripts
Schor, S. (1966) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1966.03100130097026 · Cited by 207
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.46.4.806 · Cited by 188
Promoting good statistical practices: Some suggestions
Kirk, R. E. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971185 · Cited by 184
When can we Conclude that Treatments or Programs “Don’t Work”?
Weisburd, D., Lum, C. M., & Yang, S.-M. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716202250782 · Cited by 76
Cited by
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
(2011) · Cited by 2,025
McCarthy, M. A. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511802454 · Cited by 756
Robust misinterpretation of confidence intervals
Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0572-3 · Cited by 425
Results reporting in single case experiments and single case meta-analysis
Vannest, K. J., Peltier, C., & Haas, A. (2018) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2018.04.029 · Cited by 51
Metadata
- Authors
- Fidler, F.
- Publisher
- Center for Open Science
- Published
- 2018-10-29
- OpenAlex
- W2737365078
- Cited by
- 49
- References
- 33
- Type
- posted content
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.