The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers.
Abstract
Data analysis methods in psychology still emphasize statistical significance testing, despite numerous articles demonstrating its severe deficiencies. It is now possible to use meta-analysis to show that reliance on significance testing retards the development of cumulative knowledge. But reform of teaching and practice will also require that researchers learn that the benefits that they believe flow from use of significance testing are illusory. Teachers must revamp their courses to bring students to understand that (a) reliance on significance testing retards the growth of cumulative research knowledge; (b) benefits widely believed to flow from significance testing do not in fact exist; and (c) significance testing methods must be replaced with point estimates and confidence intervals in individual studies and with meta-analyses in the integration of multiple studies. This reform is essential to the future progress of cumulative knowledge in psychological research. In 1990, Aiken, West, Sechrest, and Reno pub-lished an important article surveying the teaching of quantitative methods in graduate psychology programs. They were concerned about what was not being taught or was being inadequately taught to future researchers and the harm this might cause to research progress in psychology. For example, they found that new and important quantitative methods such as causal modeling, confirmatory factor analysis, and meta-analysis were not being taught in the majority of graduate programs. This is indeed a legitimate cause for concern. But in this article, I am concerned about the opposite: An earlier version of this article was presented as the presidential address to the Division of Evaluation,
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Self-citationFisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Meta-Analytic Procedures for Social Research.
Metcalfe, A., & Rosenthal, R. (1994) · The Statistician · doi:10.2307/2348600 · Cited by 4,249
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Beyond the two disciplines of scientific psychology.
Cronbach, L. J. (1975) · American Psychologist · doi:10.1037/h0076829 · Cited by 2,094
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.806 · Cited by 1,947
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological Bulletin · doi:10.1037/h0026141 · Cited by 1,350
Meta-Analysis: Cumulating Research Findings across Studies
Feldman, J. M., Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1984) · The Academy of Management Review · doi:10.2307/258247 · Cited by 1,199
Self-citationTheory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
Toward Transformation in Social Knowledge
Gergen, K. J. (1982) · Springer eBooks · doi:10.1007/978-1-4612-5706-6 · Cited by 947
The case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931
The test of significance in psychological research.
Bakan, D. (1966) · Psychological Bulletin · doi:10.1037/h0020412 · Cited by 911
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
On the Problem of the Most Efficient Tests of Statistical Hypotheses
Neyman, J., & Pearson, E. S. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-0919-5_6 · Cited by 742
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 729
Measurement error in psychological research: Lessons from 26 research scenarios.
Schmidt, F. L., & Hunter, J. E. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.199 · Cited by 567
Self-citationWhat do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 2×
“As conclusions from research literature come more and more to be based on findings from meta-analysis (Cooper & Hedges, 1994; Lipsey & Wilson, 1993; Schmidt, 1992), the significance test necessarily becomes less and less important.”
BackgroundSelf-citationThings I have learned (so far).
Cohen, J. (1992) · Methodological issues & strategies in clinical research. · doi:10.1037/10109-028 · Cited by 553
Vote-counting methods in research synthesis.
Hedges, L. V., & Olkin, I. (1980) · Psychological Bulletin · doi:10.1037//0033-2909.88.2.359 · Cited by 495
Accumulating Evidence: Procedures for Resolving Contradictions among Different Research Studies
Light, R., & Smith, P. (1971) · Harvard Educational Review · doi:10.17763/haer.41.4.437714870334w144 · Cited by 445
The testing of statistical hypotheses in relation to probabilities a priori
Neyman, J., & Pearson, E. S. (1933) · Mathematical Proceedings of the Cambridge Philosophical Society · doi:10.1017/s030500410001152x · Cited by 412
Statistical Inference: A Commentary for the Social and Behavioural Sciences
Oakes, M. W. (1986) · Medical Entomology and Zoology · Cited by 388
Dichotomization of continuous variables: The implications for meta-analysis.
Hunter, J. E., & Schmidt, F. L. (1990) · Journal of Applied Psychology · doi:10.1037//0021-9010.75.3.334 · Cited by 376
Self-citationReview of developments in meta-analytic method.
Bangert-Drowns, R. L. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.99.3.388 · Cited by 327
Aiken, L. S., West, S. G., Sechrest, L., & Reno, R. R. (1990) · American Psychologist · doi:10.1037//0003-066x.45.6.721 · Cited by 249
Statistical power in criterion-related validation studies.
Schmidt, F. L., Hunter, J. E., & Urry, V. W. (1976) · Journal of Applied Psychology · doi:10.1037/0021-9010.61.4.473 · Cited by 195
Self-citationThe Interpretation of Levels of Significance by Psychological Researchers
Rosenthal, R., & Gaito, J. (1963) · The Journal of Psychology · doi:10.1080/00223980.1963.9916596 · Cited by 185
On the Tyranny of Hypothesis Testing in the Social Sciences
Loftus, G. R. (1991) · Contemporary Psychology · doi:10.1037/029395 · Cited by 136
Some Statistical Problems in Research Design
Kish, L. (2017) · Routledge eBooks · doi:10.4324/9781315134918-20 · Cited by 122
The illogic of statistical inference for cumulative science
Guttman, L. (1985) · Applied Stochastic Models and Data Analysis · doi:10.1002/asm.3150010103 · Cited by 118
Cumulative research knowledge and social policy formulation: The critical role of meta-analysis.
Hunter, J. E., & Schmidt, F. L. (1996) · Psychology Public Policy and Law · doi:10.1037/1076-8971.2.2.324 · Cited by 117
Self-citationProof? No. Evidence? Yes. The Significance of Tests of Significance
Winch, R. F., & Campbell, D. T. (2017) · Routledge eBooks · doi:10.4324/9781315134918-27 · Cited by 115
Huberty, C. J. (1993) · The Journal of Experimental Education · doi:10.1080/00220973.1993.10806593 · Cited by 100
MODERATOR RESEARCH AND THE LAW OF SMALL NUMBERS1
Schmidt, F. L., & Hunter, J. E. (1978) · Personnel Psychology · doi:10.1111/j.1744-6570.1978.tb00441.x · Cited by 87
Self-citationIntroduction to Statistics for Psychology and Education
Clayton, K. N. (1984) · Medical Entomology and Zoology · Cited by 71
A WITHIN SETTING EMPIRICAL TEST OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION
Schmidt, F. L., & Hunter, J. E. (1984) · Personnel Psychology · doi:10.1111/j.1744-6570.1984.tb01453.x · Cited by 48
Self-citationStatistical Theory and Research Design
Jones, L. V. (1955) · Annual Review of Psychology · doi:10.1146/annurev.ps.06.020155.002201 · Cited by 38
Cited by
Effect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Neurocognitive deficit in schizophrenia: A quantitative review of the evidence.
Heinrichs, R. W., & Zakzanis, K. K. (1998) · Neuropsychology · doi:10.1037/0894-4105.12.3.426 · Cited by 2,798
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Metadata
- Authors
- Schmidt, F. L.
- Venue
- Psychological Methods
- Published
- 1996-06
- Volume
- 1
- Issue
- 2
- Pages
- 115-129
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W1965436337
- Cited by
- 830
- References
- 44
- FWCI
- 57.17
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.