A comment on replication,P‐values and evidence
Abstract
It is conventionally thought that a small p‐value confers high credibility on the observed alternative hypothesis, and that a repetition of the same experiment will have a high probability of resulting again in statistical significance. It is shown that if the observed difference is the true one, the probability of repeating a statistically significant result, the ‘replication probability’, is substantially lower than expected. The reason for this is a mistake that generates other seeming paradoxes: the interpretation of the post‐trial p‐value in the same way as the pre‐trial α error. The replication probability can be used as a frequentist counterpart of Bayesian and likelihood methods to show that p‐values overstate the evidence against the null hypothesis.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The probability of obtaining a statistically significant result with the same or better p value in a duplicate experiment when the p value in the first experiment was 0.05 may only be around 50%.
“Indeed, as Goodman (1992) has pointed out, the probability of obtaining a statistically significant result with the same or better p value in a duplicate experiment when the p value in the first experiment was 0.05 may only be around 50%.”
From The use of whole food animal studies in the safety assessment of genetically modified crops: Limitations and recommendations (2013) · cites this paper as Goodman (1992)
Cites
IX. On the problem of the most efficient tests of statistical hypotheses
Neyman, J., & Pearson, E. S. (1933) · Philosophical Transactions of the Royal Society of London Series A Containing Papers of a Mathematical or Physical Character · doi:10.1098/rsta.1933.0009 · Cited by 2,370 · see citing works
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · see citing works
Statistical Methods and Scientific Inference.
Bartlett, M. S., & Fisher, R. A. (1957) · Biometrika · doi:10.2307/2333276 · Cited by 1,288 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 971 · see citing works
Statistical Methods and Scientific Inference.
Jowett, G. H., & Fisher, R. A. (1957) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2985611 · Cited by 715 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548 · see citing works
Good Thinking: The Foundations of Probability and Its Applications.
Berger, J. O., & Good, I. J. (1985) · Journal of the American Statistical Association · doi:10.2307/2288079 · Cited by 390 · see citing works
The Effect of Sample Size on the Meaning of Significance Tests
Royall, R. M. (1986) · The American Statistician · doi:10.1080/00031305.1986.10475424 · Cited by 234 · see citing works
Bayesian Interpretation of Standard Inference Statements
Pratt, J. W. (1965) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1965.tb01486.x · Cited by 176 · see citing works
Clinical Trials and Statistical Verdicts: Probable Grounds for Appeal
Diamond, G. A., & FORRESTER, J. S. (1983) · Annals of Internal Medicine · doi:10.7326/0003-4819-98-3-385 · Cited by 137 · see citing works
Bickel, P. J. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/b20576 · Cited by 133 · see citing works
The Effect of Sample Size on the Meaning of Significance Tests
Royall, R. M. (1986) · The American Statistician · doi:10.2307/2684616 · Cited by 70 · see citing works
The use of the likelihood function in statistical practice
Barnard, G. A. (1967) · Project Euclid (Cornell University) · Cited by 67 · see citing works
Tests of Significance in Theory and Practice
Johnstone, D., Barnard, G. A., & Lindley, D. V. (1986) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2987965 · Cited by 65 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200 · see citing works
Self-citationWhat does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
Self-citationCumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451 · see citing works
Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars
Mayo, D. G. (2018) · doi:10.1017/9781107286184 · Cited by 376 · see citing works
Patil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243 · see citing works
A coffee can, factor analysis, and prediction of antisocial behavior: The structure of criminal risk
Kroner, D. G., Mills, J. F., & Reddon, J. R. (2005) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2004.01.011 · Cited by 173 · see citing works
Bartholomaeus, A., Parrott, W., Bondy, G., Walker, K., & on behalf of the ILSI International Food Biotechnology Committee Task Force on the Use of Mammalian Toxicology Studies in the Safety Assessment of GM Foods (2013) · Critical Reviews in Toxicology · doi:10.3109/10408444.2013.842955 · Cited by 59 · see citing works
“Indeed, as Goodman (1992) has pointed out, the probability of obtaining a statistically significant result with the same or better p value in a duplicate experiment when the p value in the first experiment was 0.05 may only be around 50%.”
BackgroundInfluentialHsiao, Y.-Y., Tofighi, D., Kruger, E. S., Lee Van Horn, M., MacKinnon, D. P., & Witkiewitz, K. (2019) · Mindfulness · doi:10.1007/s12671-018-1023-z · Cited by 23 · see citing works
A glass half full interpretation of the replicability of psychological science
Leek, J. T., Patil, P., & Peng, R. D. (2015) · arXiv · doi:10.48550/arxiv.1509.08968 · Cited by 1 · see citing works
- Authors
- Goodman, S. N.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Goodman, Steven N.
- ORCID
- Goodman, Steven N.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 1992-01
- Volume
- 11
- Issue
- 7
- Pages
- 875-879
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2136634361
- Cited by
- 309
- References
- 14
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 10 · 2023: 13 · 2022: 10 · 2021: 25 · 2020: 23 · 2019: 22 · 2018: 43 · 2017: 20 · 2016: 18 · 2015: 8 · 2014: 8 · 2013: 11 · 2012: 12
- FWCI
- 1.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- p-values, statistical significance, frequentist interpretation, Bayesian analysis, likelihood methods
- MeSH terms
- Humans, Probability, Reproducibility of Results, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
