Revised standards for statistical evidence
Open accessAbstract
Recent advances in Bayesian hypothesis testing have led to the development of uniformly most powerful Bayesian tests, which represent an objective, default class of Bayesian hypothesis tests that have the same rejection regions as classical significance tests. Based on the correspondence between these two classes of tests, it is possible to equate the size of classical hypothesis tests with evidence thresholds in Bayesian tests, and to equate P values with Bayes factors. An examination of these connections suggest that recent concerns over the lack of reproducibility of scientific studies can be attributed largely to the conduct of significance tests at unjustifiably high levels of significance. To correct this problem, evidence thresholds required for the declaration of a significant finding should be increased to 25-50:1, and to 100-200:1 for the declaration of a highly significant finding. In terms of classical hypothesis tests, these evidence standards mandate the conduct of tests at the 0.005 or 0.001 level of significance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The p < 0.05 threshold has been criticized as weak evidence against the null hypothesis when sample sizes are extremely large.
“it has been criticized as a weak evidence against the null hypothesis in the light of extremely large sample sizes (Johnson, 2013)”
From Till logout do us part? Comparison of factors predicting excessive social network sites use and addiction between Singaporean adolescents and adults (2017) · cites this paper as Johnson, 2013
Statisticians have specifically recommended a cutoff of p < 0.001 for large datasets.
“statisticians have specifically recommended a cutoff of p < 0.001 for large datasets (Johnson, 2013)”
From Till logout do us part? Comparison of factors predicting excessive social network sites use and addiction between Singaporean adolescents and adults (2017) · cites this paper as Johnson, 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558 · see citing works
Statistical Methods for Research Workers
Fisher, R. A. (1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,883 · see citing works
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1038/109132a0 · Cited by 7,837 · see citing works
Bayesian model selection in social research
Raftery, A. E. (1995) · Sociological Methodology · doi:10.2307/271063 · Cited by 6,823 · see citing works
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · see citing works
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · see citing works
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 971 · see citing works
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 <i>t</i> Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945 · see citing works
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · see citing works
Berger, J. O., & Delampady, M. (1987) · Statistical Science · doi:10.1214/ss/1177013238 · Cited by 664 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548 · see citing works
Uniformly most powerful Bayesian tests
Johnson, V. E. (2013) · The Annals of Statistics · doi:10.1214/13-aos1123 · Cited by 87 · see citing works
Self-citationWhat is the probability of replicating a statistically significant effect?
Miller, J. O. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.4.617 · Cited by 86 · see citing works
A Bayesian Perspective on Hypothesis Testing
Wagenmakers, E., & Grünwald, P. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01757.x · Cited by 65 · see citing works
UNIFORMLY MOST POWERFUL BAYESIAN TESTS
Johnson, V. E. (2013) · OakTrust (Texas A&M University Libraries) · Cited by 60 · see citing works
Self-citationEvidence that publication bias contaminated studies relating social class and unethical behavior
Francis, G. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1203591109 · Cited by 47 · see citing works
Statistical Methods for Research Workers.
Hotelling, H., & Fisher, R. A. (1927) · Journal of the American Statistical Association · doi:10.2307/2276824 · Cited by 31 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [2]Naik G. (December 2, 2011) Scientists’ elusive goal: Reproducing study results. Wall Street Journal, Health Section.
- [11]Jeffreys H. Theory of Probability. 3rd Ed. Oxford: Oxford Univ Press; 1961. [Google Scholar]
Cited by
Sorted by citations
The ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · see citing works
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962 · see citing works
What are the odds? A practical guide to computing and reporting Bayes factors
Jarosz, A. F., & Wiley, J. (2014) · The Journal of Problem Solving · doi:10.7771/1932-6246.1167 · Cited by 1,101 · see citing works
Dinu, M., Pagliai, G., Casini, A., & Sofi, F. (2018) · European Journal of Clinical Nutrition · doi:10.1038/ejcn.2017.58 · Cited by 1,016 · see citing works
Theodoratou, E., Tzoulaki, I., Zgaga, L., & Ioannidis, J. P. A. (2014) · BMJ · doi:10.1136/bmj.g2035 · Cited by 956 · see citing works
Adiposity and cancer at major anatomical sites: umbrella review of the literature
Kyrgiou, M., Kalliala, I., Markozannes, G., Gunter, M. J., Paraskevaidis, E., Gabra, H., Martin-Hirsch, P., & Tsilidis, K. K. (2017) · BMJ · doi:10.1136/bmj.j477 · Cited by 803 · see citing works
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760 · see citing works
Cupp, M. A., Cariolou, M., Tzoulaki, I., Aune, D., Evangelou, E., & Berlanga-Taylor, A. J. (2020) · BMC Medicine · doi:10.1186/s12916-020-01817-1 · Cited by 697 · see citing works
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
Dietary fiber and health outcomes: an umbrella review of systematic reviews and meta-analyses
Veronese, N., Solmi, M., Caruso, M. G., Giannelli, G., Osella, A. R., Evangelou, E., Maggi, S., Fontana, L., Stubbs, B., & Tzoulaki, I. (2018) · The American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqx082 · Cited by 536 · see citing works
Evolution of Reporting <i>P</i> Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370 · see citing works
Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars
Mayo, D. G. (2018) · doi:10.1017/9781107286184 · Cited by 376 · see citing works
FUNDAMENTAL LIMITATIONS OF HIGH CONTRAST IMAGING SET BY SMALL SAMPLE STATISTICS
Mawet, D., Milli, J., Wahhaj, Z., Pelat, D., Absil, O., Delacroix, C., Boccaletti, A., Kasper, M., Kenworthy, M., Marois, C., et al. (2014) · The Astrophysical Journal · doi:10.1088/0004-637x/792/2/97 · Cited by 369 · see citing works
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295 · see citing works
“The small degrees of evidence provided by these published reports, taken at face value, are consistent with observations by Wetzels et al. (2011) as well as the cautionary messages by Johnson (2013) and Maxwell, Lau, and Howard (2015).”
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · see citing works
McCright, A. M., Charters, M., Dentzman, K., & Dietz, T. (2016) · Topics in Cognitive Science · doi:10.1111/tops.12171 · Cited by 236 · see citing works
Are We Wasting a Good Crisis? The Availability of Psychological Research Data after the Storm
Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015) · Collabra · doi:10.1525/collabra.13 · Cited by 152 · see citing works
Ramirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138 · see citing works
Ho, S. S., Lwin, M. O., & Lee, E. W. J. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.06.002 · Cited by 117 · see citing works
Sources of error in the retracted scientific literature
Casadevall, A., Steen, R. G., & Fang, F. C. (2014) · The FASEB Journal · doi:10.1096/fj.14-256735 · Cited by 89 · see citing works
Significance testing in empirical finance: a critical review and assessment
Kim, J. H., & Ji, P. I. (2015) · Journal of Empirical Finance · doi:10.1016/j.jempfin.2015.08.006 · Cited by 86 · see citing works
Reproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77 · see citing works
Buckner, J. D., Zvolensky, M. J., Ecker, A. H., Schmidt, N. B., Lewis, E. M., Paulus, D. J., Lopez-Gamundi, P., Crapanzano, K. A., & Bakhshaie, J. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.10.014 · Cited by 55 · see citing works
“We considered a Bayesian probability of 80% or greater that an effect exists to merit further consideration (Johnson, 2013).”
Outdoor air quality and human health: An overview of reviews of observational studies
Markozannes, G., Pantavou, K., Rizos, E. C., Sindosi, O. Α., Tagkas, C., Seyfried, M., Saldanha, I. J., Hatzianastassiou, N., Nikolopoulos, G. K., & Ntzani, E. (2022) · Environmental Pollution · doi:10.1016/j.envpol.2022.119309 · Cited by 40 · see citing works
Hong, J. S., Sheriff, R., Smith, K., Tomlinson, A., Saad, F., Smith, T., Engelthaler, T., Phiri, P., Henshall, C., Ede, R., et al. (2021) · Evidence-Based Mental Health · doi:10.1136/ebmental-2021-300287 · Cited by 32 · see citing works
Peng, X., Li, J., Zhao, H., Lai, J., Lin, J., & Tang, S. (2022) · BMC Endocrine Disorders · doi:10.1186/s12902-022-01015-5 · Cited by 22 · see citing works
Improving the analysis of routine outcome measurement data: what a Bayesian approach can do for you.
de Vries, R. M., Meijer, R. R., van Bruggen, V., & Morey, R. D. (2016) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1496 · Cited by 8 · see citing works
- Authors
- Johnson, V. E.
- Affiliations
- Texas A&M University (US)
- Corresponding author
- Johnson, Valen E.
- ORCID
- Johnson, Valen E.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2013-11-11
- Volume
- 110
- Issue
- 48
- Pages
- 19313-19317
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC3845140
- OpenAlex
- W1970588407
- Cited by
- 808
- References
- 22
- Citations by year
- 2026: 11 · 2025: 17 · 2024: 30 · 2023: 52 · 2022: 48 · 2021: 55 · 2020: 52 · 2019: 67 · 2018: 94 · 2017: 115 · 2016: 94 · 2015: 86 · 2014: 85 · 2013: 2
- FWCI
- 78.55
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- OakTrust (Texas A&M University Libraries) (submittedVersion)
- Funders
- National Cancer Institute (R01CA158113)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- Bayesian hypothesis testing, Bayes factor, p-values, significance level, reproducibility, evidence thresholds
- MeSH terms
- Bayes Theorem, Statistics as Topic, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
