Redefine statistical significance
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Of the 97 original positive effects, 14% (95% CI 5-24%) had porig < 0.005.
“Of the 97 original positive effects, 42% (95% CI: [28%, 56%]) had porig < 0.05, and 14% (95% CI: [5%, 24%]) had porig < 0.005 (Benjamin et al., 2018).”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Benjamin et al., 2018
The “Why 0.005” section discusses replication rates for studies grouped by their original P values, but does not report 97 original positive effects or the claimed 14% (95% CI 5–24%). Searches for those figures and for original-study P values found no passage in our copy that verifies the claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe use of statistical methods has been demonstrated to be suboptimal in some fields.
“The use of statistical methods has been demonstrated to be suboptimal in some fields.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [21]
Over-reliance on P values with lenient thresholds (such as P < 0.05) has high prevalence.
“the over-reliance on P values with lenient thresholds (such as P < 0.05)21”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
The ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · see citing works
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · see citing works
Self-citationEvolution of Reporting <i>P</i> Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370 · see citing works
Self-citationMeta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364 · see citing works
Self-citationUsing prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- Johnson, V. E. Proc. Natl Acad. Sci. USA 110, 19313-19317 (2013).
- Dreber, A. et al. Proc. Natl Acad. Sci. USA 112, 15343-15347 (2015).
- Szucs, D. & Ioannidis, J. P. A. PLoS Biol. 15, e2000797 (2017).
- Fanelli, D., Costas, R. & Ioannidis, J. P. A. Proc. Natl Acad. Sci. USA 114, 3714-3719 (2017).
Cited by
Sorted by citations
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · see citing works
Self-citationMany Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
Self-citationIndices of Effect Existence and Significance in the Bayesian Framework
Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02767 · Cited by 983 · see citing works
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Self-citationRejecting or Accepting Parameter Values in Bayesian Estimation
Kruschke, J. K. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771304 · Cited by 734 · see citing works
Lederer, D. J., Bell, S. C., Branson, R. D., Chalmers, J. D., Marshall, R., Maslove, D. M., Ost, D. E., Punjabi, N. M., Schatz, M., Smyth, A. R., et al. (2019) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201808-564ps · Cited by 715 · see citing works
Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence
Keysers, C., Gazzola, V., & Wagenmakers, E.-J. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0660-4 · Cited by 654 · see citing works
Self-citationA meta-analysis of procedures to change implicit measures
Forscher, P. S., Lai, C. K., Axt, J. R., Ebersole, C. R., Herman, M., Devine, P. G., & Nosek, B. A. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000160 · Cited by 700 · see citing works
Self-citationYarkoni, T. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20001685 · Cited by 701 · see citing works
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 393 · see citing works
Self-citationStatistical Inference as Severe Testing: How to Get Beyond the Statistics Wars
Mayo, D. G. (2018) · doi:10.1017/9781107286184 · Cited by 376 · see citing works
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371 · see citing works
Chen, C.-F., Zarazua de Rubens, G., Noel, L., Kester, J., & Sovacool, B. K. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2019.109692 · Cited by 314 · see citing works
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Kohavi, R., Tang, D., & Xu, Y. (2020) · doi:10.1017/9781108653985 · Cited by 297 · see citing works
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · see citing works
Rice, D. B., Raffoul, H., Ioannidis, J. P. A., & Moher, D. (2020) · BMJ · doi:10.1136/bmj.m2081 · Cited by 241 · see citing works
Self-citationReproducibility of preclinical animal research improves with heterogeneity of study samples
Voelkl, B., Vogt, L., Sena, E. S., & Würbel, H. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2003693 · Cited by 262 · see citing works
Rafi, Z., & Greenland, S. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01105-9 · Cited by 243 · see citing works
Ten common statistical mistakes to watch out for when writing or reviewing a manuscript
Makin, T. R., & Orban de Xivry, J.-J. (2019) · eLife · doi:10.7554/elife.48175 · Cited by 238 · see citing works
The power of predictions: An emerging paradigm for psychological research
Hutchinson, J. B., & Barrett, L. F. (2019) · Current Directions in Psychological Science · doi:10.1177/0963721419831992 · Cited by 216 · see citing works
Practical guide to the meta-analysis of rare events
Efthimiou, O. (2018) · Evidence Based Mental Health · doi:10.1136/eb-2018-102911 · Cited by 231 · see citing works
Interactional synchrony: signals, mechanisms and benefits
Hoehl, S., Fairhurst, M., & Schirmer, A. (2021) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsaa024 · Cited by 231 · see citing works
Evidence-based umbrella review of 162 peripheral biomarkers for major mental disorders
Carvalho, A. F., Solmi, M., Sanches, M., Machado, M. O., Stubbs, B., Ajnakina, O., Sherman, C., Sun, Y. R., Liu, C. S., Brunoni, A. R., et al. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0835-5 · Cited by 204 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
Umbrella reviews: what they are and why we need them
Papatheodorou, S. (2019) · European Journal of Epidemiology · doi:10.1007/s10654-019-00505-6 · Cited by 201 · see citing works
Nonreplicable publications are cited more than replicable ones
Serra-Garcia, M., & Gneezy, U. (2021) · Science Advances · doi:10.1126/sciadv.abd1705 · Cited by 177 · see citing works
Ioannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175 · see citing works
Self-citationThe Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137 · see citing works
Transparency and reproducibility of meta-analyses in psychology: A meta-review
Polanin, J. R., Hennessy, E. A., & Tsuji, S. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620906416 · Cited by 100 · see citing works
Bendtsen, M. (2018) · Journal of Medical Internet Research · doi:10.2196/10873 · Cited by 105 · see citing works
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105 · see citing works
Self-citationKuzovkin, I., Vicente, R., Petton, M., Lachaux, J.-P., Baciu, M., Kahane, P., Rheims, S., Vidal, J. R., & Aru, J. (2018) · Communications Biology · doi:10.1038/s42003-018-0110-y · Cited by 104 · see citing works
Horstmann, K. T., Rauthmann, J. F., Sherman, R. A., & Ziegler, M. (2021) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000357 · Cited by 89 · see citing works
Wan, Y., Tobias, D. K., Dennis, K. K., Guasch-Ferré, M., Sun, Q., Rimm, E. B., Hu, F. B., Ludwig, D. S., Devinsky, O., & Willett, W. C. (2023) · BMJ · doi:10.1136/bmj-2022-073939 · Cited by 64 · see citing works
The role of strategy use in working memory training outcomes
Fellman, D., Jylkkä, J., Waris, O., Soveri, A., Ritakallio, L., Haga, S., Salmi, J., Nyman, T. J., & Laine, M. (2020) · Journal of Memory and Language · doi:10.1016/j.jml.2019.104064 · Cited by 64 · see citing works
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61 · see citing works
Alexithymia may explain the relationship between autistic traits and eating disorder psychopathology
Vuillier, L., Carter, Z., Teixeira, A. R., & Moseley, R. L. (2020) · Molecular Autism · doi:10.1186/s13229-020-00364-z · Cited by 48 · see citing works
Weiss, B., Nygart, V., Pommerencke, L. M., Carhart-Harris, R. L., & Erritzoe, D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.749788 · Cited by 47 · see citing works
The Registered Reports Revolution Lessons in Cultural Reform
Chambers, C. (2019) · Significance · doi:10.1111/j.1740-9713.2019.01299.x · Cited by 43 · see citing works
Self-citationAbstinence from masturbation and hypersexuality
Zimmer, F., & Imhoff, R. (2020) · Archives of Sexual Behavior · doi:10.1007/s10508-019-01623-8 · Cited by 46 · see citing works
Pudney, E. V., Himmelstein, M. S., Puhl, R. M., & Foster, G. D. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.112854 · Cited by 37 · see citing works
Han, H., Glenn, A. L., & Dawson, K. J. (2019) · Brain Sciences · doi:10.3390/brainsci9080198 · Cited by 34 · see citing works
Longitudinal cardio-respiratory fitness prediction through wearables in free-living environments
Spathis, D., Perez-Pozuelo, I., Gonzales, T. I., Wu, Y., Brage, S., Wareham, N., & Mascolo, C. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00719-1 · Cited by 30 · see citing works
Ziermans, T., de Bruijn, Y., Dijkhuis, R., Staal, W., & Swaab, H. (2019) · Autism · doi:10.1177/1362361318817716 · Cited by 28 · see citing works
Mindfulness for female outpatients with chronic primary headaches: an internet-based bibliotherapy
Tavallaei, V., Rezapour-Mirsaleh, Y., Rezaiemaram, P., & Saadat, S. H. (2018) · European Journal of Translational Myology · doi:10.4081/ejtm.2018.7380 · Cited by 23 · see citing works
- Authors
- Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., Cesarini, D., Chambers, C. D., Clyde, M., Cook, T. D., De Boeck, P., Dienes, Z., Dreber, A., Easwaran, K., Efferson, C., Fehr, E., Fidler, F., Field, A. P., Forster, M., George, E. I., Gonzalez, R., Goodman, S., Green, E., Green, D. P., Greenwald, A. G., Hadfield, J. D., Hedges, L. V., Held, L., Hua Ho, T., Hoijtink, H., Hruschka, D. J., Imai, K., Imbens, G., Ioannidis, J. P. A., Jeon, M., Jones, J. H., Kirchler, M., Laibson, D., List, J., Little, R., Lupia, A., Machery, E., Maxwell, S. E., McCarthy, M., Moore, D. A., Morgan, S. L., Munafó, M., Nakagawa, S., Nyhan, B., Parker, T. H., Pericchi, L., Perugini, M., Rouder, J., Rousseau, J., Savalei, V., Schönbrodt, F. D., Sellke, T., Sinclair, B., Tingley, D., Van Zandt, T., Vazire, S., Watts, D. J., Winship, C., Wolpert, R. L., Xie, Y., Young, C., Zinman, J., & Johnson, V. E.
- Affiliations
- University of Southern California (US)
- Duke University (US)
- Stockholm School of Economics (SE)
- Center for Open Science (US)
- University of Amsterdam (NL)
- University of Pennsylvania (US)
- University of North Carolina at Chapel Hill (US)
- University of Regensburg (DE)
- California Institute of Technology (US)
- New York University (US)
- Northwestern University (US)
- University of Sussex (GB)
- Texas A&M University (US)
- Royal Holloway University of London (GB)
- University of Zurich (CH)
- The University of Melbourne (AU)
- University of Wisconsin–Madison (US)
- University of Michigan (US)
- Stanford University (US)
- Rutgers, The State University of New Jersey (US)
- Columbia University (US)
- University of Washington (US)
- University of Edinburgh (GB)
- Utrecht University (NL)
- Arizona State University (US)
- Princeton University (US)
- University of California, Los Angeles (US)
- Imperial College London (GB)
- Universität Innsbruck (AT)
- Harvard University (US)
- University of Chicago (US)
- University of Notre Dame (US)
- University of California, Berkeley (US)
- Johns Hopkins University (US)
- UNSW Sydney (AU)
- Dartmouth College (US)
- Whitman College (US)
- University of Puerto Rico at Río Piedras (PR)
- University of Milano-Bicocca (IT)
- University of California, Irvine (US)
- Université Paris Dauphine-PSL (FR)
- University of British Columbia (CA)
- Ludwig-Maximilians-Universität München (DE)
- Purdue University West Lafayette (US)
- Washington University in St. Louis (US)
- University of California, Davis (US)
- Microsoft Research New York City (United States) (US)
- Corresponding author
- Benjamin, Daniel J.
- Johannesson, Magnus
- Johnson, Valen E.
- ORCID
- Benjamin, Daniel J.
- Berger, James O.
- Johannesson, Magnus
- Nosek, Brian A.
- Wagenmakers, E.-J.
- Berk, Richard
- Bollen, Kenneth A.
- Brembs, Björn
- Camerer, Colin
- Cesarini, David
- Clyde, Merlise
- Cook, Thomas D.
- Dienes, Zoltan
- Dreber, Anna
- Easwaran, Kenny
- Efferson, Charles
- Fehr, Ernst
- Fidler, Fiona
- Field, Andy P.
- Gonzalez, Richard
- Goodman, Steven
- Green, Donald P.
- Greenwald, Anthony G.
- Hadfield, Jarrod D.
- Hedges, Larry V.
- Held, Leonhard
- Hoijtink, Herbert
- Hruschka, Daniel J.
- Imai, Kosuke
- Imbens, Guido
- Ioannidis, John P. A.
- Jeon, Minjeong
- Jones, James Holland
- Kirchler, Michael
- Laibson, David
- List, John
- Little, Roderick
- Lupia, Arthur
- Maxwell, Scott E.
- McCarthy, Michael
- Moore, Don A.
- Morgan, Stephen L.
- Nakagawa, Shinichi
- Nyhan, Brendan
- Pericchi, Luis
- Perugini, Marco
- Rousseau, Judith
- Savalei, Victoria
- Schönbrodt, Felix D.
- Sinclair, Betsy
- Tingley, Dustin
- Vazire, Simine
- Watts, Duncan J.
- Winship, Christopher
- Wolpert, Robert L.
- Xie, Yu
- Johnson, Valen E.
- Venue
- Nature Human Behaviour
- Published
- 2017-09-01
- Volume
- 2
- Issue
- 1
- Pages
- 6-10
- ISSN
- 2397-3374 (ISSN-L)
- OpenAlex
- W2736848882
- Cited by
- 2,772
- References
- 6
- Citations by year
- 2026: 165 · 2025: 251 · 2024: 235 · 2023: 280 · 2022: 330 · 2021: 390 · 2020: 427 · 2019: 364 · 2018: 280 · 2017: 44 · 2016: 1 · 2015: 2
- FWCI
- 273.32
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Nature Human Behaviour (publishedVersion)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion, other-oa)
- ORCA Online Research @Cardiff (Cardiff University) (acceptedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, other-oa)
- Explore Bristol Research (acceptedVersion, other-oa)
- Sussex Research Online (University of Sussex) (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- BOA (University of Milano-Bicocca) (submittedVersion, other-oa)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- ORCA Online Research @Cardiff (Cardiff University) (submittedVersion)
- IRIS (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
- SERVAL (Université de Lausanne) (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- University of Southern California
- Columbia University
- Princeton University
- University of Pennsylvania
- Ohio State University
- California Institute of Technology
- Northwestern University
- Harvard University
- Arizona State University
- University of Washington (WA 98195)
- Johns Hopkins University
- University of Pittsburgh
- University of Notre Dame
- Dartmouth College
- Università degli Studi di Milano
- York University
- Imperial College London
- University of Sussex
- University of Edinburgh
- Cardiff University
- University of Bristol
- National University of Singapore
- University of New South Wales
- Universiteit van Amsterdam
- Universiteit Utrecht
- Universität Regensburg
- Göteborgs Universitet
- Universität Zürich
- Universität Innsbruck
- Directorate for Biological Sciences
- University of California, Los Angeles
- University of California, Irvine
- Wharton School, University of Pennsylvania
- Weinberg College of Arts and Sciences, Northwestern University
- School of Human Evolution and Social Change, Arizona State University
- Medical Research Council
- Royal Holloway, University of London
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- statistical significance, p-value threshold, new discoveries, reproducibility, research credibility, hypothesis testing
- MeSH terms
- Scholarly Communication, Data Interpretation, Statistical, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
