When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Open accessPublished version of a preprint
Abstract
Null hypothesis significance testing (NHST) has several shortcomings that are likely contributing factors behind the widely debated replication crisis of (cognitive) neuroscience, psychology, and biomedical science in general. We review these shortcomings and suggest that, after sustained negative experience, NHST should no longer be the default, dominant statistical practice of all biomedical and psychological research. If theoretical predictions are weak we should not rely on all or nothing hypothesis tests. Different inferential methods may be most suitable for different types of research questions. Whenever researchers use NHST they should justify its use, and publish pre-study power calculations and effect sizes, including negative findings. Hypothesis-testing studies should be pre-registered and optimally raw data published. The current statistics lite educational approach for students that has sustained the widespread, spurious use of NHST should be phased out.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In cases where complications arise, analysing the data with multiple methods can test the robustness of conclusions.
“In cases where complications arise it can be extremely useful to analyse the data with multiple methods to test the robustness of conclusions24,126.”
From A phylogenetic approach to comparative genomics (2025) · cites this paper as [126]
In “NHST may foster selective reporting and subjectivity” (page unavailable), the paper says different analytical choices can produce a range of results, including opposite effects. “Ways to change” (page unavailable) warns that small changes in analysis settings can alter statistical outcomes. These passages support checking whether conclusions depend on analytical choices, but the paper does not specifically recommend analysing complicated data with multiple methods to test robustness. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe use of statistical methods has been demonstrated to be suboptimal in some fields.
“The use of statistical methods has been demonstrated to be suboptimal in some fields.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [19]
Misuse and misinterpretation of null-hypothesis testing methods have high prevalence.
“Major threats that have high prevalence include the misuse and misinterpretation of null-hypothesis testing methods19,20”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) · doi:10.1201/b16018 · Cited by 27,276 · see citing works
(1988) · Cited by 16,582 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Self-citationThe control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · see citing works
Statistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 2× · see citing works
“, 2014) are in a crisis of producing too many false positive non-replicable results (Begley and Ellis, 2012; Aarts et al., 2015).”
BackgroundStatistical Decision Theory and Bayesian Analysis
Berger, J. O. (1985) · Springer Series in Statistics · doi:10.1007/978-1-4757-4286-2 · Cited by 7,722 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 6× · see citing works
“While power in principle can be adjusted easily by increasing sample size, power in many/most fields of biomedical science and psychology has been notoriously low and the situation has not improved much during the past 50 years (Cohen, 1962; Sedlmeier and Gigerenzer, 1989; Rossi, 1990; Hallahan and Rosenthal, 1996; Bu…”
BackgroundInfluentialSelf-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Information theory, inference, and learning algorithms
(2004) · Choice Reviews Online · doi:10.5860/choice.41-5949 · Cited by 6,535 · see citing works
The ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 6× · see citing works
“, the recent American Statistical Association statement on p-values (Wasserstein and Lazar, 2016) was accompanied by 21 editorials from the statisticians and methodologists who participated in crafting it and who disagreed in different aspects among themselves.”
BackgroundInfluentialJaeschke, R., Singer, J., & Guyatt, G. H. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90005-6 · Cited by 4,982 · see citing works
Information Theory, Inference, and Learning Algorithms
(2004) · Kybernetes · doi:10.1108/03684920410534506 · Cited by 4,937 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Self-citationProbability theory the logic of science
Jaynes, E. T., & Bretthorst, G. L. (2003) · Cited by 4,600 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
Probability Theory: The Logic of Science
Jaynes, E. T. (2003) · doi:10.1017/cbo9780511790423 · Cited by 4,045 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates
Eklund, A., Nichols, T. E., & Knutsson, H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1602413113 · Cited by 3,487 · see citing works
Probabilistic Reasoning in Intelligent Systems
(1988) · Elsevier eBooks · doi:10.1016/c2009-0-27609-4 · Cited by 3,371 · see citing works
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696 · see citing works
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · see citing works
IX. On the problem of the most efficient tests of statistical hypotheses
Neyman, J., & Pearson, E. S. (1933) · Philosophical Transactions of the Royal Society of London Series A Containing Papers of a Mathematical or Physical Character · doi:10.1098/rsta.1933.0009 · Cited by 2,370 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Can cognitive processes be inferred from neuroimaging data?
POLDRACK, R. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.12.004 · Cited by 2,230 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
Shaffer, J. P. (1995) · Annual Review of Psychology · doi:10.1146/annurev.ps.46.020195.003021 · Cited by 1,960 · see citing works
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.806 · Cited by 1,952 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · see citing works
Self-citationON THE PROBLEM OF THE MOST EFFICIENT TESTS OF STATISTICAL HYPOTHESES
Neyman, J., & Pearson, E. S. (1967) · University of California Press eBooks · doi:10.1525/9780520339897-008 · Cited by 1,545 · see citing works
On the relative motion of the Earth and the luminiferous ether
Michelson, A. A., & Morley, E. W. (1887) · American Journal of Science · doi:10.2475/ajs.s3-34.203.333 · Cited by 1,521 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · see citing works
Self-citationPuzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · cited 4× · see citing works
“This is particularly a problem in neuroimaging where the complexity and idiosyncrasy of analyses is such that it is usually impossible to replicate exactly what happened and why during data analysis (Kriegeskorte et al., 2009; Vul et al., 2009; Carp, 2012).”
BackgroundMethodInfluentialSterne, J. A. C. (2001) · BMJ · doi:10.1136/bmj.322.7280.226 · Cited by 1,466 · see citing works
The SAGE Handbook of Quantitative Methodology for the Social Sciences
Kaplan, D. (2004) · doi:10.4135/9781412986311 · Cited by 1,454 · see citing works
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · see citing works
A Selected Core Microbiome Drives the Early Stages of Three Popular Italian Cheese Manufactures
De Filippis, F., La Storia, A., Stellato, G., Gatti, M., & Ercolini, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089680 · Cited by 1,398 · see citing works
Hewson, P. (2014) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2014.12096_1.x · Cited by 1,368 · see citing works
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological bulletin · doi:10.1037/h0026141 · Cited by 1,363 · see citing works
Replicating genotype–phenotype associations
NCI-NHGRI Working Group on Replication in Association Studies, Chanock, S. J., Manolio, T., Boehnke, M., Boerwinkle, E., Hunter, D. J., Thomas, G., Hirschhorn, J. N., Abecasis, G., Altshuler, D., et al. (2007) · Nature · doi:10.1038/447655a · Cited by 1,352 · see citing works
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
Why We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274 · see citing works
Controlling the familywise error rate in functional neuroimaging: a comparative review
Nichols, T. E., & Hayasaka, S. (2003) · Statistical Methods in Medical Research · doi:10.1191/0962280203sm341ra · Cited by 1,210 · see citing works
Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,188 · see citing works
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090 · see citing works
Schmidt, F. L. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.115 · Cited by 1,031 · see citing works
Biomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003 · see citing works
Self-citationSivia, D. S. (2006) · doi:10.1093/oso/9780198568315.001.0001 · Cited by 1,003 · see citing works
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 971 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 3× · see citing works
“However, a is always fixed independently of what p value we find in an experiment whereas p values can be considered random variables, varying widely from experiment to experiment (Murdoch et al. 2008; Hung et al. 1997; Simonsohn et al. 2014a,b; Sterling 1959).”
BackgroundThe Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963 · see citing works
Theory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962 · see citing works
The test of significance in psychological research.
Bakan, D. (1966) · Psychological Bulletin · doi:10.1037/h0020412 · Cited by 957 · see citing works
WHY SUMMARIES OF RESEARCH ON PSYCHOLOGICAL THEORIES ARE OFTEN UNINTERPRETABLE
Meehl, P. E. (1990) · Psychological Reports · doi:10.2466/pr0.1990.66.1.195 · Cited by 901 · see citing works
Gigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890 · see citing works
Calibration of ρ Values for Testing Precise Null Hypotheses
Sellke, T., Bayarri, M. J., & Berger, J. O. (2001) · The American Statistician · doi:10.1198/000313001300339950 · Cited by 856 · see citing works
Boutron, I. (2010) · JAMA · doi:10.1001/jama.2010.651 · Cited by 842 · see citing works
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805 · see citing works
Gigerenzer, G., Swijtink, Z. G., Porter, T. M., Daston, L., Beatty, J., & Krüger, L. (1989) · Cambridge University Press eBooks · doi:10.1017/cbo9780511720482 · Cited by 770 · see citing works
Szucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761 · see citing works
Self-citationHow to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760 · see citing works
Self-citationAn exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · see citing works
Self-citationReplications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750 · see citing works
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Multiple comparisons: philosophies and illustrations
Curran-Everett, D. (2000) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.2000.279.1.r1 · Cited by 743 · see citing works
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 738 · see citing works
<i>p</i> -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717 · see citing works
Best practices in data analysis and sharing in neuroimaging using MRI
Nichols, T. E., Das, S., Eickhoff, S. B., Evans, A. C., Glatard, T., Hanke, M., Kriegeskorte, N., Milham, M. P., Poldrack, R. A., Poline, J.-B., et al. (2017) · Nature Neuroscience · doi:10.1038/nn.4500 · Cited by 714 · see citing works
Wakefield's article linking MMR vaccine and autism was fraudulent
Godlee, F., Smith, J. M., & Marcovitch, H. (2011) · BMJ · doi:10.1136/bmj.c7452 · Cited by 700 · see citing works
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697 · see citing works
A dirty dozen: twelve p-value misconceptions.
Goodman, S. (2008) · Seminars in hematology (Print) · doi:10.1053/j.seminhematol.2008.04.003 · Cited by 694 · see citing works
Berger, J. O., & Delampady, M. (1987) · Statistical Science · doi:10.1214/ss/1177013238 · Cited by 664 · see citing works
InformationTheory, Inference, and Learning Algorithms
Mackay, D. J. C. (2003) · Cited by 650 · see citing works
Trafimow, D., & Marks, M. (2015) · Basic and Applied Social Psychology · doi:10.1080/01973533.2015.1012991 · Cited by 612 · see citing works
Statistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials
Moher, D. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020048013 · Cited by 608 · see citing works
The Empire of chance: how probability changed science and everyday life
(1989) · Choice Reviews Online · doi:10.5860/choice.27-2084 · Cited by 600 · see citing works
Meta-analysis methods for genome-wide association studies and beyond
Evangelou, E., & Ioannidis, J. P. A. (2013) · Nature Reviews Genetics · doi:10.1038/nrg3472 · Cited by 596 · see citing works
Self-citationThe fallacy of placing confidence in confidence intervals
Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E.-J. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0947-8 · Cited by 577 · cited 1× · see citing works
“…of effects along with estimates of uncertainty, let these be error terms or Bayesian credible intervals (Luce, 1988; Edwards, 1972; Jaynes 2003; Schmidt, 1996; Gelman 2013; Lakens & Evers, 2014; see Morey et al. 2016 for why classical confidence intervals are not appropriate uncertainty measures).”
BackgroundInterim analysis: The alpha spending function approach
DeMets, D. L., & Lan, K. K. G. (1994) · Statistics in Medicine · doi:10.1002/sim.4780131308 · Cited by 565 · see citing works
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 2× · see citing works
“Moreover, such published false positive true H0 studies will also inevitably overestimate the effect size of the non-existent effects or of existent, but unimportantly tiny, effects (Schmidt, 1992; 1996; Sterling et al. 1995; Ioannidis 2008).”
BackgroundWhy Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · see citing works
Self-citation(1992) · doi:10.1142/1588 · Cited by 550 · see citing works
Testing Statistical Hypotheses of Equivalence and Noninferiority
Wellek, S. (2010) · doi:10.1201/ebk1439808184 · Cited by 548 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524 · see citing works
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · cited 1× · see citing works
“Publishing scripts will also provide more information on potential statistical errors (Bakker and Wicherts, 2001; Nuijten et al., 2016).”
BackgroundHow citation distortions create unfounded authority: analysis of a citation network
Greenberg, S. A. (2009) · BMJ · doi:10.1136/bmj.b2680 · Cited by 499 · see citing works
Slobbe, L., Doorduijn, J. K., Lugtenburg, P. J., el Barzouhi, A., Boersma, E. H., van Leeuwen, W. B., & Rijnders, B. J. A. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010840 · Cited by 491 · see citing works
The principled control of false positives in neuroimaging
Bennett, C. M., Wolford, G. L., & Miller, M. B. (2009) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsp053 · Cited by 486 · see citing works
Data Analysis: A Bayesian Tutorial
Fox, E. P., & Sivia, D. S. (1998) · Technometrics · doi:10.2307/1270652 · Cited by 472 · see citing works
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
The secret lives of experiments: Methods reporting in the fMRI literature
Carp, J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.07.004 · Cited by 447 · see citing works
Robust misinterpretation of confidence intervals
Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0572-3 · Cited by 425 · see citing works
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 414 · see citing works
Self-citationStatistical Inference: A Commentary for the Social and Behavioural Sciences
Oakes, M. W. (1986) · Medical Entomology and Zoology · Cited by 392 · see citing works
Khoury, M. J., & Ioannidis, J. P. A. (2014) · Science · doi:10.1126/science.aaa2709 · Cited by 390 · see citing works
Self-citationData Analysis A Bayesian Tutorial
Sivia, D. S., & Skilling, J. (2016) · Choice Reviews Online · doi:10.5860/choice.34-5742 · Cited by 382 · see citing works
Type S error rates for classical and Bayesian single and multiple comparison procedures
Gelman, A., & Tuerlinckx, F. (2000) · Computational Statistics · doi:10.1007/s001800000040 · Cited by 380 · cited 3× · see citing works
“, Wellek, 2010) or researchers may check for the sign of expected effects (Gelman and Tuerlinckx, 2000).”
BackgroundMethodInfluentialMultiple hypothesis testing in genomics
Goeman, J., & Solari, A. (2014) · Statistics in Medicine · doi:10.1002/sim.6082 · Cited by 377 · see citing works
Evolution of Reporting <i>P</i> Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370 · see citing works
Self-citationBoutron, I., Altman, D. G., Hopewell, S., Vera-Badillo, F., Tannock, I., & Ravaud, P. (2014) · Journal of Clinical Oncology · doi:10.1200/jco.2014.56.7503 · Cited by 368 · see citing works
Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Kaplan, R. M., & Irvin, V. L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132382 · Cited by 356 · see citing works
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · see citing works
How the case against the MMR vaccine was fixed
Deer, B. (2011) · BMJ · doi:10.1136/bmj.c5347 · Cited by 349 · see citing works
“A 30% Chance of Rain Tomorrow”: How Does the Public Understand Probabilistic Weather Forecasts?
Gigerenzer, G., Hertwig, R., van den Broek, E., Fasolo, B., & Katsikopoulos, K. V. (2005) · Risk Analysis · doi:10.1111/j.1539-6924.2005.00608.x · Cited by 343 · see citing works
The Cult of Statistical Significance
McCLOSKEY, D. N., & Ziliak, S. T. (2008) · University of Michigan Press eBooks · doi:10.3998/mpub.186351 · Cited by 341 · see citing works
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.58.5.646 · Cited by 323 · see citing works
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323 · see citing works
The False-positive to False-negative Ratio in Epidemiologic Studies
Ioannidis, J. P. A., Tarone, R., & McLaughlin, J. K. (2011) · Epidemiology · doi:10.1097/ede.0b013e31821b506e · Cited by 321 · see citing works
Self-citationThe Case Against Statistical Significance Testing, Revisited
Carver, R. P. (1993) · The Journal of Experimental Education · doi:10.1080/00220973.1993.10806591 · Cited by 319 · see citing works
Some Difficulties of Interpretation Encountered in the Application of the Chi-Square Test
Berkson, J. (1938) · Journal of the American Statistical Association · doi:10.1080/01621459.1938.10502329 · Cited by 317 · see citing works
Assessing the validity of the Global Activity Limitation Indicator in fourteen European countries
Berger, N., Oyen, H. J. V., Cambois, E., Fouweather, T., Jagger, C., Nusselder, W. J., & Robine, J. (2015) · BMC Medical Research Methodology · doi:10.1186/1471-2288-15-1 · Cited by 317 · see citing works
A Bayesian perspective on the bonferroni adjustment
Westfall, P. (1997) · Biometrika · doi:10.1093/biomet/84.2.419 · Cited by 316 · see citing works
Needed: A Ban on the Significance Test
Hunter, J. E. (1997) · Psychological Science · doi:10.1111/j.1467-9280.1997.tb00534.x · Cited by 314 · see citing works
Goodman, S. N. (1993) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116700 · Cited by 310 · see citing works
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295 · cited 1× · see citing works
“Looking at these numbers the replication crisis does not seem surprising: using NHST very high FRP can be expected even with modestly high H0:H1 odds and moderate bias (Etz and Vandekerckhove, 2016).”
BackgroundConfusion Over Measures of Evidence (p's) Versus Errors (α's) in Classical Statistical Testing
Hubbard, R., & Bayarri, M. J. (2003) · The American Statistician · doi:10.1198/0003130031856 · Cited by 288 · see citing works
Panagiotou, O. A., Ioannidis, J. P. A., & for the Genome-Wide Significance Project (2012) · International Journal of Epidemiology · doi:10.1093/ije/dyr178 · Cited by 286 · see citing works
Self-citationThe Behavior of the P-Value When the Alternative Hypothesis is True
Hung, H., O’Neill, R. T., Bauer, P., & Kohne, K. (1997) · Biometrics · doi:10.2307/2533093 · Cited by 281 · see citing works
Significance Tests Die Hard: The Amazing Persistence of a Probabilistic Misconception
Falk, R., & Greenbaum, C. W. (1995) · Theory & Psychology · doi:10.1177/0959354395051004 · Cited by 279 · see citing works
Patel, C. J., Burford, B. J., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.029 · Cited by 276 · see citing works
Self-citationLakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266 · see citing works
Some Difficulties of Interpretation Encountered in the Application of the Chi-Square Test
Berkson, J. (1938) · Journal of the American Statistical Association · doi:10.2307/2279690 · Cited by 259 · see citing works
Reproducible research and Biostatistics
Peng, R. D. (2009) · Biostatistics · doi:10.1093/biostatistics/kxp014 · Cited by 244 · see citing works
Sotos, A. E. C., Vanhoof, S., den Noortgate, W. V., & Onghena, P. (2007) · Educational Research Review · doi:10.1016/j.edurev.2007.04.001 · Cited by 239 · see citing works
Dirty Dozen: Twelve P-Value Misconceptions
Goodman, T. (2008) · Cited by 238 · see citing works
Gelman, A. (2012) · Epidemiology · doi:10.1097/ede.0b013e31827886f7 · Cited by 216 · see citing works
Gigerenzer, G., & Marewski, J. N. (2014) · Journal of Management · doi:10.1177/0149206314547522 · Cited by 215 · see citing works
Reproducible Research: Moving toward Research the Public Can Really Trust
Laine, C., Goodman, S. N., Griswold, M. E., & Sox, H. C. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-146-6-200703200-00154 · Cited by 213 · see citing works
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210 · see citing works
Self-citationStuart, A., & Edwards, A. W. F. (1973) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2346929 · Cited by 201 · see citing works
Is everything we eat associated with cancer? A systematic cookbook review
Schoenfeld, J. D., & Ioannidis, J. P. A. (2012) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.112.047142 · Cited by 199 · see citing works
Self-citationStatistical methods and scientific inference
Fisher, R. A. (1956) · CERN Document Server (European Organization for Nuclear Research) · Cited by 196 · see citing works
The Place of Statistics in Psychology
Nunnally, J. C. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000401 · Cited by 195 · see citing works
Citation bias favoring statistically significant studies was present in medical research
Jannot, A.-S., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.015 · Cited by 193 · see citing works
The null ritual : What you always wanted to know about significance testing but were afraid to ask
Gigerenzer, G., Krauss, S., & Vitouch, O. (2004) · Sage eBooks · doi:10.4135/9781412986311.n21 · Cited by 191 · see citing works
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.46.4.806 · Cited by 188 · see citing works
Simultaneous and selective inference: Current successes and future challenges
Benjamini, Y. (2010) · Biometrical Journal · doi:10.1002/bimj.200900299 · Cited by 169 · see citing works
I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data
Hosseini, M., Powell, M., Collins, J., Callahan-Flintoft, C., Jones, W., Bowman, H., & Wyble, B. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.09.036 · Cited by 165 · see citing works
The Null Hypothesis Testing Controversy in Psychology
Krantz, D. H. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473888 · Cited by 158 · see citing works
On the probability of making Type I errors.
Pollard, P., & Richardson, J. T. E. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.102.1.159 · Cited by 148 · see citing works
Best Practices in Data Analysis and Sharing in Neuroimaging using MRIPreprint
Nichols, T. E., Das, S., Eickhoff, S. B., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/054262 · Cited by 140 · see citing works
Problems With Null Hypothesis Significance Testing (NHST): What Do the Textbooks Say?
Gliner, J. A., Leech, N. L., & Morgan, G. A. (2002) · The Journal of Experimental Education · doi:10.1080/00220970209602058 · Cited by 136 · see citing works
Brajković, A. V., Hren, D., Sambunjak, D., Vodopivec, I., Ivaniš, A., Marušić, A., & Marušić, M. (2010) · Journal of Investigative Medicine · doi:10.2310/jim.0b013e3181cb42d9 · Cited by 135 · see citing works
Marušić, A., & Marušić, M. (2003) · Academic Medicine · doi:10.1097/00001888-200312000-00007 · Cited by 135 · see citing works
Murdoch, D. J., Tsai, Y.-L., & Adcock, J. (2008) · The American Statistician · doi:10.1198/000313008x332421 · Cited by 130 · see citing works
Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses
Bayarri, M. J., Benjamin, D. J., Berger, J. O., & Sellke, T. M. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.12.007 · Cited by 126 · see citing works
Studying the Elusive Environment in Large Scale
Patel, C. J., & Ioannidis, J. P. A. (2014) · JAMA · doi:10.1001/jama.2014.4129 · Cited by 125 · see citing works
Self-citationThe Connection Between Varying Treatment Effects and the Crisis of Unreplicable Research
Gelman, A. (2015) · Journal of Management · doi:10.1177/0149206314525208 · Cited by 125 · see citing works
Selection in Reported Epidemiological Risks: An Empirical Assessment
Kavvoura, F. K., Liberopoulos, G., & Ioannidis, J. P. A. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040079 · Cited by 114 · see citing works
Self-citationScientific Utopia: II. Restructuring incentives and practices to promote truth over publishabilityPreprint
Nosek, B. A., Spies, J. R., & Motyl, M. S. (2016) · doi:10.31219/osf.io/k4bgq · Cited by 114 · see citing works
Gelman, A., & Loken, E. (2014) · American Scientist · Cited by 111 · see citing works
Placing epidemiological results in the context of multiplicity and typical correlations of exposures
Patel, C. J., & Ioannidis, J. P. A. (2014) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2014-204195 · Cited by 110 · see citing works
Self-citationDoes parametric fMRI analysis with SPM yield valid results?—An empirical study of 1484 rest datasets
Eklund, A., Andersson, M., Josephson, C., Johannesson, M., & Knutsson, H. E. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.093 · Cited by 108 · see citing works
Statistical power: Concepts, procedures, and applications
Hallahan, M., & Rosenthal, R. (1996) · Behaviour Research and Therapy · doi:10.1016/0005-7967(95)00082-8 · Cited by 89 · see citing works
Evaluation of New Treatments in Radiation Oncology
Soares, H. P., Kumar, A., Daniels, S., Swann, S., Cantor, A. B., Hozo, I., Clark, M., Serdarević, F., Gwede, C. K., Trotti, A., et al. (2005) · JAMA · doi:10.1001/jama.293.8.970 · Cited by 85 · see citing works
Yates, F. (1951) · Journal of the American Statistical Association · doi:10.2307/2280090 · Cited by 84 · see citing works
The Analysis of Experimental Data: The Appreciation of Tea and Wine
Lindley, D. V. (1993) · Teaching Statistics · doi:10.1111/j.1467-9639.1993.tb00252.x · Cited by 83 · see citing works
Risk factors and interventions with statistically significant tiny effects
Siontis, G. C., & Ioannidis, J. P. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr099 · Cited by 80 · see citing works
Self-citationWe need statistical thinking, not statistical rituals
Gigerenzer, G. (1998) · Behavioral and Brain Sciences · doi:10.1017/s0140525x98281167 · Cited by 78 · see citing works
Outcome reporting bias in clinical trials: why monitoring matters
Ioannidis, J. P. A., Caplan, A. L., & Dal‐Ré, R. (2017) · BMJ · doi:10.1136/bmj.j408 · Cited by 76 · see citing works
Self-citationCitation bias of hepato-biliary randomized clinical trials
Gluud, L. L., & Gluud, C. N. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00513-3 · Cited by 72 · see citing works
Neuroadaptive Bayesian Optimization and Hypothesis Testing
Lorenz, R., Hampshire, A., & Leech, R. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.01.006 · Cited by 67 · see citing works
Raw data from clinical trials: within reach?
Doshi, P., Goodman, S. N., & Ioannidis, J. P. A. (2013) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2013.10.006 · Cited by 63 · see citing works
Self-citationHow Confident are Students in their Misconceptions about Hypothesis Tests?
Sotos, A. E. C., Vanhoof, S., Noortgate, W. V. D., & Onghena, P. (2009) · Journal of Statistics Education · doi:10.1080/10691898.2009.11889514 · Cited by 58 · see citing works
The concept of statistical significance and the controversy about one-tailed tests.
Eysenck, H. J. (1960) · Psychological Review · doi:10.1037/h0048412 · Cited by 54 · see citing works
Registration practices for observational studies on ClinicalTrials.gov indicated low adherence
Boccia, S., Rothman, K. J., Panić, N., Flacco, M. E., Rosso, A., Pastorino, R., Manzoli, L., Vecchia, C. L., Villari, P., Boffetta, P., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.09.009 · Cited by 53 · see citing works
Self-citationImproving functional magnetic resonance imaging reproducibility
Pernet, C. R., & Poline, J. (2015) · GigaScience · doi:10.1186/s13742-015-0055-8 · Cited by 47 · see citing works
A Tutorial on Hunting Statistical Significance by Chasing N
Szűcs, D. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01444 · Cited by 45 · see citing works
Teaching hypothesis tests – time for significant change?
Sterne, J. A. C. (2002) · Statistics in Medicine · doi:10.1002/sim.1129 · Cited by 43 · see citing works
Koren, J. (2016) · University of Michigan Library Repository · Cited by 37 · see citing works
Hypothesis Testing in Relation to Statistical Methodology
Clark, C. A. (1963) · Review of Educational Research · doi:10.2307/1169648 · Cited by 35 · see citing works
Significance Tests in Climate Science
Ambaum, M. H. P. (2010) · Journal of Climate · doi:10.1175/2010jcli3746.1 · Cited by 34 · see citing works
I TRIED A BUNCH OF THINGS: THE DANGERS OF UNEXPECTED OVERFITTING IN CLASSIFICATIONPreprint
Powell, M. A., Hosseini, M. N. G., Collins, J. J., Flintoft, C. C., Jones, W., Bowman, H., & Wyble, B. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/078816 · Cited by 32 · see citing works
Reproducible research and the substantive context
Keiding, N. (2010) · Biostatistics · doi:10.1093/biostatistics/kxq033 · Cited by 30 · see citing works
Improving the Drug Development Process
Djulbegović, B., Hozo, I., & Ioannidis, J. P. A. (2014) · JAMA · doi:10.1001/jama.2013.283742 · Cited by 25 · see citing works
Self-citationMika Kivimaki, B., Batty, G. D., Kawachi, I. I., Virtanen, M., Singh‐Manoux, A., & Brunner, E. J. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu164 · Cited by 24 · see citing works
The fallacy of the null hypothesis in soft psychology
Waller, N. G. (2004) · Applied and Preventive Psychology · doi:10.1016/j.appsy.2004.02.015 · Cited by 24 · see citing works
THE SIGNIFICANCE TEST CONTROVERSY
Giere, R. N. (1972) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/23.2.170 · Cited by 23 · see citing works
Teaching critical appraisal and statistics in anesthesia journal club
Moharari, R. S., Rahimi, E., Najafi, A., Khashayar, P., Khajavi, M., & Ap, M. (2008) · QJM · doi:10.1093/qjmed/hcn131 · Cited by 23 · see citing works
Barbeau, E. (2013) · MAA spectrum · doi:10.5948/upo9781614445159.009 · Cited by 22 · see citing works
Gelman, A. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.03.005 · Cited by 21 · see citing works
A Bayesian Approach to the Multiple Comparisons Problem
Neath, A. A., & Cavanaugh, J. E. (2021) · Journal of Data Science · doi:10.6339/jds.2006.04(2).266 · Cited by 21 · see citing works
Edwards, A. W. F. (1972) · CERN Document Server (European Organization for Nuclear Research) · Cited by 20 · see citing works
Misconceptions About Sample Size, Statistical Significance, and Treatment Effect
Wilkerson, M., & Olson, M. R. (1997) · The Journal of Psychology · doi:10.1080/00223989709603844 · Cited by 19 · see citing works
Diggle, P. J., & Zeger, S. L. (2010) · Biostatistics · doi:10.1093/biostatistics/kxq029 · Cited by 18 · see citing works
Chapter I: Hypothesis Testing in Relation to Statistical Methodology
Clark, C. A. (1963) · Review of Educational Research · doi:10.3102/00346543033005455 · Cited by 15 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence: Rejoinder
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289139 · Cited by 15 · see citing works
The Tools-to-Theory Hypothesis
Luce, R. D. (1988) · Contemporary Psychology · doi:10.1037/030460 · Cited by 7 · see citing works
Flagged first, then by citations
1 of the 197 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Chaisson E. McMillan S. (2017). Astronomy Today. Pearson.
- Cohen J. (1988). Statistical Power Analysis for the Behavioural Sciences. Academic Press.
- Fisher R. (1925). Statistical Methods for Research Workers, First Edition. Edinburgh: Oliver and Boyd.
- Jeffreys H. (1939, 1948, 1961). The Theory of Probability. Oxford: Oxford University Press.
- Smolin L. (2006). The Trouble with Physics. London: Penguin.
Cited by
Sorted by citations
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996 · see citing works
Reporting guidelines for human microbiome research: the STORMS checklist
Mirzayi, C. A., Renson, A., Massive Analysis and Quality Control Society, Furlanello, C., Sansone, S., Zohra, F., Elsafoury, S., Geistlinger, L., Kasselman, L. J., Eckenrode, K. B., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01552-x · Cited by 572 · see citing works
Szucs, D., & Ioannidis, J. P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117164 · Cited by 453 · see citing works
Self-citationKline, R. B. (2018) · Canadian Studies in Population · doi:10.25336/csp29418 · Cited by 336 · see citing works
Bayesian Random-Effects Meta-Analysis Using the <b>bayesmeta</b> <i>R</i> Package
Röver, C. (2020) · Journal of Statistical Software · doi:10.18637/jss.v093.i06 · Cited by 310 · see citing works
I can see clearly now: Reinterpreting statistical significance
Dushoff, J. G., Kain, M. P., & Bolker, B. M. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13159 · Cited by 207 · see citing works
Exploration, Inference, and Prediction in Neuroscience and Biomedicine
Bzdok, D., & Ioannidis, J. P. A. (2019) · Trends in Neurosciences · doi:10.1016/j.tins.2019.02.001 · Cited by 204 · see citing works
Self-citationThe interpretation of statistical power after the data have been gathered
Dziak, J. J., Dierker, L., & Abar, B. W. (2018) · Current Psychology · doi:10.1007/s12144-018-0018-1 · Cited by 208 · see citing works
Zhang, M. F., Dawson, J., & Kline, R. B. (2020) · British Journal of Management · doi:10.1111/1467-8551.12415 · Cited by 200 · see citing works
Ioannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175 · see citing works
Self-citationA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165 · see citing works
What Have We (Not) Learnt from Millions of Scientific Papers with P Values?
Ioannidis, J. P. A. (2019) · The American Statistician · doi:10.1080/00031305.2018.1447512 · Cited by 161 · see citing works
Self-citationEstimation for Better Inference in Neuroscience
Calin-Jageman, R. J., & Cumming, G. (2019) · eNeuro · doi:10.1523/eneuro.0205-19.2019 · Cited by 145 · see citing works
Moulder, R. G., Boker, S. M., Ramseyer, F., & Tschacher, W. (2018) · Psychological Methods · doi:10.1037/met0000172 · Cited by 108 · see citing works
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98 · see citing works
Self-citationReproducibility and replicability in zebrafish behavioral neuroscience research
Gerlai, R. T. (2018) · Pharmacology Biochemistry and Behavior · doi:10.1016/j.pbb.2018.02.005 · Cited by 100 · see citing works
Probabilistic risk assessment – the keystone for the future of toxicology
Maertens, A. (2022) · ALTEX · doi:10.14573/altex.2201081 · Cited by 101 · see citing works
Understanding Statistical Hypothesis Testing: The Logic of Statistical Inference
Emmert‐Streib, F., & Dehmer, M. (2019) · Machine Learning and Knowledge Extraction · doi:10.3390/make1030054 · Cited by 96 · see citing works
Hengartner, M. P., & Plöderl, M. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00517 · Cited by 94 · see citing works
Joffe, A. R., Wong, K., Bond, G. Y., Moez, E. K., Acton, B. V., Dinu, I. A., Yap, J. Y. K., Robertson, C. M. T., & Program, F. T. W. C. C. P. T. F. (2019) · Pediatric Transplantation · doi:10.1111/petr.13624 · Cited by 86 · see citing works
Khalid, A. M., Hosny, K. M., & Mirjalili, S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07639-x · Cited by 89 · see citing works
Towards algorithmic analytics for large-scale datasets
Bzdok, D., Nichols, T. E., & Smith, S. M. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0069-5 · Cited by 84 · see citing works
Why and how we should join the shift from significance testing to estimation
Berner, D., & Amrhein, V. (2022) · Journal of Evolutionary Biology · doi:10.1111/jeb.14009 · Cited by 78 · see citing works
Neuroanatomical norms in the UK Biobank: The impact of allometric scaling, sex, and age
Williams, C. M., Peyre, H., Toro, R., & Ramus, F. (2021) · Human Brain Mapping · doi:10.1002/hbm.25572 · Cited by 78 · see citing works
Caviola, S., Colling, L. J., Mammarella, I. C., & Szűcs, D. (2020) · Developmental Science · doi:10.1111/desc.12957 · Cited by 72 · see citing works
The continuing misuse of null hypothesis significance testing in biological anthropology
Smith, R. J. (2018) · American Journal of Physical Anthropology · doi:10.1002/ajpa.23399 · Cited by 74 · see citing works
Lockie, R. G., Dawes, J. J., Orr, R. M., & Dulla, J. M. (2020) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000003534 · Cited by 70 · see citing works
A hitchhiker’s guide to working with large, open-source neuroimaging datasets
Horien, C. L., Noble, S., Greene, A. S., Lee, K., Barron, D. S., Gao, S., O’Connor, D. A., Salehi, M., Dadashkarimi, J., Shen, X., et al. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-01005-4 · Cited by 67 · see citing works
Inference and Prediction Diverge in Biomedicine
Bzdok, D., Engemann, D. A., Thirion, B., Varoquaux, G., & Thirion, B. (2020) · Patterns · doi:10.1016/j.patter.2020.100119 · Cited by 73 · see citing works
Cristea, I. A., & Ioannidis, J. P. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0197440 · Cited by 55 · see citing works
Self-citationThe Sexual Double Standard in the Real World
Marks, M. J., Young, T. M., & Zaikman, Y. (2018) · Social Psychology · doi:10.1027/1864-9335/a000362 · Cited by 53 · see citing works
Interaction, Feedback and Task Research in Second Language Learning
Mackey, A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108589284 · Cited by 62 · see citing works
Berardi, M. L., Brosch, K., Pfarr, J., Schneider, K. L., Sültmann, A., Thomas‐Odenthal, F., Wroblewski, A., Usemann, P. L., Philipsen, A., Dannlowski, U., et al. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02594-0 · Cited by 48 · see citing works
Solmi, M., Correll, C. U., Carvalho, A. F., & Ioannidis, J. P. A. (2018) · Epidemiology and Psychiatric Sciences · doi:10.1017/s204579601800032x · Cited by 44 · see citing works
Self-citationBCOVIDOA: A Novel Binary Coronavirus Disease Optimization Algorithm for Feature Selection
Khalid, A. M., Hamza, H. M., Mirjalili, S., & Hosny, K. M. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108789 · Cited by 46 · see citing works
Analysing the impact of ChatGPT in research
Picazo-Sanchez, P., & Ortiz-Martin, L. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05298-0 · Cited by 46 · see citing works
Decoding personality trait measures from resting EEG: An exploratory report
Jach, H. K., Feuerriegel, D., & Smillie, L. D. (2020) · Cortex · doi:10.1016/j.cortex.2020.05.013 · Cited by 45 · see citing works
Treating Hypertension in Children With n-of-1 Trials
Samuel, J. P., Tyson, J. E., Green, C., Bell, C., Pedroza, C., Molony, D., & Samuels, J. (2019) · PEDIATRICS · doi:10.1542/peds.2018-1818 · Cited by 43 · see citing works
Lissa, C. J. V. (2022) · Infant and Child Development · doi:10.1002/icd.2370 · Cited by 44 · see citing works
A phylogenetic approach to comparative genomics
Dewar, A. E., Belcher, L. J., & West, S. A. (2025) · Nature Reviews Genetics · doi:10.1038/s41576-024-00803-0 · Cited by 41 · see citing works
Statistical Inference and the Replication Crisis
Colling, L. J., & Szűcs, D. (2019) · PhilPapers (PhilPapers Foundation) · doi:10.17863/cam.35452 · Cited by 41 · see citing works
Bramhall, N. F., McMillan, G. P., & Kampel, S. D. (2020) · Hearing Research · doi:10.1016/j.heares.2021.108310 · Cited by 40 · see citing works
Equivalent statistics for a one-sample t-test
Francis, G., & Jakicic, V. (2022) · Behavior Research Methods · doi:10.3758/s13428-021-01775-3 · Cited by 39 · see citing works
Conditional equivalence testing: An alternative remedy for publication bias
Campbell, H., & Gustafson, P. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195145 · Cited by 40 · see citing works
Frequentist and Bayesian inference: A conceptual primer
van Zyl, C. J. J. (2018) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2018.06.004 · Cited by 38 · see citing works
Moving beyond generalization to accurate interpretation of flexible models
Genkin, M., & Engel, T. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00242-6 · Cited by 38 · see citing works
Fischer, U., Trelle, S., Branca, M., Salanti, G., Paciaroni, M., Ferrari, C., Abend, S., Beyeler, S., Strbian, D., Thomalla, G., et al. (2022) · European Stroke Journal · doi:10.1177/23969873221106043 · Cited by 32 · see citing works
Williams, S., Carson, R., & Tóth, K. (2023) · Journal of Physiology · doi:10.1113/jp285575 · Cited by 23 · see citing works
Statistical Inference and the Replication Crisis
Colling, L. J., & Szűcs, D. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0421-4 · Cited by 47 · see citing works
The superior longitudinal fasciculus and its functional triple-network mechanisms in brooding
Pisner, D., Shumake, J., Beevers, C. G., & Schnyer, D. M. (2019) · NeuroImage Clinical · doi:10.1016/j.nicl.2019.101935 · Cited by 36 · see citing works
Job demands, organizational justice, and emotional exhaustion in prison officers
Clements, A. J., & Kinman, G. (2021) · Criminal Justice Studies · doi:10.1080/1478601x.2021.1999114 · Cited by 40 · see citing works
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 37 · see citing works
Current controversies: Null hypothesis significance testing
Sedgwick, P. M., Hammer, A., Kesmodel, U. S., & Pedersen, L. H. (2022) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.14366 · Cited by 35 · see citing works
White matter network alterations in patients with depersonalization/derealization disorder
Sierk, A., Daniels, J. K., Manthey, A., Kok, J. G., Leemans, A., Gaebler, M., Lamke, J.-P., Kruschwitz, J., & Walter, H. (2018) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.170110 · Cited by 34 · see citing works
Panteleimon Ekkekakis, A., Swinton, P. A., & Tiller, N. B. (2023) · Sports Medicine · doi:10.1007/s40279-023-01880-7 · Cited by 35 · see citing works
The Impact of Complexity on Methods and Findings in Psychological Science
Sanbonmatsu, D. M., Cooley, E. H., & Butner, J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.580111 · Cited by 37 · see citing works
New Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research
Mertens, W., & Recker, J. (2023) · Technology, work and globalization · doi:10.1007/978-3-031-38719-7_13 · Cited by 33 · see citing works
Methylphenidate and short-term cardiovascular risk
Garcia-Argibay, M., Bürkner, P.-C., Lichtenstein, P., Zhang, L., D’Onofrio, B. M., Andell, P., Chang, Z., Cortese, S., & Larsson, H. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.1349 · Cited by 31 · see citing works
P in the right place: Revisiting the evidential value of P‐values
Lytsy, P. (2018) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12319 · Cited by 30 · see citing works
Optimizing Auditory Brainstem Response Acquisition Using Interleaved Frequencies
Buran, B. N., Elkins, S. P., Kempton, J. B., Porsov, E. V., Brigande, J. V., & David, S. V. (2020) · Journal of the Association for Research in Otolaryngology · doi:10.1007/s10162-020-00754-3 · Cited by 31 · see citing works
Cote, M. P., Lubowitz, J. H., Brand, J. C., & Rossi, M. J. (2021) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2021.02.010 · Cited by 31 · see citing works
Roordink, E. M., Steenhuis, I. H., Kroeze, W., Schoonmade, L. J., Sniehotta, F. F., & van Stralen, M. M. (2021) · Psychology and Health · doi:10.1080/08870446.2021.1981900 · Cited by 31 · see citing works
A Data Science Solution to the Multiple-Testing Crisis in Financial Research
de Prado, M. L. (2019) · The Journal of Financial Data Science · doi:10.3905/jfds.2019.1.099 · Cited by 27 · see citing works
Schreiber, J. B. (2020) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2019.05.023 · Cited by 21 · see citing works
Pernet, C. R. (2017) · F1000Research · doi:10.12688/f1000research.6963.5 · Cited by 26 · see citing works
Stein, D. G., & Sayeed, I. (2018) · Neuropharmacology · doi:10.1016/j.neuropharm.2018.04.006 · Cited by 26 · see citing works
Effects of Category Aggregation on Land Change Simulation Based on Corine Land Cover Data
Varga, O. G., Pontius, R. G., Szabó, Z. C., & Szabó, S. (2020) · Remote Sensing · doi:10.3390/rs12081314 · Cited by 26 · see citing works
Valizade, D., Schulz, F., & Nicoara, C. (2022) · British Journal of Management · doi:10.1111/1467-8551.12678 · Cited by 25 · see citing works
Vincent-Lamarre, P., & Larivière, V. (2019) · Quantitative Science Studies · doi:10.1162/qss_a_00125 · Cited by 25 · see citing works
Published correlational effect sizes in social and developmental psychology
Weinerová, J., Szűcs, D., & Ioannidis, J. P. A. (2022) · Royal Society Open Science · doi:10.1098/rsos.220311 · Cited by 23 · see citing works
Self-citationStatistical errors in software engineering experiments
Ch., R. P. R., Dieste, Ó., Fonseca, E. R., & Juristo, N. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180161 · Cited by 22 · see citing works
Wang, F., Yin, J., Hu, L., Wang, M., Liu, X., Miller, K., & Wittek, A. (2022) · Accident Analysis & Prevention · doi:10.1016/j.aap.2022.106718 · Cited by 22 · see citing works
Rubin, M. (2019) · Synthese · doi:10.1007/s11229-019-02433-0 · Cited by 23 · see citing works
Beyond regression: Unpacking research of human complex systems with qualitative comparative analysis
Xu, X.-X., Tahir, S., Khan, K. B., Sajid, M. A., & Safdar, M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e31457 · Cited by 19 · see citing works
Differences in Fitness between Firefighter Trainee Academy Classes and Normative Percentile Rankings
Lockie, R. G., Orr, R. M., Montes, F., Ruvalcaba, T. J., & Dawes, J. J. (2022) · Sustainability · doi:10.3390/su14116548 · Cited by 21 · see citing works
Sierk, A., Manthey, A., Brakemeier, E., Walter, H., & Daniels, J. K. (2020) · Brain Imaging and Behavior · doi:10.1007/s11682-020-00274-x · Cited by 21 · see citing works
Holman, M. E., Chang, G. A., Ghatas, M. P., Saha, P. K., Zhang, X., Khan, M. R., Sima, A. P., Adler, R. A., & Gorgey, A. S. (2021) · Osteoporosis International · doi:10.1007/s00198-020-05778-2 · Cited by 19 · see citing works
A Formative Mixed-Methods Study of Emotional Responsiveness in Telepalliative Care
Hutchinson, R. N., Anderson, E., Ruben, M. A., Manning, N., John, L., Daruvala, A., Rizzo, D. M., Eppstein, M. J., Gramling, R. E., & Han, P. K. J. (2022) · Journal of Palliative Medicine · doi:10.1089/jpm.2021.0589 · Cited by 19 · see citing works
Bleakley, C., Matthews, M., & Smoliga, J. M. (2020) · Journal of sport and health science/Journal of Sport and Health Science · doi:10.1016/j.jshs.2020.11.002 · Cited by 18 · see citing works
Lytsy, P., Hartman, M., & Pingel, R. (2022) · Upsala Journal of Medical Sciences · doi:10.48101/ujms.v127.8760 · Cited by 18 · see citing works
Khalid, A. M., Hamza, H. M., Mirjalili, S., & Hosny, K. M. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08587-w · Cited by 18 · see citing works
Accounting research and the significance test crisis
Johnstone, D. J. (2021) · Critical Perspectives on Accounting · doi:10.1016/j.cpa.2021.102296 · Cited by 18 · see citing works
Problem with p values: why p values do not tell you if your treatment is likely to work
Price, R., Bethune, R. M., & Massey, L. H. (2019) · Postgraduate Medical Journal · doi:10.1136/postgradmedj-2019-137079 · Cited by 18 · see citing works
Why we habitually engage in null-hypothesis significance testing: A qualitative study
Stunt, J. J., van Grootel, L., Bouter, L. M., Trafimow, D., Hoekstra, T., & de Boer, M. R. (2021) · PLoS ONE · doi:10.1371/journal.pone.0258330 · Cited by 17 · see citing works
Farewell to Bright-Line: A Guide to Reporting Quantitative Results Without the S-Word
Cummins, K. M., & Marks, C. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00815 · Cited by 17 · see citing works
Individual alpha frequency modulates sleep-related emotional memory consolidation
Cross, Z. R., Santamaria, A., Corcoran, A. W., Chatburn, A., Alday, P. M., Coussens, S. W., & Kohler, M. J. (2020) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2020.107660 · Cited by 17 · see citing works
Dul, J., van der Laan, E., Kuik, R., & Karwowski, M. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01493 · Cited by 17 · see citing works
The Evidence Base for Prevention of Infectious Bovine Keratoconjunctivitis Through Vaccination
Maier, G. U., O'Connor, A. M. C., & Sheedy, D. B. (2021) · Veterinary Clinics of North America Food Animal Practice · doi:10.1016/j.cvfa.2021.03.009 · Cited by 16 · see citing works
Reito, A. R., Raittio, L., & Helminen, O. (2019) · PeerJ · doi:10.7717/peerj.6813 · Cited by 16 · see citing works
Establishment of big data evaluation model for green and sustainable development of enterprises
Meiyou, D., & Yao, Y. (2022) · Journal of King Saud University - Science · doi:10.1016/j.jksus.2022.102041 · Cited by 16 · see citing works
How failure to falsify in high-volume science contributes to the replication crisis
Rajtmajer, S. M., Errington, T. M., & Hillary, F. G. (2022) · eLife · doi:10.7554/elife.78830 · Cited by 16 · see citing works
Salminen, L. E., Wilcox, R. R., Zhu, A. H., Riedel, B. C., Ching, C. R. K., Rashid, F. M., Thomopoulos, S. I., Saremi, A., Harrison, M. B., Ragothaman, A., et al. (2019) · Cerebral Cortex · doi:10.1093/cercor/bhz060 · Cited by 16 · see citing works
Zweifach, A. (2025) · SLAS DISCOVERY · doi:10.1016/j.slasd.2025.100253 · Cited by 15 · see citing works
Neural and cognitive correlates of performance in dynamic multi-modal settings
Dziego, C. A., Bornkessel‐Schlesewsky, I., Jano, S., Chatburn, A., Schlesewsky, M., Immink, M. A., Sinha, R., Irons, J. L., Schmitt, M., Chen, S., et al. (2023) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2023.108483 · Cited by 19 · see citing works
Nuzzo, J., Pinto, M. D., & Nosaka, K. (2023) · Scandinavian Journal of Medicine & Science in Sports · doi:10.1111/sms.14330 · Cited by 15 · see citing works
Interaction, Feedback and Task Research in Second Language Learning : Methods and Design
Mackey, A. (2020) · Cited by 15 · see citing works
Needed: More Reliable Bioeffects Studies at “High Band” 5G Frequencies
Foster, K. R., & Vijayalaxmi, V. (2021) · Frontiers in Communications and Networks · doi:10.3389/frcmn.2021.721925 · Cited by 15 · see citing works
Most recommended medical interventions reach P < 0.005 for their primary outcomes in meta-analyses
Koletsi, D., Solmi, M., Pandis, N., Fleming, P. S., Correll, C. U., & Ioannidis, J. P. A. (2019) · International Journal of Epidemiology · doi:10.1093/ije/dyz241 · Cited by 14 · see citing works
Self-citationShao, Y., Shen, Y., Zhao, L. Q., Guo, X., Niu, C., & Liu, F. (2020) · Journal of Cancer · doi:10.7150/jca.37150 · Cited by 14 · see citing works
Experimental Disease-Modifying Agents for Frontotemporal Lobar Degeneration
Giunta, M., Solje, E., Gardoni, F., Borroni, B., & Benussi, A. (2021) · Journal of Experimental Pharmacology · doi:10.2147/jep.s262352 · Cited by 14 · see citing works
- Authors
- Szucs, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Venue
- Frontiers in Human Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2017-08-03
- Volume
- 11
- Pages
- 390-390
- ISSN
- 1662-5161 (ISSN-L)
- PMC
- PMC5540883
- OpenAlex
- W2951307730
- Cited by
- 298
- References
- 197
- Citations by year
- 2026: 18 · 2025: 28 · 2024: 21 · 2023: 34 · 2022: 44 · 2021: 31 · 2020: 48 · 2019: 39 · 2018: 29 · 2017: 6
- FWCI
- 6.14
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Frontiers in Human Neuroscience (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo (cc-by)
- Funders
- James S. McDonnell Foundation (220020370)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- null hypothesis significance testing, replication crisis, statistical inference, preregistration, power analysis, effect size, negative results, statistics education, research methodology critique, hypothesis testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
