Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature
Open accessPublished version of a preprint
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64-1.46) for nominally statistically significant results and D = 0.24 (0.11-0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite methodological challenges10–11, many of the findings of cognitive neuroscience provide a solid basis on which to build.”
From Cognitive computational neuroscience (2018) · cites this paper as [10]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In several disciplines, the average statistical power of studies was found to be significantly below the recommended 80% level (13 -16).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“such studies showed that power varies widely between subfields or methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For most published research findings to be false in psychology and neuroscience, for example, one must assume that the hypotheses tested in these disciplines are correct much less than 50% of the time (14, 18).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“and has contributed, in part, to the replication crisis in other fields (Button et al., 2013; Szucs and Ioannidis, 2017; Ioannidis et al., 2017).”
From With Little Power Comes Great Responsibility (2020) · cites this paper as Szucs and Ioannidis, 2017
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Effect size estimates: Current use, calculations, and interpretation.
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024338 · Cited by 5,199
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 9×
“…examined p value reporting errors in the cognitive neuroscience literature as recent studies found that 12.5% to 20% of psychology papers have major p value reporting errors which affect judgments about statistical significance (Bakker and Wicherts, 2011; Nuijten et al. 2015; Veldkamp et al. 2014).”
BackgroundMethodResultInfluentialSelf-citationCluster failure: why fMRI inferences for spatial extent have inflated false-positive rates
Eklund, A., Nichols, T. E., & Knutsson, H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1602413113 · Cited by 3,487
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
Cited by
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationReproducible brain-wide association studies require thousands of individuals
Marek, S., Tervo-Clemmens, B., Calabro, F. J., Montez, D. F., Kay, B. P., Hatoum, A. S., Donohue, M. R., Foran, W., Miller, R. L., Hendrickson, T. J., et al. (2022) · Nature · doi:10.1038/s41586-022-04492-9 · Cited by 1,928
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
“…on being true for many studies run today (Cohen, 1962; Dumas-Mallet, Button, Boraud, Gonon, & Munafò, 2017; Fraley & Vazire, 2014; Maxwell, 2004; Szucs & Ioannidis, 2017a; Vankov, Bowers, & Munafo, 2014), despite the fact that many articles on power calculation and required sample sizes have…”
BackgroundIndices of Effect Existence and Significance in the Bayesian Framework
Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02767 · Cited by 983
Cross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897
“Analysis of the literature suggests that publications are indeed too often on the edge of significance (Szucs and Ioannidis, 2016) and are vastly biased by selection according to the prevailing opinions (Ioannidis, 2008).”
BackgroundEffect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology
Brydges, C. R. (2019) · Innovation in Aging · doi:10.1093/geroni/igz036 · Cited by 826
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Ten simple rules for conducting umbrella reviews
Fusar-Poli, P., & Radua, J. (2018) · Evidence Based Mental Health · doi:10.1136/ebmental-2018-300014 · Cited by 724
Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence
Keysers, C., Gazzola, V., & Wagenmakers, E.-J. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0660-4 · Cited by 654
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
Small Effects: The Indispensable Foundation for a Cumulative Psychological Science
Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620984483 · Cited by 538
“…effect sizes (Button et al., 2013; Schäfer & Schwarz, 2019; Szucs & Ioannidis, 2017), effect sizes obtained from well-powered preregistered studies (Funder & Ozer, 2019; Miller, 2019; Schäfer & Schwarz, 2019; Schooler, 2011; Szucs & Ioannidis, 2017) accurately capture highly reliable effects.”
BackgroundEstimating power in (generalized) linear mixed models: An open introduction and tutorial in R
Kumle, L., Võ, M. L.-H., & Draschkow, D. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01546-0 · Cited by 552
“Given the struggle for reproducibility in various subdomains of psychology (Ioannidis, 2005; Szucs & Ioannidis, 2017; Yong, 2012), adopting effect sizes from published data involves the risk of performing the analyses on inflated effect sizes, which in turn would result in an underpowered design.”
MethodCognitive computational neuroscience
Kriegeskorte, N., & Douglas, P. K. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0210-5 · Cited by 535
When and why defaults influence decisions: a meta-analysis of default effects
JACHIMOWICZ, J. M., DUNCAN, S., WEBER, E. U., & JOHNSON, E. J. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2018.43 · Cited by 481
Szucs, D., & Ioannidis, J. P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117164 · Cited by 453
Self-citationThe psycho Package: an Efficient and Publishing-Oriented Workflow for Psychological Science
Makowski, D. (2018) · The Journal of Open Source Software · doi:10.21105/joss.00470 · Cited by 379
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 370
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Small sample sizes reduce the replicability of task-based fMRI studies
Turner, B. O., Paul, E. J., Miller, M. B., & Barbey, A. K. (2018) · Communications Biology · doi:10.1038/s42003-018-0073-z · Cited by 387
Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations
McKiernan, E. C., Schimanski, L. A., Muñoz Nieves, C., Matthias, L., Niles, M. T., & Alperin, J. P. (2019) · eLife · doi:10.7554/elife.47338 · Cited by 347
Eliot, L., Ahmed, A., Khan, H., & Patel, J. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.02.026 · Cited by 360
The relation between statistical power and inference in fMRI
Cremers, H. R., Wager, T. D., & Yarkoni, T. (2017) · PLOS ONE · doi:10.1371/journal.pone.0184923 · Cited by 349
Survey-software implicit association tests: A methodological and empirical analysis
Carpenter, T. P., Pogacar, R., Pullig, C., Kouril, M., Aguilar, S., LaBouff, J., Isenberg, N., & Chakroff, A. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01293-3 · Cited by 310
“…noted earlier, IAT studies are typically conducted using small, in-person samples; however, small samples are less informative, yielding imprecise parameter estimates (e.g., slopes, correlations) and have been criticized as a factor leading to issues with replica-bility (Szucs & Ioannidis, 2017).”
BackgroundPower to Detect What? Considerations for Planning and Evaluating Sample Size
Giner-Sorolla, R., Montoya, A. K., Reifman, A., Carpenter, T., Lewis, N. A., Aberson, C. L., Bostyn, D. H., Conrique, B. G., Ng, B. W., Schoemann, A. M., et al. (2024) · Personality and Social Psychology Review · doi:10.1177/10888683241228328 · Cited by 221
“However, 3.5 false positives are also expected, so nearly half of all significant effects are not true in the population (Szucs & Ioannidis, 2017).”
BackgroundInfluentialElliott, M. L., Knodt, A. R., Cooke, M., Kim, M. J., Melzer, T. R., Keenan, R., Ireland, D., Ramrakha, S., Poulton, R., Caspi, A., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.01.068 · Cited by 304
“…to replicate these findings, they nevertheless suggest that GFC may provide a generalizable, practical framework that researchers with resting-state and task data can use to drive a cumulative, predictive neuroscience of individual differences (Button et al., 2013; Szucs and Ioannidis, 2017).”
InfluentialWhen Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
Self-citationAre Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence
Appel, M., Marker, C., & Gnambs, T. (2020) · Review of General Psychology · doi:10.1177/1089268019880891 · Cited by 290
“Moreover, meta-analyses can alleviate the problem of insufficient power in many individual studies (Szucs & Ioannidis, 2017) to identify small effects that are typically encountered in psychological research (Bosco, Aguinis, Singh, Field, & Pierce, 2015; Gignac & Szodorai, 2016).”
BackgroundChase, H. W., Boudewyn, M. A., Carter, C. S., & Phillips, M. L. (2020) · Molecular Psychiatry · doi:10.1038/s41380-019-0499-9 · Cited by 286
“Specifically, the relatively modest sample size of most studies in the tDCS literature may be problematic, given that estimates of behavioral effect sizes are often moderate or small [44].”
BackgroundMathieu, M., Eschleman, K. J., & Cheng, D. (2019) · Journal of Occupational Health Psychology · doi:10.1037/ocp0000135 · Cited by 235
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263
“…neuroscience and psychology articles published between 2011 and 2014 found that the median power to detect small, medium, and large effects was 12%, 44%, and 73%, respectively; in other words, there had been no improvement since the first power studies were conducted (Szucs & Ioannidis, 2017).”
BackgroundRedefine statistical significance
Benjamin, D. J., Berger, J., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K., Brembs, B., Brown, L., Camerer, C., et al. (2017) · doi:10.31234/osf.io/mky9j · Cited by 211
Self-citationMachine Learning in Psychometrics and Psychological Research
Orrù, G., Monaro, M., Conversano, C., Gemignani, A., & Sartori, G. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02970 · Cited by 195
“In summarizing their assessment of replicability Szucs and Ioannidis (2017) concluded that: “ Assuminga realisticrangeof priorprobabilities fornullhypotheses, false report probability is likely to exceed 50% for the whole literature.”
BackgroundThe interpretation of statistical power after the data have been gathered
Dziak, J. J., Dierker, L. C., & Abar, B. (2020) · Current Psychology · doi:10.1007/s12144-018-0018-1 · Cited by 202
The power of predictions: An emerging paradigm for psychological research
Hutchinson, J. B., & Barrett, L. F. (2019) · Current Directions in Psychological Science · doi:10.1177/0963721419831992 · Cited by 216
“Thus, observed effects will be less likely to replicate as a consequence (Szucs & Ioannidis, 2017).”
Prestigious Science Journals Struggle to Reach Even Average Reliability
Brembs, B. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00037 · Cited by 213
“In their case, unlike in the neuroscience case, statistical power is even negatively correlated with journal rank (Szucs and Ioannidis, 2017).”
BackgroundNeuroanatomy of developmental dyslexia: Pitfalls and promise
Ramus, F., Altarelli, I., Jednoróg, K., Zhao, J., & Scotto di Covella, L. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.08.001 · Cited by 187
Fox, A. S., & Shackman, A. J. (2019) · Neuroscience Letters · doi:10.1016/j.neulet.2017.11.056 · Cited by 194
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019) · Psychophysiology · doi:10.1111/psyp.13437 · Cited by 184
The Science of Prognosis in Psychiatry: A Review
Fusar-Poli, P., Hijazi, Z., Stahl, D., & Steyerberg, E. W. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2530 · Cited by 197
Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions
Higginson, A. D., & Munafò, M. R. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000995 · Cited by 192
Alday, P. M. (2019) · Psychophysiology · doi:10.1111/psyp.13451 · Cited by 197
“The computational complexity is hard to address, but the requirement for more data is again in line with contemporary best practice to address the chronic lack of power in neuroscience (cf. Button et al., 2013; Szucs & Ioannidis, 2017).”
BackgroundAvoid Cohen’s ‘Small’, ‘Medium’, and ‘Large’ for Power Analysis
Correll, J., Mellinger, C., McClelland, G. H., & Judd, C. M. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.12.009 · Cited by 164
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187
“…have rendered the discipline highly exposed to bias, potentially resulting in a large volume of exaggerated and misleading results (Ioannidis, Munafò, et al., 2014; John et al., 2012; Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012; Simmons et al., 2011; Szucs & Ioannidis, 2017).”
ResultSelf-citationReplicability and Prediction: Lessons and Challenges from GWAS
Marigorta, U. M., Rodríguez, J. A., Gibson, G., & Navarro, A. (2018) · Trends in Genetics · doi:10.1016/j.tig.2018.03.005 · Cited by 185
The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known
Calin-Jageman, R. J., & Cumming, G. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518266 · Cited by 166
“These problems are not unique to neurohormone research, rather they are endemic across many of the behavioral and life sciences (Button et al. 2013; Dumas-Mallet et al. 2017; Szucs and Ioannidis 2017).”
BackgroundCognitive task information is transferred between brain regions via resting-state network topology
Ito, T., Kulkarni, K. R., Schultz, D. H., Mill, R. D., Chen, R. H., Solomyak, L. I., & Cole, M. W. (2017) · Nature Communications · doi:10.1038/s41467-017-01000-w · Cited by 174
“Given the recent interest in reproducibility in neuroscience and other fields 33,34 , we replicated the hub-like characteristic of cognitive control networks 23,28,29,35 before moving forward with analyses that build on these previous findings.”
MethodIoannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175
Self-citationZickfeld, J., Schubert, T. W., Seibt, B., Lyshol, J. K. B., Arriaga, P., Basabe, N., Blaut, A., Caballero, A., Carrera, P., Dalḡar, İ., et al. (2018) · Emotion · doi:10.1037/emo0000450 · Cited by 183
What Have We (Not) Learnt from Millions of Scientific Papers with P Values?
Ioannidis, J. P. A. (2019) · The American Statistician · doi:10.1080/00031305.2018.1447512 · Cited by 159
Self-citationKohl, S. H., Mehler, D. M. A., Lührs, M., Thibault, R. T., Konrad, K., & Sorger, B. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00594 · Cited by 161
On the future of international joint venture research
Nippa, M., & Reuer, J. J. (2019) · Journal of International Business Studies · doi:10.1057/s41267-019-00212-0 · Cited by 190
“Of the reasons that explain ‘why most published research findings are false’ (Ioannidis, 2005; Ioannidis et al., 2017; Szucs & Ioannidis, 2017), many are also at work in the international business field (Ellis, 2010).”
BackgroundAssessing inter individual differences with task related functional neuroimaging
Lebreton, M., Bavard, S., Daunizeau, J., & Palminteri, S. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0681-8 · Cited by 160
Lafit, G., Adolf, J., Dejonckheere, E., Myin‐Germeys, I., Viechtbauer, W., & Ceulemans, E. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920978738 · Cited by 165
Are specific learning disorders truly specific, and are they disorders?
Peters, L., & Ansari, D. (2019) · Trends in Neuroscience and Education · doi:10.1016/j.tine.2019.100115 · Cited by 151
Estimation for Better Inference in Neuroscience
Calin-Jageman, R. J., & Cumming, G. (2019) · eNeuro · doi:10.1523/eneuro.0205-19.2019 · Cited by 166
“Neuroscience studies are often too small, yielding noisy results that are relatively uninformative ( aka, lowpower research; Button et al., 2013; Dumas-Mallet et al., 2017; Szucs and Ioannidis, 2017b ).”
Background#EEGManyLabs: Investigating the replicability of influential EEG experiments
Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C., Beste, C., Bland, A. R., Bradford, D. E., Bublatzky, F., Busch, N. A., et al. (2021) · Cortex · doi:10.1016/j.cortex.2021.03.013 · Cited by 163
Jas, M., Larson, E., Engemann, D. A., Leppäkangas, J., Taulu, S., Hämäläinen, M., & Gramfort, A. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00530 · Cited by 152
“When to Stop? Obviously, in the light of the current replication crisis, clear rules need to be established on when to stop improving the data analysis (Simmons et al., 2011; Szucs and Ioannidis, 2017).”
BackgroundThe Quest for the FFA and Where It Led
Kanwisher, N. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.1706-16.2016 · Cited by 150
Dougherty, M. R. P., & Horne, Z. (2022) · Royal Society Open Science · doi:10.1098/rsos.220334 · Cited by 156
“Szucs D, Ioannidis JPA.”
BackgroundResultInfluentialOn the phantom-like appearance of bilingualism effects on neurocognition: (How) should we proceed?
Leivada, E., Westergaard, M. R., Duñabeitia, J. A., & Rothman, J. (2020) · Bilingualism Language and Cognition · doi:10.1017/s1366728920000358 · Cited by 147
The role of the microbiome in the neurobiology of social behaviour
Sarkar, A., Harty, S., Johnson, K. V. ‐., Moeller, A. H., Carmody, R. N., Lehto, S. M., Erdman, S. E., Dunbar, R. I. M., & Burnet, P. W. J. (2020) · Biological Reviews · doi:10.1111/brv.12603 · Cited by 135
“…sample sizes in these reports, alongside the known prevalence of very low statistical power in cognitive neuroscience and brain-imaging research (Button et al., 2013; Szucs & Ioannidis, 2017), it may be that many of the most intriguing microbiome–brain associations in humans are false positives.”
BackgroundSchmidt‐Pogoda, A., Bonberg, N., Koecke, M. H. M., Strecker, J., Wellmann, J., Bruckmann, N. M., Beuker, C., Schäbitz, W., Meuth, S. G., Wiendl, H., et al. (2019) · Annals of Neurology · doi:10.1002/ana.25643 · Cited by 131
“3 means that 30% of findings that would be statistically nonsignificant in the absence of bias will be reported as significant.(18) To estimate the true report probability of experimental stroke studies, we performed power calculations and estimated a mean power of 17% (Fig 6).”
BackgroundMethodResultUnderstanding the Replication Crisis as a Base Rate Fallacy
Bird, A. (2018) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axy051 · Cited by 144
“2 for small effect sizes, in cognitive neuroscience and psychology (Szucs and Ioannidis 2017) and some subfields of biomedicine (but not others) (Dumas-Mallet et al.”
Minority reports: Consciousness and the prefrontal cortex
Michel, M., & Morales, J. (2019) · Mind & Language · doi:10.1111/mila.12264 · Cited by 141
“Unfortunately, this type of errors does not come as a great surprise in a field as underpowered as neuroscience (Cremers, Wager, & Yarkoni, 2017; Szucs & Ioannidis, 2017).”
BackgroundNewbury, C. R., Crowley, R., Rastle, K., & Tamminen, J. (2021) · Psychological Bulletin · doi:10.1037/bul0000348 · Cited by 136
“Szucs and Ioannidis (2017) conducted an analysis of almost 4,000 cognitive neuroscience and psychology papers and found that the overall mean power to detect small, medium, and large effects was 17%, 49%, and 71%, respectively, with even lower power in the subfield of cognitive neuroscience.”
BackgroundResultDonolato, E., Cardillo, R., Mammarella, I. C., & Melby‐Lervåg, M. (2021) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13536 · Cited by 135
Geiger, S. J., Brick, C., Nalborczyk, L., Bosshard, A., & Jostmann, N. B. (2021) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2021.101694 · Cited by 131
Rodd, J. M. (2023) · Journal of Memory and Language · doi:10.1016/j.jml.2023.104472 · Cited by 130
“…consensus that published studies in the psychological sciences have been systematically underpowered, and that studies with low statistical power are more likely to report effects that cannot be replicated by other researchers (Brysbaert, 2019; Szucs & Ioannidis, 2017; Fraley & Vazire, 2014).”
BackgroundDevelopmental predictors of offending and persistence in crime: A systematic review of meta-analyses
Basto-Pereira, M., & Farrington, D. P. (2022) · Aggression and Violent Behavior · doi:10.1016/j.avb.2022.101761 · Cited by 112
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 116
Dynamic causal brain circuits during working memory and their functional controllability
Cai, W., Ryali, S., Pasumarthy, R., Talasila, V., & Menon, V. (2021) · Nature Communications · doi:10.1038/s41467-021-23509-x · Cited by 105
Yang, Y., Sánchez‐Tójar, A., O’Dea, R. E., Noble, D. W. A., Koricheva, J., Jennions, M. D., Parker, T. H., Lagisz, M., & Nakagawa, S. (2023) · BMC Biology · doi:10.1186/s12915-022-01485-y · Cited by 119
Evolutionarily conserved role of oxytocin in social fear contagion in zebrafish
Akinrinade, I. D., Kareklas, K., Teles, M. C., Reis, T. K., Gliksberg, M., Petri, G., Levkowitz, G., & Oliveira, R. F. (2023) · Science · doi:10.1126/science.abq5158 · Cited by 112
Interoception primes emotional processing: multimodal evidence from neurodegeneration
Salamone, P. C., Legaz, A., Sedeño, L., Moguilner, S., Fraile-Vazquez, M., Campo, C. G., Fittipaldi, S., Yoris, A., Miranda, M., Birba, A., et al. (2021) · The Journal of Neuroscience · doi:10.1523/jneurosci.2578-20.2021 · Cited by 103
“…comparisons between HCs and each patient group), we considered the following parameters: a large effect size of f=0.40, an a level of p= 0.05, and a power of 0.80, higher than the median of powers analyzed in previous studies in the area for detecting large effect sizes (Szucs and Ioannidis, 2017).”
ResultInfluentialThe psychology of ultra-marathon runners: A systematic review
Roebuck, G. S., Fitzgerald, P. B., Urquhart, D. M., Ng, S. K., Cicuttini, F., & Fitzgibbon, B. M. (2018) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2018.04.004 · Cited by 118
The influence of the respiratory cycle on reaction times in sensory-cognitive paradigms
Johannknecht, M., & Kayser, C. (2022) · Scientific Reports · doi:10.1038/s41598-022-06364-8 · Cited by 101
Sep, M. S. C., Steenmeijer, A., & Kennis, M. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.09.029 · Cited by 100
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Self-citationHow to Beat Science and Influence People: Policy Makers and Propaganda in Epistemic Networks
Weatherall, J. O., O’Connor, C., & Bruner, J. (2018) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axy062 · Cited by 111
fMRI replicability depends upon sufficient individual-level data
Nee, D. E. (2019) · Communications Biology · doi:10.1038/s42003-019-0378-6 · Cited by 100
Rhythms in cognition: The evidence revisited
Keitel, C., Ruzzoli, M., Dugué, L., Busch, N. A., & Benwell, C. (2022) · European Journal of Neuroscience · doi:10.1111/ejn.15740 · Cited by 97
“Finally, publication bias in favour of statistically significant results may have distorted the evidence and undermined the reliability of the published literature (Szucs & Ioannidis, 2017).”
ResultFusar-Poli, P., Werbeloff, N., Rutigliano, G., Oliver, D., Davies, C., Stahl, D., McGuire, P., & Osborn, D. (2019) · Schizophrenia Bulletin · doi:10.1093/schbul/sby070 · Cited by 95
Progress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95
A Guide for Calculating Study-Level Statistical Power for Meta-Analyses
Quintana, D. S. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221147260 · Cited by 95
“This range is selected as the default because the majority of reported effect sizes in psychological sciences (Szucs & Ioannidis, 2017) are between 0 and 1, although this range can be adjusted (see below).”
BackgroundSakaluk, J. K., Williams, A. J., Kilshaw, R. E., & Rhyner, K. T. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000421 · Cited by 120
Kapur, M., Hattie, J., Grossman, I., & Sinha, T. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.956416 · Cited by 100
“However, there needs to be some care in interpreting, as journals that are more cited do not necessarily and always publish the highest quality studies (Munafo and Flint, 2010; Fraley and Vazire, 2014; Szucs and Ioannidis, 2017).”
Power, Dominance, and Constraint: A Note on the Appeal of Different Design Traditions
Rouder, J. N., & Haaf, J. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917745058 · Cited by 99
“One long-standing, salient concern is that too many experiments are underpowered (Cohen, 1962; Maxwell, 2004; Szucs & Ioannidis, 2017).”
Background10 years of Bayesian theories of autism: A comprehensive review
Angeletos Chrysaitis, N., & Seriès, P. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.105022 · Cited by 92
Mathur, M. B., & VanderWeele, T. J. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1464 · Cited by 96
Szűcs, D., & Myers, T. (2016) · Trends in Neuroscience and Education · doi:10.1016/j.tine.2016.11.002 · Cited by 94
Multiple testing correction over contrasts for brain imaging
Alberton, B. A. V., Nichols, T. E., Gamba, H. R., & Winkler, A. M. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116760 · Cited by 91
“Moreover, presumably much of the published literature on the topic suffers from low statistical power, regardless (for further discussion, see Mumford, 2012; Button et al., 2013; Cremers et al., 2017; Szucs and Ioannidis, 2017).”
Carbine, K. A., Rodeback, R. E., Modersitzki, E., Miner, M., LeCheminant, J. D., & Larson, M. J. (2018) · Appetite · doi:10.1016/j.appet.2018.05.135 · Cited by 96
An Examination of Effect Sizes and Statistical Power in Speech, Language, and Hearing Research
Gaeta, L., & Brydges, C. R. (2020) · Journal of Speech Language and Hearing Research · doi:10.1044/2020_jslhr-19-00299 · Cited by 96
Effect sizes of associations between neuroimaging measures and affective symptoms: A meta‐analysis
Feng, C., Thompson, W. K., & Paulus, M. P. (2021) · Depression and Anxiety · doi:10.1002/da.23215 · Cited by 95
Opportunities for increased reproducibility and replicability of developmental neuroimaging
Klapwijk, E. T., van den Bos, W., Tamnes, C. K., Raschle, N. M., & Mills, K. L. (2020) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2020.100902 · Cited by 89
Filmer, H. L., Mattingley, J. B., & Dux, P. E. (2019) · Cortex · doi:10.1016/j.cortex.2019.10.006 · Cited by 89
van den Akker, O. R., van Assen, M. A. L. M., Bakker, M., Elsherif, M. M., Wong, T. K., & Wicherts, J. M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02277-0 · Cited by 97
“The higher statistical power and larger sample sizes in pre-registered than non-preregistered studies are important, considering earlier findings that sample sizes across psychology are often insufficient to find meaningful effects (Bakker et al., 2016; Szucs & Ioannidis, 2017).”
BackgroundThe impact of sample size on the reproducibility of voxel-based lesion-deficit mappings
Lorca‐Puls, D. L., Gajardo‐Vidal, A., White, J., Seghier, M. L., Leff, A. P., Green, D. W., Crinion, J., Ludersdorfer, P., Hope, T. M. H., Bowman, H., et al. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.03.014 · Cited by 91
“, 2017; Ingre, 2013; Ioannidis, 2008) and how inadequate statistical power can lead to replication failures (Anderson et al., 2017; Bakker et al., 2012; Perugini et al., 2014; Simonsohn et al., 2014a; Szucs and Ioannidis, 2017).”
BackgroundResultInfluentialWith Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84
Metadata
- Authors
- Szucs, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-03-02
- Volume
- 15
- Issue
- 3
- Pages
- e2000797
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5333800
- OpenAlex
- W2951081425
- Cited by
- 761
- References
- 62
- Citations by year
- 2026: 28 · 2025: 43 · 2024: 120 · 2023: 75 · 2022: 80 · 2021: 76 · 2020: 94 · 2019: 102 · 2018: 91 · 2017: 46
- FWCI
- 81.43
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-03-05)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo (cc-by)
- Funders
- James S. McDonnell Foundation (220020370)
- Universiteit van Tilburg
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- effect size, statistical power, cognitive neuroscience, psychology literature, sample size, replication success, journal impact factor, null hypothesis, publication bias, statistical records
- MeSH terms
- Cognitive Neuroscience, Humans, Probability, Psychology, Publications, Surveys and Questionnaires, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.