Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature
Open accessPublished version of a preprint
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64-1.46) for nominally statistically significant results and D = 0.24 (0.11-0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite methodological challenges10–11, many of the findings of cognitive neuroscience provide a solid basis on which to build.”
From Cognitive computational neuroscience (2018) · cites this paper as [10]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In several disciplines, the average statistical power of studies was found to be significantly below the recommended 80% level (13 -16).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“such studies showed that power varies widely between subfields or methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For most published research findings to be false in psychology and neuroscience, for example, one must assume that the hypotheses tested in these disciplines are correct much less than 50% of the time (14, 18).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“and has contributed, in part, to the replication crisis in other fields (Button et al., 2013; Szucs and Ioannidis, 2017; Ioannidis et al., 2017).”
From With Little Power Comes Great Responsibility (2020) · cites this paper as Szucs and Ioannidis, 2017
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Effect size estimates: Current use, calculations, and interpretation.
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024338 · Cited by 5,199
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 9×
“…examined p value reporting errors in the cognitive neuroscience literature as recent studies found that 12.5% to 20% of psychology papers have major p value reporting errors which affect judgments about statistical significance (Bakker and Wicherts, 2011; Nuijten et al. 2015; Veldkamp et al. 2014).”
BackgroundMethodResultInfluentialSelf-citationCluster failure: why fMRI inferences for spatial extent have inflated false-positive rates
Eklund, A., Nichols, T. E., & Knutsson, H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1602413113 · Cited by 3,487
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
Cited by
Items 201–300 of 787
Gernigon, C., Hartigh, R. J. R. D., Vallacher, R. R., & van Geert, P. L. C. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231187324 · Cited by 43
“Among the recommendations formulated to overcome the crisis of reproducibility in psychological science is the search for statistical power (Simmons et al., 2011; Szucs & Ioannidis, 2017), which would be better guaranteed by large-scale studies.”
From theories to models to predictions: A Bayesian model comparison approach
Rouder, J. N., Haaf, J. M., & Aust, F. (2017) · Communication Monographs · doi:10.1080/03637751.2017.1394581 · Cited by 39
“The awareness that many analyses are underpowered (Rossi, 1990; Sedlmeier & Gigerenzer, 1989; Szucs & Ioannidis, 2017), based on questionable statistical practices (John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simonsohn, 2011), seem implausible in aggregate (Francis, 2012; Ioannidis, 2005)…”
BackgroundReplicability and generalizability in population psychiatric neuroimaging
Marek, S. A., & Laumann, T. O. (2024) · Neuropsychopharmacology · doi:10.1038/s41386-024-01960-w · Cited by 41
“Classically, population psychiatric neuroimaging studies have reported effects based on small samples (i.e., tens to a few hundred participants) [5, 6].”
Changing the culture of peer review for a more inclusive and equitable psychological science.
Aly, M., Colunga, E., Crockett, M. J., Goldrick, M., Gómez, P., Kung, F. Y. H., McKee, P. C., Pérez, M., Stilwell, S. M., & Diekman, A. B. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001461 · Cited by 37
Redefining effect size interpretations for psychotherapy RCTs in depression
Nordahl‐Hansen, A., Cogo‐Moreira, H., Panjeh, S., & Quintana, D. S. (2023) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2023.11.009 · Cited by 36
Zur Methodologie von Replikationsstudien
Erdfelder, E., & Ulrich, R. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000387 · Cited by 33
Peters, G.-J., & Crutzen, R. (2020) · Psychology and Health · doi:10.1080/08870446.2020.1757098 · Cited by 32
Brydges, C. R., & Bielak, A. A. M. (2019) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbz033 · Cited by 35
Mazza, C., Orrù, G., Burla, F., Monaro, M., Ferracuti, S., Colasanti, M., & Roma, P. (2019) · PLoS ONE · doi:10.1371/journal.pone.0227113 · Cited by 32
“In summarizing their assessment of replicability, Szucs and Ioannidis [50] concluded that: “Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature.”
BackgroundLoosen, A. M., Seow, T. X. F., & Hauser, T. U. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02427-y · Cited by 33
“However, we also did not identify any statistically significant associations between these symptom scores and the measures investigated here (cf. Supplemental Information). generalisability of neurocognitive findings (Button et al., 2013; Poldrack et al., 2017; Szucs & Ioannidis, 2017).”
BackgroundCairo, A. H., Green, J. D., Forsyth, D. R., Behler, A. M. C., & Raldiris, T. L. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167220903896 · Cited by 34
The Epistemic Importance of Establishing the Absence of an Effect
Fidler, F. M., Thorn, F. S., Barnett, A., Kambouris, S., & Kruger, A. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770407 · Cited by 32
“The average statistical power of psychology research has been estimated to be under 50% for the average effect sizes seen in psychology research (Cohen, 1962; Szucs & Ioannidis, 2017).”
BackgroundFleming, K. M., Coote, S. B., & Herring, M. P. (2019) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2019.101573 · Cited by 36
Pennington, C. R., Litchfield, D., McLatchie, N. M., & Heim, D. (2018) · European Journal of Social Psychology · doi:10.1002/ejsp.2540 · Cited by 33
“This may not be the best approach, however, because it is likely that some effect sizes reported in previous research are inflated due to small sample sizes or publication bias (Ioannidis, 2008; Nuijten, van Assen, Veldkamp, & Wicherts, 2015; Szucs & Ioannidis, 2017).”
ResultOptimal processing of surface facial EMG to identify emotional expressions: A data-driven approach
Rutkowska, J. M., Ghilardi, T., Vacaru, S. V., van Schaik, J. E., Meyer, M., Hunnius, S., & Oostenveld, R. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02421-4 · Cited by 38
Retrospective median power, false positive meta‐analysis and large‐scale replication
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1529 · Cited by 34
Self-citationAssessing the Evidential Value of Mental Fatigue and Exercise Research
Holgado, D., Mesquida, C., & Román-Caballero, R. (2023) · Sports Medicine · doi:10.1007/s40279-023-01926-w · Cited by 33
“Second, studies with low power designs are more likely to produce overestimated effect sizes [29, 104–106].”
BackgroundIrreproducible text-book “knowledge”: The effects of color bands on zebra finch fitness
Wang, D., Forstmeier, W., Ihle, M., Khadraoui, M., Jerónimo, S., Martin, K., & Kempenaers, B. (2018) · Evolution · doi:10.1111/evo.13459 · Cited by 35
“Evolution 72-4: 961–976 journals and funding agencies that prioritize novelty over solid replication (Song and Gilbody 1998; Collaboration 2015; Benjamin et al. 2017; Forstmeier et al. 2017; Szucs and Ioannidis 2017).”
BackgroundToward dynamic phenotypes and the scalable measurement of human behavior
Germine, L. T., Strong, R. W., Singh, S. M., & Sliwinski, M. J. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-020-0757-1 · Cited by 37
Schuster, R., Kaiser, T., Terhorst, Y., Meßner, E.-M., Strohmeier, L.-M., & Laireiter, A. (2021) · Psychological Medicine · doi:10.1017/s003329172100129x · Cited by 31
Oliver, D., Wong, C. M. J., Bøg, M., Jönsson, L., Kinon, B. J., Wehnert, A., Jørgensen, K. T., Irving, J., Stahl, D., McGuire, P., et al. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-01032-9 · Cited by 26
Kleinert, T., Waldschütz, M., Blau, J., Heinrichs, M., & Schiller, B. (2026) · Communications Psychology · doi:10.1038/s44271-025-00391-7 · Cited by 37
When we can trust computers (and when we can't)
Coveney, P. V., & Highfield, R. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0067 · Cited by 31
“Even worse performance is found in cognitive neuroscience [13].”
BackgroundHensel, W. M. (2020) · European Journal for Philosophy of Science · doi:10.1007/s13194-020-00317-6 · Cited by 31
“…the probability that a hypothesis being tested is actually false – are likely below 10% and the rate of false positives is likely over 90% (cf. Szucs & Ioannidis 2017 for similar estimates based on a different methodology and Bird 2018 for a general argument that low reproducibility may be the…”
BackgroundCombrisson, E., Allegra, M., Basanisi, R., Ince, R. A. A., Giordano, B. L., Bastin, J., & Brovelli, A. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119347 · Cited by 34
Saunders, B., Milyavskaya, M., & Inzlicht, M. (2022) · Nature Communications · doi:10.1038/s41467-022-30786-7 · Cited by 30
Power to the People: A Beginner’s Tutorial to Power Analysis using jamovi
Bartlett, J. E., & Charles, S. J. (2022) · Meta-Psychology · doi:10.15626/mp.2021.3078 · Cited by 31
Reprint of "Methodological and reporting quality in laboratory studies of human eating behavior"
Robinson, E., Bevelander, K. E., Field, M., & Jones, A. (2018) · Appetite · doi:10.1016/j.appet.2018.06.037 · Cited by 29
Human faces capture attention and attract first saccades without longer fixation
Morrisey, M. N., Hofrichter, R., & Rutherford, M. D. (2019) · Visual Cognition · doi:10.1080/13506285.2019.1631925 · Cited by 28
“This target number was chosen based on expected effect sizes, and because many studies in the field of psychology are under-powered (Ioannidis, 2005; Szucs & Ioannidis, 2017).”
MethodEllis, R. J. (2022) · eNeuro · doi:10.1523/eneuro.0017-22.2022 · Cited by 30
“Along with the studies of statistical power across neuroscience from Button, Nord, and colleagues, Szucs and Ioannidis examined effect sizes and statistical power in 26,841 statistical tests from 3801 cognitive neuroscience and psychology papers published between 2011 and 2014 in 18 specialty journals (Szucs and Ioann…”
BackgroundSingle-case cognitive neuropsychology in the age of big data
Medina, J., & Fischer‐Baum, S. (2017) · Cognitive Neuropsychology · doi:10.1080/02643294.2017.1321537 · Cited by 29
“Researchers are rightly concerned about how the small sample sizes that are common in behavioural and neuroimaging increase the likelihood of false-positive results and decrease the likelihood of replication (Button et al., 2013; Ioan-nidis, 2005; Szucs & Ioannidis, 2017; Yarkoni, 2009).”
BackgroundThe Sniffing Effect: Olfactory Sensitivity and Olfactory Imagery in Advertising
Lin, M.-H. J., Cross, S. N. N., Laczniak, R. N., & Childers, T. L. (2018) · Journal of Advertising · doi:10.1080/00913367.2017.1410739 · Cited by 31
“However, the practical challenges of time andmoney are often associated with the limitations that come with the use of such methods, including power issues and sample size, as cautioned by researchers (Szucs and Ioannidis 2017).”
BackgroundNature as a potential modulator of the error-related negativity: A registered report
LoTemplio, S. B., Scott, E. E., McDonnell, A. S., Hopman, R. J., Castro, S. C., McNay, G. D., McKinney, T. L., Greenberg, K., Payne, B. R., & Strayer, D. L. (2020) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2020.06.014 · Cited by 28
Suárez‐García, A., Hernández, M. Á., Arce, E., & Ribas, J. R. (2024) · Applied Sciences · doi:10.3390/app14104189 · Cited by 30
Stice, E. M., & Yokum, S. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.08.035 · Cited by 26
Chen, Y., & Canli, T. (2022) · Personality Neuroscience · doi:10.1017/pen.2021.5 · Cited by 32
“MRI studies have come under criticism for reporting underpowered and non-replicable findings (Button et al., 2013; Szucs & Ioannidis, 2017).”
BackgroundPublished correlational effect sizes in social and developmental psychology
Weinerová, J., Szűcs, D., & Ioannidis, J. P. A. (2022) · Royal Society Open Science · doi:10.1098/rsos.220311 · Cited by 28
“Szűcs D, Ioannidis JPA.”
BackgroundMethodInfluentialSelf-citationSteinfath, M., Vogl, S., Violet, N., Schwarz, F., Mielke, H., Selhorst, T., Greiner, M., & Schönfelder, G. (2018) · PLoS ONE · doi:10.1371/journal.pone.0202762 · Cited by 26
Action Effects on Visual Perception of Distances: A Multilevel Bayesian Meta-Analysis
Molto, L., Nalborczyk, L., Palluel-Germain, R., & Morgado, N. (2020) · Psychological Science · doi:10.1177/0956797619900336 · Cited by 28
“Given the usual effect sizes observed in psychology, we did not expect the average effect size to be larger than 1.5 (Szucs & Ioannidis, 2017).”
BackgroundInfluentialQuestionable research practices following pre-registrationPreprint
Ikeda, A., Xu, H.-Q., Fuji, N., Zhu, S., & Yamada, Y. (2019) · PsyArXiv (OSF Preprints) · doi:10.24602/sjpr.62.3_281 · Cited by 26
Examining Psychological Science Through Systematic Meta-Method Analysis: A Call for Research
Elson, M. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919863296 · Cited by 26
Martínez‐Pérez, V., Campoy, G., Palmero, L. B., & Fuentes, L. J. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00683 · Cited by 27
“Moreover, given the issues of reproducibility in cognitive neurosciences (Szucs and Ioannidis, 2017), and the current state of the art of NIBS, with non-trivial evidence collected that challenges reliable effects of a single session of tDCS on behavior (Bestmann et al., 2015; Horvath et al., 2015;…”
BackgroundEffects of Transcranial Direct Current Stimulation on Brain Networks Related to Creative Thinking
Koizumi, K., Ueda, K., Li, Z., & Nakao, M. (2020) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.541052 · Cited by 28
#EEGManyLabs: Investigating the Replicability of Influential EEG ExperimentsPreprint
Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C., Beste, C., Bland, A. R., Bradford, D. E., Bublatzky, F., Busch, N. A., et al. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/528nr · Cited by 27
“Further, they are characterised by the use of rich, but also noisy, multi-dimensional data sets, which allows for a multitude of analytical choices (Szucs & Ioannidis, 2017), and thereby the “ garden of forking paths ” (Gelman & Loken, 2013).”
Estimating power in (generalized) linear mixed models: an open introduction and tutorial in R.Preprint
Kumle, L., Võ, M. L. ‐., & Draschkow, D. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/vxfbh · Cited by 26
Fraley, R. C., Chong, J. Y., Baacke, K. A., Greco, A. J., Guan, H., & Vazire, S. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221120217 · Cited by 28
“More recent surveys adopt similar approaches but in some cases, supplement manual coding with computer scripts that use regular expressions to extract information of interest (e.g., Szucs & Ioannidis, 2017).”
BackgroundMethodInfluentialElevated levels of mixed-hand preference in dyslexia: Meta-analyses of 68 studies
Packheiser, J., Παπαδάτου-Παστού, Μ., Koufaki, A., Paracchini, S., Stein, C. C., Schmitz, J., & Ocklenburg, S. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105420 · Cited by 26
Exploring perceptions of meaningfulness in visual representations of bivariate relationships
Beribisky, N., Davidson, H. A., & Cribbie, R. A. (2019) · PeerJ · doi:10.7717/peerj.6853 · Cited by 25
Lafit, G., Adolf, J., Dejonckheere, E., Myin‐Germeys, I., Viechtbauer, W., & Ceulemans, E. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/dq6ky · Cited by 25
What Has Replication Ever Done for Us? Insights from Neuroimaging of Speech Perception
Evans, S. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00041 · Cited by 25
“Unfortunately, functional imaging studies often have small sample sizes (e.g., n < 20) resulting in low statistical power and inflated effect sizes, making them less likely to be successfully reproduced (Carp, 2012; Button et al., 2013; Szucs and Ioannidis, 2016; Poldrack et al., 2017).”
BackgroundCampbell’s Law Explains the Replication Crisis: Pre-Registration Badges Are History Repeating
Klonsky, E. D. (2024) · Assessment · doi:10.1177/10731911241253430 · Cited by 27
“…Science (https://www.cos.io); and the proliferation of papers highlighting features that help support replicability, including large sample sizes (Szucs & Ioannidis, 2017), valid measurement (Lilienfeld & Strother, 2020), analytic robustness checks (Steegen et al., 2016), and replication across…”
Si, X., Li, S., Xiang, S., Yu, J., & Ming, D. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac25d9 · Cited by 26
“For example, Szucs et al found that for small effect size (Cohen’s d < 0.2), the median statistical power of cognitive neuroscience studies was 0.11; for medium effect size (0.2 < d < 0.5), the median statistical power of cognitive neuroscience studies was 0.40 [65].”
InfluentialLynn, M. (2017) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2017.10.004 · Cited by 23
Ingre, M., & Nilsonne, G. (2018) · Royal Society Open Science · doi:10.1098/rsos.181190 · Cited by 23
“The likely range suggests 71%—73% power, which seems optimistic for this research; such large statistical power has been indicated only for larger than medium effect sizes in psychological research (Rossi, 1990; Szucs & Ioannidis, 2017).”
BackgroundInfluentialNeurolaw in Australia: The Use of Neuroscience in Australian Criminal Proceedings
Alimardani, A., & Chin, J. M. (2019) · Neuroethics · doi:10.1007/s12152-018-09395-z · Cited by 22
Eliciting Big Data From Small, Young, or Non-standard Languages: 10 Experimental Challenges
Leivada, E., D’Alessandro, R., & Grohmann, K. Κ. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00313 · Cited by 26
“, 2010), and (iii) a small sample size may facilitate obtaining statistically significant—hence, publishable—results that are, however, not doing justice to real-world conditions by means of finding an effect in the selected sample, when there is none at the population (i.e., false positives/Type I errors; Szucs and I…”
BackgroundBrauchli, C., Leipold, S., & Jäncke, L. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116513 · Cited by 24
“In recent years, studies in neuroscience have repeatedly been criticized for analyzing data from small samples (Button et al., 2013; Nord et al., 2017; Szucs and Ioannidis, 2017).”
BackgroundDemidenko, M. I., Huntley, E. D., Jahn, A., Thomason, M. E., Monk, C. S., & Keating, D. P. (2020) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2020.100798 · Cited by 22
Power(ful) guidelines for experimental economists
Vasilaky, K., & Brock, J. (2020) · Journal of the Economic Science Association · doi:10.1007/s40881-020-00090-5 · Cited by 22
Changing the logic of replication: A case from infant studies
Margoni, F., & Shepperd, M. (2020) · Infant Behavior and Development · doi:10.1016/j.infbeh.2020.101483 · Cited by 22
Puntis, S., Oliver, D., & Fusar-Poli, P. (2021) · Schizophrenia Research · doi:10.1016/j.schres.2021.01.005 · Cited by 21
Schuster, R., Schreyer, M. L., Kaiser, T., Berger, T., Klein, J. P., Moritz, S., Laireiter, A.-R., & Trutschnig, W. (2020) · Internet Interventions · doi:10.1016/j.invent.2020.100313 · Cited by 21
Improving Reading Through Videogames and Digital Apps: A Systematic Review
Ostiz-Blanco, M., Bernácer, J., Garcia-Arbizu, I., Diaz-Sanchez, P., Rello, L., Lallier, M., & Arrondo, G. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.652948 · Cited by 25
“This categorization was driven by the fact that low power combined with a high proportion of statistically significant results could indicate a high proportion of false positives in the literature (Szucs and Ioannidis, 2017).”
ResultLi, W., Germine, L. T., Mehr, S. A., Srinivasan, M., & Hartshorne, J. K. (2022) · Infant and Child Development · doi:10.1002/icd.2348 · Cited by 22
“…Hartshorne & Schachner, 2012; Maddock, 1999; Maxwell, 2004; Mone et al., 1996; Osborne, 2008; Rossi, 1997; Schäfer & Schwarz, 2019; Sedlmeier & Gigerenzer, 1992; Stanley et al., 2018; Szucs & Ioannidis, 2017; Vankov et al., 2014; Ward, 2002).1 That is, average statistical power is less than 50%.”
Greber, M., Rogenmoser, L., Elmer, S., & Jäncke, L. (2018) · eNeuro · doi:10.1523/eneuro.0333-18.2018 · Cited by 22
“Due to the resource-intensive data acquisition, many neuroscientific studies use small sample sizes, resulting in low power (Szucs and Ioannidis, 2017).”
BackgroundAdvancing the Cognitive Science of Religion through Replication and Open Science
Hoogeveen, S., & van Elk, M. (2020) · Journal for the Cognitive Science of Religion · doi:10.1558/jcsr.39039 · Cited by 21
Smalheiser, N. R., Graetz, E. E., Yu, Z., & Wang, J. (2021) · PLoS ONE · doi:10.1371/journal.pone.0243668 · Cited by 24
Frederick, D. E., Brown, A., Tacopina, S., Mehta, N., Vujovic, M., Brim, E., Amina, T., Fixsen, B. R., & Kay, L. M. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.1797-16.2017 · Cited by 21
Default-Switching: The Hidden Cost of DefaultsPreprint
Jachimowicz, J. M., Duncan, S., & Weber, E. U. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2727301 · Cited by 23
“We specifically code for decade to determine whether default effect sizes have changed over the time that they have been studied, as effect sizes in published research frequently decrease over time (Szucs et al., 2015).”
BackgroundLievore, R., Caviola, S., & Mammarella, I. C. (2024) · Learning and Individual Differences · doi:10.1016/j.lindif.2024.102459 · Cited by 25
Correia, V. J. (2021) · SN Computer Science · doi:10.1007/s42979-021-00962-5 · Cited by 22
“Although significant results are important to note and interpret, it should be acknowledged that these significant findings merely highlight notions which appear worthy of further investigation and replication [89, 90].”
BackgroundIrving, D., Clark, R. W. A., Lewandowsky, S., & Allen, P. (2022) · Journal of Experimental Psychology Applied · doi:10.1037/xap0000408 · Cited by 18
Antoniou, K. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-023-02245-x · Cited by 24
The Benefits of Reporting Critical-Effect-Size Values
Perugini, A., Gambarota, F., Toffalini, E., Lakens, D., Pastore, M., Finos, L., Core Team Psicostat, & Altoè, G. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251335298 · Cited by 21
“Researchers often publish studies with low power for medium to small effect sizes (Szucs & Ioannidis, 2017) and use resource limitations as the main justification for the sample size (Lakens, 2022).”
The influence of partnering on the occurrence of construction requirement conflicts and disputes
Phillips-Alonge, O. K. (2018) · International Journal of Construction Management · doi:10.1080/15623599.2018.1435236 · Cited by 20
“The 95% confidence interval was estimated in addition to the effect size ‘Cohen’s r2’ according to the suggestions of Szucs and Ioannidis (2017) to determine if the sample and population means are genuinely different.”
MethodUlrich, R., Miller, J. O., & Erdfelder, E. (2018) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000319 · Cited by 21
“Moreover, it proceeds from the exact sampling distribution of t-tests rather than the normal approximation, an approximation that might be poor because of the small sample sizes in many studies (Szucs & Ioannidis, 2017).”
BackgroundAydin, E., Gokhale, M., Azizoğlu, S., & Suphioglu, C. (2019) · Cells · doi:10.3390/cells8060620 · Cited by 20
Ruby, E., Maniscalco, B., & Peters, M. A. K. (2018) · Consciousness and Cognition · doi:10.1016/j.concog.2018.04.009 · Cited by 23
Optimizing the scientific study of suicide with open and transparent research practices
Carpenter, T. P., & Law, K. C. (2021) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12665 · Cited by 24
Schneider, J., Matheson, E. L., Tinoco, A., Gentili, C., White, P., Boucher, C., Silva-Breen, H. L., Goorevich, A., Diedrichs, P. C., & LaVoi, N. M. (2023) · Journal of Applied Sport Psychology · doi:10.1080/10413200.2023.2212023 · Cited by 19
“Small samples are also associated with large variances of the estimated effect, increasing the chances of obtaining large effect sizes, which are likely biased and unlikely to be replicated, due to low power (Szucs & Ioannidis, 2017).”
BackgroundGandhi, L., Manning, B. S., & Duckworth, A. L. (2024) · Current Directions in Psychological Science · doi:10.1177/09637214241268222 · Cited by 26
Initial Evidence for Increased Weather Salience in Autism Spectrum Conditions
Bolton, M. J., Blumberg, W. G., Ault, L. K., Mogil, H. M., & Hanes, S. H. (2020) · Weather Climate and Society · doi:10.1175/wcas-d-18-0100.1 · Cited by 20
“3 Relatively common in psychological science when attempting to identify meaningful findings that actually occur in the real world, since many studies are underpowered (Szucs and Ioannidis 2017).”
BackgroundPhysical Activity Following Hip Arthroscopy in Young and Middle-Aged Adults: A Systematic Review
Jones, D., Crossley, K. M., Ackerman, I. N., Hart, H. F., Dundules, K., O’Brien, M., Mentiplay, B. F., Heerey, J., & Kemp, J. L. (2020) · Sports Medicine - Open · doi:10.1186/s40798-020-0234-8 · Cited by 19
“This may increase potential for bias and magnification of positive effects [159].”
BackgroundReligious belief and cognitive conflict sensitivity: A preregistered fMRI study
Hoogeveen, S., Snoek, L., & van Elk, M. (2020) · Cortex · doi:10.1016/j.cortex.2020.04.011 · Cited by 20
Paul, K., Short, C. A., Beauducel, A., Carsten, H. P., Härpfer, K., Hennig, J., Hewig, J. S., Hildebrandt, A., Kührt, C., Mueller, E. M., et al. (2022) · Personality Science · doi:10.5964/ps.7177 · Cited by 22
“…undisclosed flexibility in data analysis are major drivers of the current replicability crisis (Open Science Collaboration, 2015; Simmons et al., 2011), to which psychology and cognitive neuroscience are not immune (Button et al., 2013; Open Science Collaboration, 2015; Szucs & Ioannidis, 2017).”
BackgroundEstimation accuracy in the psychological sciences
Davis‐Stober, C. P., Dana, J. D., & Rouder, J. N. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207239 · Cited by 22
“A recent study by Szucs and Ioannidis [21] examined more than 100,000 statistical records from about 10,000 cognitive neuroscience and psychology papers published within the past five years.”
BackgroundTirrell, J. M., Gansert, P. K., Dowling, E. M., Geldhof, G. J., Lerner, J. V., King, P. E., Iraheta, G., Williams, K. E., Sim, A. T. R., & Lerner, R. M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000363 · Cited by 19
“Given the practical and economic constraints we faced in regard to sampling enough youth from enough sites to use nesting analyses, such as random slopes in hierarchical linear modeling, we instead used data analysis procedures that assessed effect sizes (see Cohen, 1988; Szucs & Ioannidis, 2017).”
MethodA Bayes Factor for Replications of ANOVA Results
Harms, C. (2018) · The American Statistician · doi:10.1080/00031305.2018.1518787 · Cited by 24
“Since power is notoriously low in psychological science (Sedlmeier and Gigerenzer 1989; Button et al. 2013; Szucs and Ioannidis 2017), confidence intervals tend to be generally wide for original studies from the published literature (Morey and Lakens 2016).”
BackgroundThe New Statistics for Neuroscience Majors: Thinking in Effect SizesPreprint
Calin-Jageman, R. J. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/zvm9a · Cited by 19
“For example, once adjusted for total brain volume there doesn’t seem to be any sexual dimorphism in the volume of the hippocampus (Tan et al., 2016).”
BackgroundInfluentialSzűcs, D., & Ioannidis, J. P. A. (2021) · PLoS Biology · doi:10.1371/journal.pbio.3001151 · Cited by 19
Self-citationKornbrot, D. E., Wiseman, R. J., & Georgiou, G. J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0192808 · Cited by 18
Multivariate prediction of cognitive performance from the sleep electroencephalogram
Ujma, P. P., Bódizs, R., Dresler, M., Simor, P., Purcell, S., Stone, K. L., Yaffe, K. S., & Redline, S. S. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120319 · Cited by 19
“2 differences in cognitive performance arise due to the summed effects of many small biological differences – then the signal-noise ratio of the detected associations is poor in small samples, leading to many false positive findings which do not replicate while the true associations may remain undetected (Button et al…”
BackgroundSleep and intelligence: critical review and future directions
Ujma, P. P., Bódizs, R., & Dresler, M. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.01.009 · Cited by 18
Should Psychology Journals Adopt Specialized Statistical Review?
Hardwicke, T. E., Frank, M. C., Vazire, S., & Goodman, S. N. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919858428 · Cited by 23
“…sample sizes that statistical tests are unlikely to be sufficiently powered to detect plausible effects (e.g., Button et al., 2013; Cohen, 1962; Fraley & Vazire, 2014; Sedlmeier & Gigerenzer, 1989; Stanley, Carter, & Doucouliagos, 2018; Szucs & Ioannidis, 2017; Vankov, Bowers, & Munafò, 2014).”
BackgroundA Systematic Review on the Evolution of Power Analysis Practices in Psychological Research
Vankelecom, L., Schacht, O., Laroy, N., Loeys, T., & Moerkerke, B. (2025) · Psychologica Belgica · doi:10.5334/pb.1318 · Cited by 19
Gilmore, R. O., Cole, P. M., Verma, S., van Aken, M. A. G., & Worthman, C. M. (2020) · Child Development Perspectives · doi:10.1111/cdep.12360 · Cited by 18
“…have come under especially intense criticism, much of it directed at specific weaknesses: Studies lack statistical power to detect reported effects (Szucs & Ioannidis, 2017), analysis workflows cannot be readily reproduced, statistical errors are common (Nuitjen et al., 2015), independent…”
BackgroundRengasamy, M. L., Moriarity, D. P., Kraynak, T. E., Tervo‐Clemmens, B., & Price, R. B. (2023) · Neuropsychopharmacology · doi:10.1038/s41386-023-01621-4 · Cited by 19
Are mental distress and cognitive impairment related in temporal lobe epilepsy?
Demin, K., Berger, J., Holtkamp, M., & Bengner, T. (2018) · Epilepsy Research · doi:10.1016/j.eplepsyres.2018.07.020 · Cited by 18
Demidenko, M. I., Weigard, A. S., Ganesan, K., Jang, H., Jahn, A., Huntley, E. D., & Keating, D. P. (2021) · Brain and Behavior · doi:10.1002/brb3.2093 · Cited by 19
“Currently, a large proportion of fMRI studies are substantially underpowered for finding the effect they are testing (Szucs & Ioannidis, 2017, 2020).”
BackgroundMachery, É. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00354-9 · Cited by 24
Metadata
- Authors
- Szucs, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-03-02
- Volume
- 15
- Issue
- 3
- Pages
- e2000797
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5333800
- OpenAlex
- W2951081425
- Cited by
- 761
- References
- 62
- Citations by year
- 2026: 28 · 2025: 43 · 2024: 120 · 2023: 75 · 2022: 80 · 2021: 76 · 2020: 94 · 2019: 102 · 2018: 91 · 2017: 46
- FWCI
- 81.43
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-03-05)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo (cc-by)
- Funders
- James S. McDonnell Foundation (220020370)
- Universiteit van Tilburg
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- effect size, statistical power, cognitive neuroscience, psychology literature, sample size, replication success, journal impact factor, null hypothesis, publication bias, statistical records
- MeSH terms
- Cognitive Neuroscience, Humans, Probability, Psychology, Publications, Surveys and Questionnaires, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.