Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature
Open accessPublished version of a preprint
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64-1.46) for nominally statistically significant results and D = 0.24 (0.11-0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite methodological challenges10–11, many of the findings of cognitive neuroscience provide a solid basis on which to build.”
From Cognitive computational neuroscience (2018) · cites this paper as [10]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In several disciplines, the average statistical power of studies was found to be significantly below the recommended 80% level (13 -16).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“such studies showed that power varies widely between subfields or methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For most published research findings to be false in psychology and neuroscience, for example, one must assume that the hypotheses tested in these disciplines are correct much less than 50% of the time (14, 18).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“and has contributed, in part, to the replication crisis in other fields (Button et al., 2013; Szucs and Ioannidis, 2017; Ioannidis et al., 2017).”
From With Little Power Comes Great Responsibility (2020) · cites this paper as Szucs and Ioannidis, 2017
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Effect size estimates: Current use, calculations, and interpretation.
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024338 · Cited by 5,199
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 9×
“…examined p value reporting errors in the cognitive neuroscience literature as recent studies found that 12.5% to 20% of psychology papers have major p value reporting errors which affect judgments about statistical significance (Bakker and Wicherts, 2011; Nuijten et al. 2015; Veldkamp et al. 2014).”
BackgroundMethodResultInfluentialSelf-citationCluster failure: why fMRI inferences for spatial extent have inflated false-positive rates
Eklund, A., Nichols, T. E., & Knutsson, H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1602413113 · Cited by 3,487
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
Cited by
Items 101–200 of 787
Azari, B., Westlin, C., Satpute, A. B., Hutchinson, J. B., Kragel, P. A., Hoemann, K., Khan, Z., Wormwood, J. B., Quigley, K. S., Erdogmus, D., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-77117-8 · Cited by 90
Enhancing precision in human neuroscience
Nebe, S., Reutter, M., Baker, D. H., Bölte, J., Domes, G., Gamer, M., Gärtner, A., Gießing, C., Gurr, C., Hilger, K., et al. (2023) · eLife · doi:10.7554/elife.85980 · Cited by 82
Null Effects and Publication Bias in Special Education Research
Cook, B. G., & Therrien, W. J. (2017) · Behavioral Disorders · doi:10.1177/0198742917709473 · Cited by 82
“Because research findings are subject to random error as well as multiple types of systematic bias (Cook, 2014), study findings indicating that a practice works may sometimes be invalid (Ioannidis, 2005; Szucs & Ioannidis, 2017)—leading to ineffective practice and policies based on those findings.”
BackgroundSolla, F., Tran, A., Bertoncelli, D., Musoff, C., & Bertoncelli, C. M. (2018) · Clinical Spine Surgery A Spine Publication · doi:10.1097/bsd.0000000000000695 · Cited by 81
No evidence for a core deficit in developmental dyscalculia or mathematical learning disabilities
Mammarella, I. C., Toffalini, E., Caviola, S., Colling, L. J., & Szűcs, D. (2021) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13397 · Cited by 78
Algermissen, J., & Mehler, D. M. A. (2018) · Journal of Neurophysiology · doi:10.1152/jn.00765.2017 · Cited by 70
“Recent estimates suggest that, for this reason, more than 50% of published findings in neuroscience are likely to be false positives (Szucs and Ioannidis 2017): treatments that are reported to work may not work reliably, genes that are reported to contribute to a phenotype may contribute little,…”
BackgroundCaviola, S., Colling, L. J., Mammarella, I. C., & Szűcs, D. (2020) · Developmental Science · doi:10.1111/desc.12957 · Cited by 72
“We conclude that our results replicate the previously reported zero-order correlations from the large study of Halberda et al. (2012). Two other large studies measured larger effect sizes than us and Halberda et al. (2012). Lyons et al.”
BackgroundMethodResultInfluentialLópez‐Nicolás, R., López-López, J. A., Rubio‐Aparicio, M., & Sánchez‐Meca, J. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01644-z · Cited by 80
Primbs, M. A., Pennington, C. R., Lakens, D., Silan, M. A. A., Lieck, D., Forscher, P. S., Buchanan, E. M., & Westwood, S. J. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221100420 · Cited by 75
“Given a tradition of running small underpowered studies and selectively reporting statistically significant results (see Button et al., 2013; Szucs & Ioannidis, 2017), the mechanism Götz et al. described is, in fact, reversed: It is not a pressure to publish large effects that encourages…”
BackgroundNBS-Predict: A prediction-based extension of the network-based statistic
Serin, E., Zalesky, A., Matory, A., Walter, H., & Kruschwitz, J. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118625 · Cited by 76
Distinct roles of theta and alpha oscillations in the involuntary capture of goal-directed attention
Harris, A. M., Dux, P. E., Jones, C. N., & Mattingley, J. B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.03.008 · Cited by 73
“…on the low rates of replication of experimental results in several scientific fields, including cognitive science (Open Science Collaboration 2015; Szucs & Ioannidis 2016), economics (Camerer et al. 2016), and medicine (Begley & Ellis, 2012; Prinz et al. 2011), this replication provided…”
A hitchhiker’s guide to working with large, open-source neuroimaging datasets
Horien, C. L., Noble, S., Greene, A. S., Lee, K., Barron, D. S., Gao, S., O’Connor, D. A., Salehi, M., Dadashkarimi, J., Shen, X., et al. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-01005-4 · Cited by 76
“While a large number of subjects permits a closer estimate of how sample effect sizes map onto true population effect sizes 67, 68, 69 , even small effects with potentially little practical importance can be “ statistically significant.”
BackgroundElliott, M. L., Knodt, A. R., Ireland, D. R., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/681700 · Cited by 72
Psychedelic Research and the Need for Transparency: Polishing Alice’s Looking Glass
Petranker, R., Anderson, T., & Farb, N. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01681 · Cited by 69
“Open materials and data is even more important for expensive neuroimaging where data is scarce and researcher degrees of freedom are high, resulting in especially limited replicability (Button et al., 2013; Szucs and Ioannidis, 2017).”
BackgroundNeuroadaptive Bayesian Optimization and Hypothesis Testing
Lorenz, R., Hampshire, A., & Leech, R. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.01.006 · Cited by 71
Ryali, S., Zhang, Y., Angeles, C. D. L., Supekar, K., & Menon, V. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2310012121 · Cited by 85
“Our second goal was to address the reproducibility crisis in sex differences research (52, 53) by investigating the replicability and generalization of sex differences in the functional organization of the human brain.”
Intracranial source activity (eLORETA) related to scalp‐level asymmetry scores and depression status
Smith, E. E., Cavanagh, J. F., & Allen, J. J. B. (2017) · Psychophysiology · doi:10.1111/psyp.13019 · Cited by 74
“…to poor spatial sampling of the scalp and small sample sizes: montages with few channels ( 64) may miss contributions from small patches of cortex situated between electrodes (Nunez and Srinivasan, 2006), and small sample sizes are prone to irreproducible results (Szucs and Ioannidis, 2016).”
BackgroundHanel, P. H. P., & Mehler, D. (2019) · Public Understanding of Science · doi:10.1177/0963662519834193 · Cited by 59
“We use the term small/marginal effect in a relative sense: 25% of the effect sizes reported in psychological literature are smaller than d = 0.30 (25th to 75th percentile: 0.29- 0.96; Figure 3a in Szucs & Ioannidis, 2017).”
MethodInfluentialData quality over data quantity in computational cognitive neuroscience
Kolossa, A., & Kopp, B. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.01.005 · Cited by 68
“…and Ulrich (2013); Paap and Sawi (2016); Loken and Gelman (2017); but see Chapman and Chapman (1973) and Strauss (2001) for early discussions) and in cognitive neuroscience (see the work cited above plus e.g., Leue et al. (2013); Steele et al. (2016); Fischer et al. (2017); Thigpen et al. (2017)).”
Good scientific practice in EEG and MEG research: Progress and perspectives
Niso, G., Krol, L. R., Combrisson, E., Dubarry, A., Elliott, M., Clément, F., Héjja‐Brichard, Y., Herbst, S. K., Jerbi, K., Ković, V., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119056 · Cited by 75
Quantifying the benefits of using decision models with response time and accuracy data
Stafford, T., Pirrone, A., Croucher, M., & Krystalli, A. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01372-w · Cited by 69
Diffusion-Weighted Imaging in Mild Traumatic Brain Injury: A Systematic Review of the Literature
Lindsey, H. M., Hodges, C. B., Greer, K. M., Wilde, E. A., & Merkley, T. L. (2021) · Neuropsychology Review · doi:10.1007/s11065-021-09485-5 · Cited by 74
Neuroimaging of individual differences: A latent variable modeling perspective
Cooper, S. R., Jackson, J. J., Barch, D. M., & Braver, T. S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.12.022 · Cited by 57
“The importance of power cannot be overstated, and low power across all of psychological and neuroscience research has come under scrutiny in recent years (Button et al., 2013; Open Science Collaboration, 2015; Szucs and Ioannidis, 2016).”
Generalization Bias in Science
Peters, U., Krauss, A., & Braganza, O. (2022) · Cognitive Science · doi:10.1111/cogs.13188 · Cited by 63
“In fact, recent studies found that, across the cognitive and behavioral sciences, published sample sizes are persistently too small to support the purported conclusions (Button et al., 2013; Smaldino & McElreath, 2016; Szucs & Ioannidis, 2017).”
BackgroundThe Effect of a Head‐mounted Low Vision Device on Visual Function
Wittich, W., Lorenzini, M.-C., Markowitz, S. N., Tolentino, M., Gartner, S., Goldstein, J. E., & Dagnelie, G. (2018) · Optometry and Vision Science · doi:10.1097/opx.0000000000001262 · Cited by 63
Meta-analysis of the relation between interview anxiety and interview performance.
Powell, D. M., Stanley, D. J., & Brown, K. N. (2018) · Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement · doi:10.1037/cbs0000108 · Cited by 62
“Given the tendency to draw conclusions from single studies, it is unfortunate that most studies are under powered and have correspondingly low informational value (Ioannidis, 2005; Szucs & Ioannidis, 2017).”
BackgroundRegulating Child Sex Robots: Restriction or Experimentation?
Danaher, J. (2019) · Medical Law Review · doi:10.1093/medlaw/fwz002 · Cited by 73
Power-Enhanced Funnel Plots for Meta-Analysis: The Sunset Funnel Plot
Kossmeier, M., Tran, U. S., & Voracek, M. (2020) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000392 · Cited by 54
Sex is a defining feature of neuroimaging phenotypes in major brain disorders
Salminen, L. E., Tubi, M. A., Bright, J. K., Thomopoulos, S. I., Wieand, A., & Thompson, P. M. (2021) · Human Brain Mapping · doi:10.1002/hbm.25438 · Cited by 68
The Power of Replicated Measures to Increase Statistical Power
Goulet, M.-A., & Cousineau, D. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919849434 · Cited by 67
“A systematic lack of statistical power can be harmful for cumulative science (Abraham & Russell, 2008; Szucs & Ioannidis, 2017).”
BackgroundThe Potential for Outdoor Nature-Based Interventions in the Treatment and Prevention of Depression
Owens, M., & Bunce, H. L. I. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.740210 · Cited by 56
Permutation tests are robust and powerful at 0.5% and 5% significance levels
Noguchi, K., Konietschke, F., Marmolejo‐Ramos, F., & Pauly, M. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01595-5 · Cited by 67
“It is worth noting that we agree with Wasserstein and Lazar (2016) who emphasize that “Scientific conclusions [...] should not be based only on whether a p-value passes a specific threshold”. Instead of thresholding, they recommend researchers to consider many other contextual factors to gather scientific evidence rat…”
BackgroundResultStatistical power in network neuroscience
Helwegen, K., Libedinsky, I., & van den Heuvel, M. P. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.12.011 · Cited by 63
Nonparametric multiple comparisons
Noguchi, K., Abel, R. S., Marmolejo‐Ramos, F., & Konietschke, F. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01247-9 · Cited by 68
Clayson, P. E., Carbine, K. A., & Larson, M. J. (2020) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2020.01.005 · Cited by 55
Legislating Phonics: Settled Science or Political Polemics?
Reinking, D., Hruby, G. G., & Risko, V. J. (2023) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/01614681231155688 · Cited by 62
“…among brain researchers, either theoretically or in the interpretation of findings (Poldrack et al., 2017; Price, 2017; Ramus et al., 2018; Szücs & Ioannidis, 2017), and many are reluctant to speculate about, let alone apply, their data toward instructional issues (Katzir & Pare-Blagoev,…”
BackgroundDissociating frontoparietal brain networks with neuroadaptive Bayesian optimization
Lorenz, R., Violante, I. R., Monti, R. P., Montana, G., Hampshire, A., & Leech, R. (2018) · Nature Communications · doi:10.1038/s41467-018-03657-3 · Cited by 63
“Importantly, we found consistent replicable results across and within each individual subject; this is not a trivial advance in light of the growing concerns about the reproducibility of fMRI fi ndings 47 – 51 .”
BackgroundMethodological and reporting quality in laboratory studies of human eating behavior
Robinson, E., Bevelander, K. E., Field, M., & Jones, A. (2018) · Appetite · doi:10.1016/j.appet.2018.02.008 · Cited by 59
“For example, very small sample sizes are thought to increase the likelihood of false positive results and inflate effect size estimates (Szucs & Ioannidis, 2017).”
Lamberink, H. J., Otte, W. M., Sinke, M. R. T., Lakens, D., Glasziou, P., Tijdink, J. K., & Vinkers, C. H. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.06.014 · Cited by 64
Estimating the evidential value of significant results in psychological science
Aczél, B., Pálfi, B., & Szászi, B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0182651 · Cited by 56
Population modeling with machine learning can enhance measures of mental health
Dadi, K., Varoquaux, G. P., Houenou, J., Bzdok, D., Thirion, B., & Engemann, D. A. (2021) · GigaScience · doi:10.1093/gigascience/giab071 · Cited by 60
“, 2010), small sample sizes 5 (Szucs and Ioannidis, 2017), and reliance on dichotomized groups, i.”
Quantitative Political Science Research Is Greatly Underpowered
Arel‐Bundock, V., Briggs, R. C., Doucouliagos, H., Aviña, M. M., & Stanley, T. D. (2024) · The Journal of Politics · doi:10.1086/734279 · Cited by 66
“14 Using a somewhat different method, Szucs and Ioannidis (2017) find similarly low power in psychology, cognitive neuroscience, and medically-oriented neuroscience.”
How faculty define quality, prestige, and impact of academic journals
Morales, E., McKiernan, E. C., Niles, M. T., Schimanski, L. A., & Alperín, J. P. (2021) · PLoS ONE · doi:10.1371/journal.pone.0257340 · Cited by 59
Methodological challenges in the comparison of infant fMRI across age groups
Cusack, R., McCuaig, O., & Linke, A. C. (2017) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2017.11.003 · Cited by 56
Statistical Inference and the Replication Crisis
Colling, L. J., & Szűcs, D. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0421-4 · Cited by 62
“Some journals have gone so far as to ban p values from their pages (Trafimow and Marks, 2014) while others have suggested that what gets to be called statistically significant should be redefined (Benjamin et al., 2017).”
BackgroundBiomarkers for Alzheimer's Disease Defined by a Novel Brain Functional Network Measure
Song, Z., Deng, B., Wang, J., & Wang, R. (2019) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2018.2834546 · Cited by 50
“78 (higher than the median and mean power of large effects detection in the current studies summarized in [62]), and (iv) allocation ratio N2/N1= 1.”
BackgroundSaunders, B., & Inzlicht, M. (2020) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2020.05.008 · Cited by 50
A systematic and meta-analytic review of the impact of sleep restriction on memory formation
Crowley, R., Alderman, E., Javadi, A., & Tamminen, J. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105929 · Cited by 58
“These figures are somewhat higher than those typically found in psychology and cognitive neuro-science (Szucs and Ioannidis, 2017), but the widely accepted convention is still much higher at 80 % (di Stephano, 2003).”
BackgroundAbreu‐Mendoza, R. A., Chamorro, Y., García-Barrera, M. A., & Matute, E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0209267 · Cited by 51
Cockrell, C., & An, G. C. (2017) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2017.07.016 · Cited by 53
Cognitive and Neural Correlates of Mathematical Giftedness in Adults and Children: A Review
Myers, T., Carey, E., & Szűcs, D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01646 · Cited by 53
“…negatives), exaggerating measured effect sizes associated with statistically significant effects, and having a high false report probability (i.e., the probability that a statistically significant finding is false; Pollard and Richardson, 1987; Button et al., 2013; Szucs and Ioannidis, 2017).”
BackgroundFochler, M., & Sigl, L. (2018) · Science as Culture · doi:10.1080/09505431.2018.1485640 · Cited by 35
“…suppresses ‘the original thinking required to make fundamental discoveries’ (Alberts et al., 2014, p. 5774),may result in a rise in the number of published false positive results (Szucs and Ioannidis, 2017), and could increase the likelihood of misconduct in research practices (van Wesel, 2016).”
BackgroundResting Frontal Eeg Asymmetry and Personality Traits: A Meta–Analysis
Kuper, N., Käckenmester, W., & Wacker, J. (2019) · European Journal of Personality · doi:10.1002/per.2197 · Cited by 51
“This finding might be in line with the recent observation that higher impact journals tend to publish studies with less statistical power (Szucs & Ioannidis, 2017; cf. Fanelli, Costas, & Ioannidis, 2017 for a criticism of impact factor as an indicator of bias).”
ResultHong, Y.-W., Yoo, Y., Han, J.-H., Wager, T. D., & Woo, C. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.03.070 · Cited by 50
Preregistration of Analyses of Preexisting Data
Mertens, G., & Krypotos, A.-M. (2019) · Psychologica Belgica · doi:10.5334/pb.493 · Cited by 48
Cross-Temporal Meta-Analysis: A Conceptual and Empirical Critique
Rudolph, C. W., Costanza, D. P., Wright, C., & Zacher, H. (2019) · Journal of Business and Psychology · doi:10.1007/s10869-019-09659-2 · Cited by 53
P > .05: The incorrect interpretation of “not significant” results is a significant problem
Smith, R. J. (2020) · American Journal of Physical Anthropology · doi:10.1002/ajpa.24092 · Cited by 57
Trübutschek, D., Yang, Y., Gianelli, C., Čėsnaitė, E., Fischer, N. L., Vinding, M. C., Marshall, T. R., Algermissen, J., Pascarella, A., Puoliväli, T., et al. (2023) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02087 · Cited by 47
“Low replicability and reproducibility of research findings have eroded trust in even some of the most established results (e.g., Szucs & Ioannidis, 2017; Open Science Collaboration, 2015).”
ResultThe Prior Odds of Testing a True Effect in Cognitive and Social Psychology
Wilson, B. M., & Wixted, J. T. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918767122 · Cited by 47
“Maximum likelihood fit of the ex-Gaussian distribution to the significant effect sizes taken from Szucs and Ioannidis’s (2017) data set.”
MethodEffect size and statistical power in the rodent fear conditioning literature – A systematic review
Carneiro, C. F. D., Moulin, T. C., Macleod, M. R., & Amaral, O. B. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196258 · Cited by 54
“7% of statistically significant ones) fell under the “large” category by Cohen’s original proposal for psychology studies (19), which has been adopted by other authors as well (15,21).”
BackgroundMethodToffalini, E., Giofrè, D., Pastore, M., Carretti, B., Fraccadori, F., & Szűcs, D. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01549-x · Cited by 53
“It is in line with studies generally published in cognitive psychology and only slightly larger than the average sample size in neuroimaging studies (Szucs & Ioannidis, 2017, 2020).”
BackgroundResultOn gambling research, social science and the consequences of commercial gambling
Livingstone, C. H., Adams, P. J., Cassidy, R., Markham, F., Reith, G., Rintoul, A., Schüll, N. D., Woolley, R., & Young, M. (2017) · International Gambling Studies · doi:10.1080/14459795.2017.1377748 · Cited by 46
“As Szucs and Ioannidis (2017) have shown, the false report probability of cognitive neuroscience and of psychology papers published in toplevel journals is over 50% (Szucs & Ioannidis, 2017).”
BackgroundPuoliväli, T., Palva, S., & Palva, J. M. (2020) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2020.108654 · Cited by 53
Research Integrity definitions and challenges
Armond, A. C. V., Cobey, K. D., & Moher, D. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111367 · Cited by 55
The case for preregistering all region of interest (ROI) analyses in neuroimaging research
Gentili, C., Cecchetti, L., Handjaras, G., Lettieri, G., & Cristea, I. A. (2020) · European Journal of Neuroscience · doi:10.1111/ejn.14954 · Cited by 47
“…sizes and resulting small statistical power to detect effects that are not large (Button et al., 2013), combined with selective reporting biases (Szucs & Ioannidis, 2017) and inadequate analytic choices (Eklund, Nichols, & Knutsson, 2016). fMRI studies often rely on whole-brain…”
BackgroundBrini, S., Brudasca, N. I., Hodkinson, A., Kaluzinska, K., Wach, A., Storman, D., Prokop-Dorner, A., Jemioło, P., & Bala, M. M. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102236 · Cited by 44
Kong, P.-R., & Han, K.-T. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.172197 · Cited by 51
Carbine, K. A., & Larson, M. J. (2019) · Health Psychology Review · doi:10.1080/17437199.2019.1622144 · Cited by 45
Task-Based Functional Connectivity in Attention-Deficit/Hyperactivity Disorder: A Systematic Review
Kowalczyk, O. S., Mehta, M. A., O’Daly, O. G., & Criaud, M. (2022) · Biological Psychiatry Global Open Science · doi:10.1016/j.bpsgos.2021.10.006 · Cited by 41
Mind-wandering: mechanistic insights from lesion, tDCS, and iEEG
Kam, J. W. Y., Mittner, M., & Knight, R. T. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.12.005 · Cited by 47
Ehring, T. W., Limburg, K., Kunze, A. E., Wittekind, C. E., Werner, G. G., Wolkenstein, L., Guzey, M., & Cludius, B. (2022) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102163 · Cited by 49
The mediating effect of language on the development of cognitive and affective theory of mind
Bigelow, F. J., Clark, G. M., Lum, J. A. G., & Enticott, P. G. (2021) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2021.105158 · Cited by 46
Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis
Nuijten, M. B., van Assen, M. A. L. M., Augusteijn, H. E. M., Crompvoets, E. A. V., & Wicherts, J. M. (2020) · Journal of Intelligence · doi:10.3390/jintelligence8040036 · Cited by 45
LISA improves statistical analysis for fMRI
Lohmann, G., Stelzer, J., Lacosse, E., Kumar, V., Mueller, K., Kuehn, E., Grodd, W., & Scheffler, K. (2018) · Nature Communications · doi:10.1038/s41467-018-06304-z · Cited by 49
From E-Commerce to the Metaverse: A Neuroscientific Analysis of Digital Consumer Behavior
Fici, A., Bilucaglia, M., Casiraghi, C., Rossi, C., Chiarelli, S., Columbano, M., Micheletto, V., Zito, M., & Russo, V. (2024) · Behavioral Sciences · doi:10.3390/bs14070596 · Cited by 44
Reliable novelty: New should not trump true
Brembs, B. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000117 · Cited by 44
Zhang, Y., Padmanabhan, A., Gross, J. J., & Menon, V. (2019) · Journal of Neuroscience · doi:10.1523/jneurosci.0220-19.2019 · Cited by 43
“, 2018), high false-positive rates (Eklund et al., 2016; Szucs and Ioannidis, 2017), and lack of replicability in neuroimaging research (Ioannidis, 2005; Eklund et al.”
BackgroundMethodInfluentialTandon, N., & Tandon, R. (2019) · Schizophrenia Research · doi:10.1016/j.schres.2019.08.032 · Cited by 41
Transparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 46
“In addition, overinterpretation and misuse of inferential statistics can occur in cases of low statistical power (Button et al., 2013; Szucs & Ioannidis, 2017), lenient and arbitrary thresholds for statistical significance (Benjamin et al., 2018; Lakens et al., 2018), errors in reporting p-values…”
BackgroundBeltz, A. M. (2022) · Frontiers in Neuroendocrinology · doi:10.1016/j.yfrne.2022.101037 · Cited by 41
Social cognitive network neuroscience
Krendl, A. C., & Betzel, R. F. (2022) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsac020 · Cited by 46
“Second, it provides potential resources by which to minimize concerns about power and sample sizes in neuroscience research (e.g. Button et al. , 2013; David et al. , 2013; Szucs and Ioannidis, 2017; Clayson et al. , 2019).”
Spiegelhalter, D. J. (2017) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12302 · Cited by 49
Targeting Next Generations to Change the Common Practice of Underpowered Research
Crutzen, R., & Peters, G.-J. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01184 · Cited by 39
“Underpowered studies remain ubiquitous (Maxwell, 2004; Bakker et al., 2012; Button et al., 2013; Turner et al., 2013; Szucs and Ioannidis, 2017) despite strong pleas to change this practice (Cohen, 1988, 1990, 1992).”
BackgroundReliability and validity of machine vision for the assessment of facial expressions
Beringer, M., Spohn, F., Hildebrandt, A., Wacker, J., & Recio, G. (2019) · Cognitive Systems Research · doi:10.1016/j.cogsys.2019.03.009 · Cited by 44
“Bad smells” in software analytics papers
Menzies, T., & Shepperd, M. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.04.005 · Cited by 37
Bakker, M., Veldkamp, C. L. S., van den Akker, O. R., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236079 · Cited by 45
“[22] also showed that researchers highly overestimate the power of studies when effect sizes are small to medium-sized, which are quite common in psychology [2, 3, 23].”
BackgroundIsolating the sources of pipeline‐variability in group‐level task‐fMRI results
Bowring, A., Nichols, T. E., & Maumet, C. (2021) · Human Brain Mapping · doi:10.1002/hbm.25713 · Cited by 43
“…practices – where only methods that return a favourable outcome are likely to end up being published – the consequences of this can be severe, leading to fMRI e ↵ ect sizes that are misrepresented and often overstated in the neuroimaging literature (Simmons et al., 2011, Szucs and Ioannidis, 2017).”
BackgroundBarroca, N. C. B., Santa, G. D., Suchecki, D., Garcia‐Cairasco, N., & Umeoka, E. H. D. L. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104771 · Cited by 41
Ethical and Legal Implications of the Methodological Crisis in Neuroimaging
Kellmeyer, P. (2017) · Cambridge Quarterly of Healthcare Ethics · doi:10.1017/s096318011700007x · Cited by 42
Mixed-effects design analysis for experimental phonetics
Kirby, J., & Sonderegger, M. (2018) · Journal of Phonetics · doi:10.1016/j.wocn.2018.05.005 · Cited by 42
Survey-Software Implicit Association Tests: A Methodological and Empirical AnalysisPreprint
Carpenter, T., Pogacar, R., Pullig, C., Kouril, M., Aguilar, S. J., LaBouff, J. P., Isenberg, N., & Chakroff, A. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/hgy3z · Cited by 47
Identification of Two Distinct Working Memory-Related Brain Networks in Healthy Young Adults
Egli, T., Coynel, D., Spalek, K., Fastenrath, M., Freytag, V., Heck, A., Loos, E., Auschra, B., Papassotiropoulos, A., de Quervain, D. J. ‐., et al. (2018) · eNeuro · doi:10.1523/eneuro.0222-17.2018 · Cited by 42
“…with a large number of statistical tests is a prevalent source of critique regarding the existing neuroimaging literature (Poldrack et al., 2017; Szucs and Ioannidis, 2017), especially in combination with multiple high-dimensional datasets such as imaging genetic studies (Bigos and Weinberger,…”
BackgroundOlsen, A., Babikian, T., Bigler, E. D., Caeyenberghs, K., Conde, V., Dams-O’Connor, K., Dobryakova, E., Genova, H. M., Grafman, J. H., Haberg, A. K., et al. (2020) · Brain Imaging and Behavior · doi:10.1007/s11682-020-00313-7 · Cited by 38
Dębska, A., Łuniewska, M., Zubek, J., Chyl, K., Dynak, A., Dzięgiel‐Fivet, G., Plewko, J., Jednoróg, K., & Grabowska, A. (2021) · Developmental Science · doi:10.1111/desc.13173 · Cited by 39
“The small samples not only limited the statistical power of the between-group (Sassenberg & Ditrich, 2019; Szucs & Ioannidis, 2017) but particularly affected the method of finding deficits.”
BackgroundTadros, M., Newby, J. M., Li, S., & Werner‐Seidler, A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0317125 · Cited by 41
Statistical power: Implications for planning MEG studies
Chaumon, M., Puce, A., & George, N. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117894 · Cited by 38
Power, S. A., & Vélez, G. (2020) · Review of General Psychology · doi:10.1177/1089268020915841 · Cited by 39
“…and analyses) and HARKing (hypothesizing after the results are known; see Nelson et al., 2018)—some social scientists have argued for lowering the acceptable p-value threshold or refocusing on adequate statistical power (Benjamin et al., 2018; Szucs & Ioannidis, 2017; Muthukrishna & Henrich, 2019).”
BackgroundAcetyl-L-carnitine for patients with hepatic encephalopathy
Martí-Carvajal, A. J., Gluud, C. N., Arévalo-Rodríguez, I., & Martí-Amarista, C. E. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011451.pub2 · Cited by 43
“…interpret all estimates of treatment eLect reported in this review, possibly with the exception of those related to blood ammonium levels, as potentially false positive (inflated or overestimated beneficial eLects) or as potentially false negative (underestimated harms) (Gaskin 2014; Szucs 2017).”
BackgroundFaculty appointment and promotion in Taiwan’s medical schools, a systematic analysis
Yeh, J., Shulruf, B., Lee, H., Huang, P., Kuo, W.-H., Hwang, T.-C., & Chen, C. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03435-2 · Cited by 35
An N-pact factor for clinical psychological research.
Reardon, K. W., Smack, A. J., Herzhoff, K., & Tackett, J. L. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000435 · Cited by 35
Metadata
- Authors
- Szucs, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-03-02
- Volume
- 15
- Issue
- 3
- Pages
- e2000797
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5333800
- OpenAlex
- W2951081425
- Cited by
- 761
- References
- 62
- Citations by year
- 2026: 28 · 2025: 43 · 2024: 120 · 2023: 75 · 2022: 80 · 2021: 76 · 2020: 94 · 2019: 102 · 2018: 91 · 2017: 46
- FWCI
- 81.43
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-03-05)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo (cc-by)
- Funders
- James S. McDonnell Foundation (220020370)
- Universiteit van Tilburg
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- effect size, statistical power, cognitive neuroscience, psychology literature, sample size, replication success, journal impact factor, null hypothesis, publication bias, statistical records
- MeSH terms
- Cognitive Neuroscience, Humans, Probability, Psychology, Publications, Surveys and Questionnaires, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.