Why Most Discovered True Associations Are Inflated
The DOI matches its Crossref record; 111 of the 111 references it cites were found in a registry or an index.
Abstract
Newly discovered true (non-null) associations often have inflated effects compared with the true effect sizes. I discuss here the main reasons for this inflation. First, theoretical considerations prove that when true discovery is claimed based on crossing a threshold of statistical significance and the discovery study is underpowered, the observed effects are expected to be inflated. This has been demonstrated in various fields ranging from early stopped clinical trials to genome-wide associations. Second, flexible analyses coupled with selective reporting may inflate the published discovered effects. The vibration ratio (the ratio of the largest vs. smallest effect on the same association approached with different analytic choices) can be very large. Third, effects may be inflated at the stage of interpretation due to diverse conflicts of interest. Discovered effects are not always inflated, and under some circumstances may be deflated-for example, in the setting of late discovery of associations in sequentially accumulated overpowered evidence, in some types of misclassification from measurement error, and in conflicts causing reverse biases. Finally, I discuss potential approaches to this problem. These include being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery ph…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Given that initial effect sizes are often inflated 61 and that we sought power to detect a difference between two groups, we doubled this estimate and aimed to test ~28 participants in each group with some attrition expected.”
From Adults with autism overestimate the volatility of the sensory environment (2017) · cites this paper as [61]
In the citing paper
“While this may reflect exaggeration of original effect sizes or publication bias (Ioannidis 2008), some degree of inflation is expected even in its absence, as our selection process filtered for original experiments with significant differences.”
From Estimating the replicability of Brazilian biomedical science (2025) · cites this paper as Ioannidis 2008
In the citing paper
“Further, in situations when an underpowered study detects a true effect, the estimate of this effect size is likely to be highly exaggerated, a phenomenon often referred to as the “winner’s curse” (15).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [15]
In the citing paper
“Researchers have strong incentives to interpret their results as supporting or extending the Bayesian framework, rather than challenging it—leading to a literature that systematically over-represents supportive evidence (Ioannidis 2008; Nosek et al. 2012).”
From The myth of the Bayesian brain (2025) · cites this paper as Ioannidis 2008
In the citing paper
“This flexibility in the synthesis of the evidence provides the assessors with the opportunity to inflate or disinflate the efficacy of a given technology, a phenomenon that was observed in scientific publications.[24]”
From Health technology assessments conducted in health care facilities: A strategic practice? Findings from a content analysis of HTA reports (2017) · cites this paper as [24]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationGene expression profiling predicts clinical outcome of breast cancer
van 't Veer, L. J., Dai, H., van de Vijver, M. J., He, Y. D., Hart, A. A. M., Mao, M., Peterse, H. L., van der Kooy, K., Marton, M. J., Witteveen, A. T., et al. (2002) · Nature · doi:10.1038/415530a · Cited by 9,536
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“This article will not deal with the debate on whether research findings are credible.(1) I will focus instead on the interesting subset of research findings that are true.”
BackgroundSelf-citationEffect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Nissen, S. E., & Wolski, K. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072761 · Cited by 4,675
The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2007) · Epidemiology · doi:10.1097/ede.0b013e3181577511 · Cited by 2,779
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544
Cited by
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
“and trials of lower quality also tend to show larger intervention effects.”
BackgroundSelf-citationPower failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationThe ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
“This poses a high risk of overoptimistic expectations of the marker’s performance.(21,22) Moreover, we are interested in only the incremental value of a marker, on top of predictors that are readily accessible.”
BackgroundMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationFlegal, K. M., Kit, B. K., Orpana, H., & Graubard, B. I. (2013) · JAMA · doi:10.1001/jama.2012.113905 · Cited by 3,998
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,916
“It simply reflects the fact that small samples necessarily produce more variable estimates than large samples (Gelman & Weakliem, 2009; Ioannidis, 2008; Yarkoni, 2009), and that, given the strong prevailing bias in favor of positive, large, or otherwise “interesting” effects (Nosek, Spies, & Motyl, 2012), it is the la…”
BackgroundInfluentialHow Big Is “Big”? Interpreting Effect Sizes in L2 Research
Plonsky, L., & Oswald, F. L. (2014) · Language Learning · doi:10.1111/lang.12079 · Cited by 1,960
“Such experiments would tend to yield larger effect sizes (Ioannidis, 2008; Kline, 2004).”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
“As a result, published evidence is unrepresentative of reality (Ioannidis, 2008; Pashler & Harris, 2012).”
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Forstmeier, W., & Schielzeth, H. (2011) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1038-5 · Cited by 1,120
“This is another aspect of the winner's curse that even if the claimed effects are true, effect size estimates based on selection from a larger set of candidate effects might give misleadingly high effect sizes (Göring et al. 2001; Ioannidis 2008).”
InfluentialReporting Recommendations for Tumor Marker Prognostic Studies (REMARK): Explanation and Elaboration
Altman, D. G., McShane, L. M., Sauerbrei, W., & Taube, S. E. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001216 · Cited by 1,013
“Selective reporting of studies, or selective reporting of only some analyses within studies, both lead to larger effects being seen in smaller studies, and literature that is biased towards overestimating the prognostic importance of tumor markers [204].”
ResultUsing the Student's t-test with extremely small sample sizes
de Winter, J. C. F. (2013) · University of Massachusetts Amherst · doi:10.7275/e4r6-dj05 · Cited by 992
Kapur, S., Phillips, A. G., & Insel, T. R. (2012) · Molecular Psychiatry · doi:10.1038/mp.2012.105 · Cited by 980
Westfall, J., Kenny, D. A., & Judd, C. M. (2014) · Journal of Experimental Psychology: General · doi:10.1037/xge0000014 · Cited by 928
“…salient in light of the growing awareness in psychology and other experimental sciences of the problems created and maintained by a proliferation of chronically underpowered studies (Asendorpf et al., 2013; Bakker, van Dijk, & Wicherts, 2012; Button et al., 2013; Ioannidis, 2008; Schimmack, 2012).”
BackgroundCross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897
“This is sometimes known as vibration, and the key quantity is the ratio between the effect size and the variations due to analytical choices (Ioannidis, 2008).”
BackgroundType 2 diabetes and cancer: umbrella review of meta-analyses of observational studies
Tsilidis, K. K., Kasimis, J. C., Lopez, D. S., Ntzani, E. E., & Ioannidis, J. P. A. (2015) · BMJ · doi:10.1136/bmj.g7607 · Cited by 860
Self-citationEffect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology
Brydges, C. R. (2019) · Innovation in Aging · doi:10.1093/geroni/igz036 · Cited by 826
“In addition, underpowered studies are more likely to report an overly inflated effect size (Ioannidis, 2008) through questionable research practices such as p-hacking (Simmons, Nelson, & Simonsohn, 2011), which can result in a greater likelihood of failed attempts to replicate the finding (Maxwell, 2004).”
BackgroundInfluentialEthical Standards in Sport and Exercise Science Research: 2020 Update
Harriss, D. J., MacSween, A., & Atkinson, G. (2019) · International Journal of Sports Medicine · doi:10.1055/a-1015-3123 · Cited by 735
“Nevertheless, it is important for researchers to appreciate the possibility that an unusually large sample effect size was observed through the sampling error associated with a small sample size, sometimes referred to as “The Winners Curse” [25].”
Adiposity and cancer at major anatomical sites: umbrella review of the literature
Kyrgiou, M., Kalliala, I., Markozannes, G., Gunter, M. J., Paraskevaidis, E., Gabra, H., Martin-Hirsch, P., & Tsilidis, K. K. (2017) · BMJ · doi:10.1136/bmj.j477 · Cited by 803
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Szucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761
Self-citationHow to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
“Many new proposed associations and/or effects are false or grossly exaggerated [2,3], and translation of knowledge into useful applications is often slow and potentially inefficient [4].”
BackgroundSelf-citationWicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
“Second, their strategic use in research may inflate effect sizes (Ioannidis, 2008; Bakker et al., 2012; Simonsohn et al., 2014; van Aert et al., 2016).”
BackgroundSmall Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742
“In addition, because of publication bias, most true effects are overestimated in the published literature (Hedges, 1984; Ioannidis, 2008; Lane & Dunlap, 1978), so most sufficiently powered replication studies of true effects will obtain statistically significantly smaller estimates than the originals.”
Common Regulatory Variation Impacts Gene Expression in a Cell Type–Dependent Manner
Dimas, A. S., Deutsch, S., Stranger, B. E., Montgomery, S. B., Borel, C., Attar-Cohen, H., Ingle, C., Beazley, C., Arcelus, M. G., Sekowska, M., et al. (2009) · Science · doi:10.1126/science.1174148 · Cited by 745
p -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717
“First, it shows the dramatic inflation of effect size generated by publication bias (Hedges, 1984; Ioannidis, 2008; Lane & Dunlap, 1978).”
BackgroundIoannidis, J. P. A. (2009) · Canadian Medical Association Journal · doi:10.1503/cmaj.081086 · Cited by 716
Self-citationYarkoni, T. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20001685 · Cited by 701
Donoho, D. (2017) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2017.1384734 · Cited by 695
Risk factors for endometrial cancer: An umbrella review of the literature
Raglan, O., Kalliala, I., Markozannes, G., Cividini, S., Gunter, M. J., Nautiyal, J., Gabra, H., Paraskevaidis, E., Martin‐Hirsch, P., Tsilidis, K. K., et al. (2019) · International Journal of Cancer · doi:10.1002/ijc.31961 · Cited by 648
Per-Protocol Analyses of Pragmatic Trials
Hernán, M. A., & Robins, J. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmsm1605385 · Cited by 660
Personality in 100,000 Words: A large-scale analysis of personality and word use among bloggers
Yarkoni, T. (2010) · Journal of Research in Personality · doi:10.1016/j.jrp.2010.04.001 · Cited by 637
“These seemingly paradoxical results are easily reconciled if one supposes that the true effects under investigation were actually relatively modest in both studies, but that Hirsh and Peterson's (2009) results, which stemmed from a much smaller sample (N = 94), were more susceptible to sampling error, and hence, effec…”
BackgroundResultStatistical power and significance testing in large-scale genetic studies
Sham, P. C., & Purcell, S. M. (2014) · Nature Reviews Genetics · doi:10.1038/nrg3706 · Cited by 573
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563
“…simply conceptual replications (Makel et al., 2012; Nosek et al., 2012), which probably suffer from confirmation bias (Wagenmakers et al., 2012) in which authors try to get the result they want (the “fine-tune factor,” or what I have previously described as “vibration of effects” (Ioannidis, 2008).”
BackgroundInfluentialSelf-citationEfficacy of cognitive bias modification interventions in anxiety and depression: Meta-analysis
Cristea, I. A., Kok, R. N., & Cuijpers, P. (2015) · British Journal of Psychiatry · doi:10.1192/bjp.bp.114.146761 · Cited by 553
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551
Self-citationWhy Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al-Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 502
“An important role of systematic reviews may be to correct the inflated effects present in the initial studies published in famous journals [12], but this process may be similarly prolonged and even systematic reviews may perpetuate inflated results [13,14].”
Self-citationPersonalized evidence based medicine: predictive approaches to heterogeneous treatment effects
Kent, D. M., Steyerberg, E., & van Klaveren, D. (2018) · BMJ · doi:10.1136/bmj.k4245 · Cited by 479
Sauerbrei, W., Taube, S. E., McShane, L. M., Cavenagh, M. M., & Altman, D. G. (2018) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djy088 · Cited by 486
Meta-analysis of data from animal studies: A practical guide
Vesterinen, H. M., Sena, E. S., Egan, K. J., Hirst, T. C., Churolov, L., Currie, G. L., Antonic, A., Howells, D. W., & Macleod, M. R. (2014) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2013.09.010 · Cited by 488
Web-Based, Participant-Driven Studies Yield Novel Genetic Associations for Common Traits
Eriksson, N., Macpherson, J. M., Tung, J. Y., Hon, L. S., Naughton, B., Saxonov, S., Avey, L., Wojcicki, A., Pe'er, I., & Mountain, J. (2010) · PLoS Genetics · doi:10.1371/journal.pgen.1000993 · Cited by 479
“Such a bias is a concern [46].”
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Self-citationAdults with autism overestimate the volatility of the sensory environment
Lawson, R. P., Mathys, C., & Rees, G. (2017) · Nature Neuroscience · doi:10.1038/nn.4615 · Cited by 470
The Predictive Approaches to Treatment effect Heterogeneity (PATH) statement
Kent, D. M., Paulus, J. K., van Klaveren, D., D'Agostino, R., Goodman, S., Hayward, R., Ioannidis, J. P. A., Patrick-Lake, B., Morton, S., Pencina, M., et al. (2020) · Annals of Internal Medicine · doi:10.7326/m18-3667 · Cited by 469
Self-citationPresidential Address: The Scientific Outlook in Financial Economics
HARVEY, C. R. (2017) · The Journal of Finance · doi:10.1111/jofi.12530 · Cited by 461
Color and psychological functioning: a review of theoretical and empirical work
Elliot, A. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00368 · Cited by 374
“Underpowered samples commonly produce overestimated effect size estimates (Ioannidis, 2008), and basing subsequent sample sizes on such estimates simply perpetuates the problem.”
BackgroundAnatomy of health effects of Mediterranean diet: Greek EPIC prospective cohort study
Trichopoulou, A., Bamia, C., & Trichopoulos, D. (2009) · BMJ · doi:10.1136/bmj.b2337 · Cited by 436
“although other methodological reasons could also be invoked.”
Szucs, D., & Ioannidis, J. P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117164 · Cited by 453
Self-citationMarcos-Zambrano, L. J., Karaduzovic-Hadziabdic, K., Loncar Turukalo, T., Przymus, P., Trajkovik, V., Aasmets, O., Berland, M., Gruca, A., Hasic, J., Hron, K., et al. (2021) · Frontiers in Microbiology · doi:10.3389/fmicb.2021.634511 · Cited by 416
“This estimate is on average, overestimated because of the “winner’s curse” (Ioannidis, 2008).”
BackgroundMaking big data open: data sharing in neuroimaging
Poldrack, R. A., & Gorgolewski, K. J. (2014) · Nature Neuroscience · doi:10.1038/nn.3818 · Cited by 425
Evaluation of a decided sample size in machine learning applications
Rajput, D., Wang, W.-J., & Chen, C.-C. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05156-9 · Cited by 411
Abundant contribution of short tandem repeats to gene expression variation in humans
Gymrek, M., Willems, T., Guilmatre, A., Zeng, H., Markus, B., Georgiev, S., Daly, M. J., Price, A. L., Pritchard, J. K., Sharp, A. J., et al. (2016) · Nature Genetics · doi:10.1038/ng.3461 · Cited by 410
Kim, M. S., Kim, W. J., Khera, A. V., Kim, J. Y., Yon, D. K., Lee, S. W., Shin, J. I., & Won, H.-H. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab454 · Cited by 378
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
A Meta-Analytic Review of Stand-Alone Interventions to Improve Body Image
Alleva, J. M., Sheeran, P., Webb, T. L., Martijn, C., & Miles, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0139177 · Cited by 319
“Coyne, Thombs, and Hagedoorn [54] recently critiqued interventions in the field of behavioural medicine for overrelying on small, underpowered trials [55,56].”
BackgroundOn doing relevant and rigorous experiments: Review and recommendations
Lonati, S., Quiroga, B. F., Zehnder, C., & Antonakis, J. (2018) · Journal of Operations Management · doi:10.1016/j.jom.2018.10.003 · Cited by 354
“at the research field level, small sample sizes tend to inflate the effect size of the treatment, given that only significant results are usually published (Antonakis, 2017; Button et al., 2013; Ioannidis, 2008).”
BackgroundIllusions in regression analysis
Armstrong, J. S. (2012) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2012.02.001 · Cited by 322
Validating, augmenting and refining genome-wide association signals
Ioannidis, J. P. A., Thomas, G., & Daly, M. J. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2544 · Cited by 375
Self-citationReporting recommendations for tumor marker prognostic studies (REMARK): explanation and elaboration
Altman, D. G., McShane, L. M., Sauerbrei, W., & Taube, S. E. (2012) · BMC Medicine · doi:10.1186/1741-7015-10-51 · Cited by 372
“Selective reporting of studies, or selective reporting of only some analyses within studies, both lead to larger effects being seen in smaller studies, and literature that is biased towards overestimating the prognostic importance of tumor markers [204].”
ResultFour reasons to prefer Bayesian analyses over significance testing
Dienes, Z., & Mclatchie, N. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1266-z · Cited by 354
Låg, T., & Sæle, R. G. (2019) · AERA Open · doi:10.1177/2332858419870489 · Cited by 265
“Furthermore, there is a risk that small, underpowered studies overestimate any effects reported, partly because statistical tests in such studies will reach significance only if the sample mean difference happens to be very large (Button et al., 2013; Ioannidis, 2008; Sterne, Gavaghan, & Egger, 2000).”
BackgroundPublication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215052 · Cited by 334
Kell, D. B. (2010) · Archives of Toxicology · doi:10.1007/s00204-010-0577-x · Cited by 361
“Note, however, that such methods need careful handling from a statistical point of view (e.g. Altman and Royston 2000; Broadhurst and Kell 2006; Hastie et al. 2001; Ioannidis 2005, 2008).”
BackgroundDoes Self-Control Training Improve Self-Control? A Meta-Analysis
Friese, M., Frankenbach, J., Job, V., & Loschelder, D. D. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617697076 · Cited by 334
“If this happens, the published literature is not representative of the full body of research and overestimates the population effect size (Ioannidis, 2008).”
BackgroundLarge Cross-National Differences in Gene × Socioeconomic Status Interaction on Intelligence
Tucker-Drob, E. M., & Bates, T. C. (2016) · Psychological Science · doi:10.1177/0956797615612727 · Cited by 359
The relation between statistical power and inference in fMRI
Cremers, H. R., Wager, T. D., & Yarkoni, T. (2017) · PLOS ONE · doi:10.1371/journal.pone.0184923 · Cited by 349
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 347
“Further, in situations when an underpowered study detects a true effect, the estimate of this effect size is likely to be highly exaggerated, a phenomenon often referred to as the “winner’s curse” (15).”
BackgroundRecommendations for Biomarker Identification and Qualification in Clinical Proteomics
Mischak, H., Allmaier, G., Apweiler, R., Attwood, T., Baumann, M., Benigni, A., Bennett, S. E., Bischoff, R., Bongcam-Rudloff, E., Capasso, G., et al. (2010) · Science Translational Medicine · doi:10.1126/scitranslmed.3001249 · Cited by 334
“Generally, the strength of associations is expected to be inflated in the original, discovery analysis (37).”
BackgroundSelf-citationScherag, A., Dina, C., Hinney, A., Vatin, V., Scherag, S., Vogel, C. I. G., Müller, T. D., Grallert, H., Wichmann, H.-E., Balkau, B., et al. (2010) · PLoS Genetics · doi:10.1371/journal.pgen.1000916 · Cited by 327
“To address, to some extent, problems of the ‘bias-variance trade-off’ and the ‘winners curse’ [34], the largest GENERALIZATION population-based sample KORA (n = 12,002) was chosen for this modelling.”
MethodThe earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
“As wrote Ioannidis (2008), "at the time of first postulated discovery, we usually cannot tell whether an association exists at all, let alone judge its effect size."”
BackgroundCulverhouse, R. C., Saccone, N. L., Horton, A. C., Ma, Y., Anstey, K. J., Banaschewski, T., Burmeister, M., Cohen-Woods, S., Etain, B., Fisher, H. L., et al. (2018) · Molecular Psychiatry · doi:10.1038/mp.2017.44 · Cited by 321
“In addition, small studies are subject to multiple statistical issues, including publication bias (exacerbated for small studies) and the winner’s curse (which makes it likely, even if a true effect is detected, that the magnitude will be exaggerated).(70) In fact, a 2013 analysis of neuroscience publications conclude…”
BackgroundThe False-positive to False-negative Ratio in Epidemiologic Studies
Ioannidis, J. P. A., Tarone, R., & McLaughlin, J. K. (2011) · Epidemiology · doi:10.1097/ede.0b013e31821b506e · Cited by 321
Self-citationSmall studies are more heterogeneous than large ones: a meta-meta-analysis
IntHout, J., Ioannidis, J. P. A., Borm, G. F., & Goeman, J. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.017 · Cited by 314
Self-citationBaker, D. H., Vilidaitė, G., Lygo-Frett, F., Smith, A. K., Flack, T. R., Gouws, A. D., & Andrews, T. J. (2021) · Psychological Methods · doi:10.1037/met0000337 · Cited by 347
“Furthermore, any real effects that are detected are likely to have inflated effect sizes (Colquhoun, 2014; Ioannidis, 2008).”
Attention bias modification for social anxiety: A systematic review and meta-analysis
Heeren, A., Mogoașe, C., Philippot, P., & McNally, R. J. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.06.001 · Cited by 312
“Substandard quality is often associated with artificial inflation of ESs (Ioannidis, 2008).”
BackgroundThe Role of Genetics in the Etiology of Schizophrenia
Gejman, P. V., Sanders, A. R., & Duan, J. (2010) · Psychiatric Clinics of North America · doi:10.1016/j.psc.2009.12.003 · Cited by 296
“Other potential causes of false-positives are multiple analyses and selective reporting.(86) It is possible that genetic heterogeneity in some specific cases precludes a replication, but it seems unlikely that this is a robust general argument.”
Elliott, M. L., Knodt, A. R., Cooke, M., Kim, M. J., Melzer, T. R., Keenan, R., Ireland, D., Ramrakha, S., Poulton, R., Caspi, A., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.01.068 · Cited by 304
A Hitchhiker's Guide to Functional Magnetic Resonance Imaging
Soares, J. M., Magalhães, R., Moreira, P. S., Sousa, A., Ganz, E., Sampaio, A., Alves, V., Marques, P., & Sousa, N. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00515 · Cited by 272
“(DPABI; Yan et al., 2016) http://rfmri.org/dpabi Preprocessing, analysis, and statistical analysis”
Replication in Genome-Wide Association Studies
Kraft, P., Zeggini, E., & Ioannidis, J. P. A. (2009) · Statistical Science · doi:10.1214/09-sts290 · Cited by 290
“Signals selected based on statistical significance thresholds in underpowered settings are likely to have (on average) inflated effects due to the winner’s curse phenomenon [28, 69, 70, 75].”
BackgroundSelf-citationExcess Significance Bias in the Literature on Brain Volume Abnormalities
Ioannidis, J. P. A. (2011) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.28 · Cited by 299
Self-citationWhen Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
“Moreover, such published false positive true H0 studies will also inevitably overestimate the effect size of the non-existent effects or of existent, but unimportantly tiny, effects (Schmidt, 1992; 1996; Sterling et al. 1995; Ioannidis 2008).”
BackgroundSelf-citationA Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295
“Studies with smaller, non-significant effects are rarely published, driving up the average effect size (Ioannidis, 2008).”
Evaluation of excess significance bias in animal studies of neurological diseases
Tsilidis, K. K., Panagiotou, O. A., Sena, E. S., Aretouli, E., Evangelou, E., Howells, D. W., Salman, R. A.-S., Macleod, M. R., & Ioannidis, J. P. A. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001609 · Cited by 295
“Second, selective analysis and outcome reporting biases may emerge when there are many analyses that can be performed, but only the analysis with the ‘‘best’’ results is presented resulting in potentially misleading findings [10].”
BackgroundSelf-citationMeta-Analysis in Genome-Wide Association Studies
Zeggini, E., & Ioannidis, J. P. (2009) · Pharmacogenomics · doi:10.2217/14622416.10.2.191 · Cited by 283
Self-citationWerling, D. M., Brand, H., An, J.-Y., Stone, M. R., Zhu, L., Glessner, J. T., Collins, R. L., Dong, S., Layer, R. M., Markenscoff-Papadimitriou, E., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0107-y · Cited by 293
Eirich, R., McArthur, B. A., Anhorn, C., McGuinness, C., Christakis, D. A., & Madigan, S. (2022) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2022.0155 · Cited by 293
Olsen, G. W., Butenhoff, J. L., & Zobel, L. R. (2009) · Reproductive Toxicology · doi:10.1016/j.reprotox.2009.02.001 · Cited by 287
Yarkoni, T. (2019) · doi:10.31234/osf.io/jqw35 · Cited by 244
Patel, C. J., Burford, B., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.029 · Cited by 274
“Consequently, there is large heterogeneity in how investigators associate variables [2].”
Self-citationBiases in Randomized Trials: A Conversation Between Trialists and Epidemiologists
Mansournia, M. A., Higgins, J. P. T., Sterne, J. A. C., & Hernán, M. A. (2017) · Epidemiology · doi:10.1097/ede.0000000000000564 · Cited by 285
Power, precision, and sample size estimation in sport and exercise science research
Abt, G., Boreham, C., Davison, G., Jackson, R., Nevill, A., Wallace, E., & Williams, M. (2020) · Journal of Sports Sciences · doi:10.1080/02640414.2020.1776002 · Cited by 241
“As outlined by Brysbaert (2019) and others (Button et al., 2013; Ioannidis, 2005, 2008; Ioannidis et al., 2011) the problems with underpowered studies are numerous.”
BackgroundAttentional bias for threat: Crisis or opportunity?
McNally, R. J. (2019) · Clinical Psychology Review · doi:10.1016/j.cpr.2018.05.005 · Cited by 246
Kim, J. Y., Son, M. J., Son, C. Y., Radua, J., Eisenhut, M., Gressier, F., Koyanagi, A., Carvalho, A. F., Stubbs, B., Solmi, M., et al. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30181-6 · Cited by 271
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- Epidemiology
- Published
- 2008-09
- Volume
- 19
- Issue
- 5
- Pages
- 640-648
- ISSN
- 1044-3983 (ISSN-L)
- OpenAlex
- W2150215887
- Semantic Scholar
- 0373f18622b7b64e99c7e56340ab2011dbd00b5d
- Cited by
- 1,495
- References
- 109
- Citations by year
- 2026: 91 · 2025: 87 · 2024: 78 · 2023: 112 · 2022: 94 · 2021: 99 · 2020: 103 · 2019: 131 · 2018: 108 · 2017: 101 · 2016: 87 · 2015: 77 · 2014: 72 · 2013: 60 · 2012: 50
- FWCI
- 15.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- effect size inflation, statistical significance thresholds, underpowered studies, selective reporting, conflicts of interest, reverse bias, measurement error, replication
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Humans, Sensitivity and Specificity, Models, Statistical, Linkage Disequilibrium, Molecular Epidemiology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.