Why Most Discovered True Associations Are Inflated
The DOI matches its Crossref record; 111 of the 111 references it cites were found in a registry or an index.
Abstract
Newly discovered true (non-null) associations often have inflated effects compared with the true effect sizes. I discuss here the main reasons for this inflation. First, theoretical considerations prove that when true discovery is claimed based on crossing a threshold of statistical significance and the discovery study is underpowered, the observed effects are expected to be inflated. This has been demonstrated in various fields ranging from early stopped clinical trials to genome-wide associations. Second, flexible analyses coupled with selective reporting may inflate the published discovered effects. The vibration ratio (the ratio of the largest vs. smallest effect on the same association approached with different analytic choices) can be very large. Third, effects may be inflated at the stage of interpretation due to diverse conflicts of interest. Discovered effects are not always inflated, and under some circumstances may be deflated-for example, in the setting of late discovery of associations in sequentially accumulated overpowered evidence, in some types of misclassification from measurement error, and in conflicts causing reverse biases. Finally, I discuss potential approaches to this problem. These include being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery ph…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Given that initial effect sizes are often inflated 61 and that we sought power to detect a difference between two groups, we doubled this estimate and aimed to test ~28 participants in each group with some attrition expected.”
From Adults with autism overestimate the volatility of the sensory environment (2017) · cites this paper as [61]
In the citing paper
“While this may reflect exaggeration of original effect sizes or publication bias (Ioannidis 2008), some degree of inflation is expected even in its absence, as our selection process filtered for original experiments with significant differences.”
From Estimating the replicability of Brazilian biomedical science (2025) · cites this paper as Ioannidis 2008
In the citing paper
“Further, in situations when an underpowered study detects a true effect, the estimate of this effect size is likely to be highly exaggerated, a phenomenon often referred to as the “winner’s curse” (15).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [15]
In the citing paper
“Researchers have strong incentives to interpret their results as supporting or extending the Bayesian framework, rather than challenging it—leading to a literature that systematically over-represents supportive evidence (Ioannidis 2008; Nosek et al. 2012).”
From The myth of the Bayesian brain (2025) · cites this paper as Ioannidis 2008
In the citing paper
“This flexibility in the synthesis of the evidence provides the assessors with the opportunity to inflate or disinflate the efficacy of a given technology, a phenomenon that was observed in scientific publications.[24]”
From Health technology assessments conducted in health care facilities: A strategic practice? Findings from a content analysis of HTA reports (2017) · cites this paper as [24]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationGene expression profiling predicts clinical outcome of breast cancer
van 't Veer, L. J., Dai, H., van de Vijver, M. J., He, Y. D., Hart, A. A. M., Mao, M., Peterse, H. L., van der Kooy, K., Marton, M. J., Witteveen, A. T., et al. (2002) · Nature · doi:10.1038/415530a · Cited by 9,536
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“This article will not deal with the debate on whether research findings are credible.(1) I will focus instead on the interesting subset of research findings that are true.”
BackgroundSelf-citationEffect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Nissen, S. E., & Wolski, K. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072761 · Cited by 4,675
The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2007) · Epidemiology · doi:10.1097/ede.0b013e3181577511 · Cited by 2,779
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544
Cited by
Items 201–300 of 1,004
Does psychotherapy work? An umbrella review of meta‐analyses of randomized controlled trials
Dragioti, E., Karathanos, V., Gerdle, B., & Evangelou, E. (2017) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12713 · Cited by 111
“All new RCTs in psychotherapy should be correctly registered and their statistical methods and analysis should be clearly reported before the implementation of a study to eliminate selective analysis results and selective reporting biases (19, 24), especially when there is evidence for nonfinancial conflict of interes…”
BackgroundLong-Term Effects of Frequent In–Center Hemodialysis
Chertow, G. M., Levin, N. W., Beck, G. J., Daugirdas, J. T., Eggers, P. W., Kliger, A. S., Larive, B., Rocco, M. V., Greene, T., & for the Frequent Hemodialysis Network (FHN) Trials Group (2015) · Journal of the American Society of Nephrology · doi:10.1681/asn.2015040426 · Cited by 118
Siontis, K. C. M., Patsopoulos, N. A., & Ioannidis, J. P. A. (2010) · European Journal of Human Genetics · doi:10.1038/ejhg.2010.26 · Cited by 117
“Furthermore, one should caution that whenever associations are selected on the basis of statistical significance thresholds, effect sizes of the selected associations that pass the required threshold may be inflated compared with the true effect”
BackgroundSelf-citationPair-bonding, fatherhood, and the role of testosterone: a meta-analytic review
Grebe, N. M., Sarafin, R. E., Strenth, C. R., & Zilioli, S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.01.010 · Cited by 116
How to make more published research true
Ioannidis, J. P. A. (2015) · Acimed · Cited by 138
Self-citationHarris, J. D., Brand, J. C., Cote, M. P., & Dhawan, A. (2017) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2017.01.055 · Cited by 116
Refinement of Experimental Design and Conduct in Laboratory Animal Research
Bailoo, J. D., Reichlin, T. S., & Wurbel, H. (2014) · ILAR Journal · doi:10.1093/ilar/ilu037 · Cited by 107
van Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0084896 · Cited by 107
“The bias introduced by the contemporary scientific publishing system is quite severe [13,16,34,35] and it is clear that we as scientists need to focus more on getting it right than on getting it published [36].”
BackgroundRocco, M. V., Daugirdas, J. T., Greene, T., Lockridge, R. S., Chan, C. T., Pierratos, A., Lindsay, R. M., Larive, B., Chertow, G. M., Beck, G. J., et al. (2015) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2015.02.331 · Cited by 113
“US citizens: Medicare 29 25 (86) 25 22 (88) 24 22 (92) 23 20 (87) US citizens: Medicaid or state medical assistance 29 9 (31) 25 4 (16) 24 9 (38) 23 4 (17)”
Social Decision-Making and the Brain: A Comparative Perspective
Tremblay, S., Sharika, K. M., & Platt, M. L. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.01.007 · Cited by 105
Placing epidemiological results in the context of multiplicity and typical correlations of exposures
Patel, C. J., & Ioannidis, J. P. A. (2014) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2014-204195 · Cited by 117
Self-citationSerghiou, S., Kyriakopoulou, A., & Ioannidis, J. P. A. (2016) · Molecular Cancer · doi:10.1186/s12943-016-0535-1 · Cited by 115
“Small studies are known, both theoretically and empirically, to be associated with inflated estimates of effect size [58], not as much due to their limited sample size, as for lower quality standards, publication bias and selective reporting [59], which is why they lead to so-called ‘small-study effects’.”
BackgroundSelf-citationHerder, C., Gala, T. D. L. H., Carstensen‐Kirberg, M., Huth, C., Zierer, A., Wahl, S., Sudduth-Klinger, J., Kuulasmaa, K., Peretz, D., Ligthart, S., et al. (2017) · Arteriosclerosis Thrombosis and Vascular Biology · doi:10.1161/atvbaha.117.309307 · Cited by 122
Guidance on the assessment of the biological relevance of data in scientific assessments
Hardy, A., Benford, D., Halldorsson, T. I., Jeger, M. J., Knutsen, H. K., More, S. J., Naegeli, H., Noteborn, H. P. J. M., Ockleford, C., Ricci, A., et al. (2017) · EFSA Journal · doi:10.2903/j.efsa.2017.4970 · Cited by 119
“Underpowered studies may both fail to detect an effect with a relevant size and be more likely to inflate true effects or report spurious effects (Ioannidis, 2005, 2008; La Caze and Duffull, 2011).”
BackgroundNakaoka, H., & Inoue, I. (2009) · Journal of Human Genetics · doi:10.1038/jhg.2009.95 · Cited by 111
Yang, Y., Sánchez‐Tójar, A., O’Dea, R. E., Noble, D. W. A., Koricheva, J., Jennions, M. D., Parker, T. H., Lagisz, M., & Nakagawa, S. (2023) · BMC Biology · doi:10.1186/s12915-022-01485-y · Cited by 120
The relevance of alpha phase in human perception
Ruzzoli, M., Cuello, M. T., Fernández, L. M., & Soto‐Faraco, S. (2019) · Cortex · doi:10.1016/j.cortex.2019.05.012 · Cited by 116
Improving natural product research translation: From source to clinical trial
Sorkin, B. C., Kuszak, A. J., Bloss, G., Fukagawa, N. K., Hoffman, F. A., Jafari, M., Barrett, B. P., Brown, P. N., Bushman, F. D., Casper, S. J., et al. (2019) · The FASEB Journal · doi:10.1096/fj.201902143r · Cited by 121
Disclosure Versus Recognition: Inferences from Subsequent Events
Michels, J. (2016) · Journal of Accounting Research · doi:10.1111/1475-679x.12128 · Cited by 118
O’Keefe, D. J. (2015) · Health Communication · doi:10.1080/10410236.2014.974123 · Cited by 98
“…this naturally leads to a published research literature that can be misleading (Chan, Hrobjartsson, Haahr, Gøtzsche, & Altman, 2004; Dwan, Gamble, Williamson, Kirkham, & the Reporting Bias Group, 2013; Gerber & Malhotra, 2008; Ioannidis, 2005, 2008; Simmons, Nelson, & Simonsohn, 2011).”
A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments
Dmitriev, P., Gupta, S., Kim, D. W., & Vaz, G. J. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098024 · Cited by 129
“If an underpowered metric turns out to be statistically significant, then it is more likely that the observed change of the metric will be exaggerated compared to the true effect size [26], which is often referred to as an example of Winner’s curse [27].”
BackgroundTemporal Instability of Evidence Base: A Threat to Policy Making?
Koricheva, J., & Kulinskaya, E. (2019) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2019.05.006 · Cited by 104
Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021) · PLOS Biology · doi:10.1371/journal.pbio.3001162 · Cited by 103
“Across RCTs, there is a major risk for inflated estimates and problems with randomization, allocation concealment, and blinding [2,3].”
BackgroundMillard, L. A. C., Davies, N. M., Timpson, N. J., Tilling, K., Flach, P. A., & Smith, G. D. (2015) · Scientific Reports · doi:10.1038/srep16645 · Cited by 96
Tian, T., Wang, M., Lin, S., Guo, Y., Dai, Z., Liu, K., Yang, P., Dai, C., Zhu, Y., Zheng, Y., et al. (2018) · Molecular Therapy — Nucleic Acids · doi:10.1016/j.omtn.2018.05.018 · Cited by 103
“98,99 Lastly, language bias may exist since only two languages were used in the literature review.”
Ladanie, A., Schmitt, A. M., Speich, B., Naudet, F., Agarwal, A., Pereira, T., Sclafani, F., Herbrand, A. K., Briel, M., Martín-Liberal, J., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.24406 · Cited by 107
Self-citationTaylor, P. A., Reynolds, R. C., Calhoun, V. D., González-Castillo, J., Handwerker, D. A., Bandettini, P. A., Mejia, A. F., & Chen, G. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120138 · Cited by 112
Predicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y., Nosek, B. A., Johannesson, M., & Dreber, A. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 104
Chung, M. K., House, J. S., Akhtari, F. S., Makris, K. C., Langston, M. A., Islam, K. T., Holmes, P., Chadeau‐Hyam, M., Smirnov, A. I., Du, X., et al. (2024) · Exposome · doi:10.1093/exposome/osae001 · Cited by 116
Neuroimaging in attention-deficit/hyperactivity disorder
Pereira-Sánchez, V., & Castellanos, F. X. (2020) · Current Opinion in Psychiatry · doi:10.1097/yco.0000000000000669 · Cited by 113
Methodological Variation in Empirical Corporate Finance
Mitton, T. (2022) · The Review of Financial Studies · doi:10.1093/rfs/hhab030 · Cited by 92
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“This flexibility can lead to large vibration of effects whereby many different results can be obtained from the same data and research question; results from the same study may often point in opposite directions (the so-called Janus phenomenon) (Ioannidis 2008, Patel et al. 2015).”
Self-citationPredicting Violent Behavior: What Can Neuroscience Add?
Poldrack, R. A., Monahan, J. T., Imrey, P. B., Reyna, V. F., Raichle, M. E., Faigman, D. L., & Buckholtz, J. W. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.11.003 · Cited by 109
He, Y., Li, X., Gasevic, D., Brunt, E., McLachlan, F., Millenson, M., Timofeeva, M., Ioannidis, J. P. A., Campbell, H., & Theodoratou, E. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0808 · Cited by 96
Self-citationTraining working memory: Limits of transfer
Sprenger, A. M., Atkins, S. M., Bolger, D. J., Harbison, J. I., Novick, J. M., Chrabaszcz, J. S., Weems, S. A., Smith, V., Bobb, S., Bunting, M. F., et al. (2013) · Intelligence · doi:10.1016/j.intell.2013.07.013 · Cited by 96
Bootstrapping in Applied Linguistics: Assessing its Potential Using Shared Data
Plonsky, L., Egbert, J., & Laflair, G. T. (2014) · Applied Linguistics · doi:10.1093/applin/amu001 · Cited by 89
“…this practice as a Type M (for magnitude) error, a notion closely related to the author- and editorially-rooted bias in favor publishing statistically significant results usually discussed in the context of meta-analysis (see also Norris and Ortega 2006; Ioannidis 2008; Author and Co-author 2012).”
BackgroundResultFusar-Poli, P., Werbeloff, N., Rutigliano, G., Oliver, D., Davies, C., Stahl, D., McGuire, P., & Osborn, D. (2019) · Schizophrenia Bulletin · doi:10.1093/schbul/sby070 · Cited by 95
A Guide for Calculating Study-Level Statistical Power for Meta-Analyses
Quintana, D. S. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221147260 · Cited by 95
“In addition, such study design/test combinations tend to be associated with questionable research practices (Dwan et al., 2008) and are more likely to report exaggerated effect sizes (Ioannidis, 2008; Rochefort-Maranda, 2021).”
BackgroundStegenga, J. (2013) · Topoi · doi:10.1007/s11245-013-9189-4 · Cited by 106
Just Add Data: automated predictive modeling for knowledge discovery and feature selection
Tsamardinos, I., Charonyktakis, P., Papoutsoglou, G., Borboudakis, G., Lakiotaki, K., Zenklusen, J., Juhl, H., Chatzaki, E., & Lagani, V. (2022) · npj Precision Oncology · doi:10.1038/s41698-022-00274-8 · Cited by 59
Cybulski, L., Mayo-Wilson, E., & Grant, S. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000115 · Cited by 90
Sample Size Determination and Statistical Power Analysis in PLS Using R: An Annotated Tutorial
Aguirre-Urreta, M., & Rönkkö, M. (2015) · Communications of the Association for Information Systems · doi:10.17705/1cais.03603 · Cited by 81
“This is an issue that has been discussed in the medical literature, where the overstatement of a treatment effect has major practical implications (Ioannidis, 2008; Yarkoni, 2009), but it has also started to make its way into the social and organizational sciences (e.”
BackgroundMethodElevated Risk of Preeclampsia in Pregnant Women With Depression: Depression or Antidepressants?
Palmsten, K., Setoguchi, S., Margulis, A. V., Patrick, A. R., & Hernández-Díaz, S. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kwr394 · Cited by 90
An Examination of Effect Sizes and Statistical Power in Speech, Language, and Hearing Research
Gaeta, L., & Brydges, C. R. (2020) · Journal of Speech Language and Hearing Research · doi:10.1044/2020_jslhr-19-00299 · Cited by 96
Grebe, N. M., Giudice, M. D., Thompson, M. E., Nickels, N., Ponzi, D., Zilioli, S., Maestripieri, D., & Gangestad, S. W. (2019) · Hormones and Behavior · doi:10.1016/j.yhbeh.2019.01.006 · Cited by 99
“Statistical power of only 16% presents problems of both an inflated false negative rate (i.e., given a real T´C interaction effect, underpowered studies will often fail to detect it) and an inflated estimate of effect size (“the winner’s curse”; see Ioannidis, 2008)5.”
Nicotinic acetylcholine receptor variation and response to smoking cessation therapies
Bergen, A. W., Javitz, H. S., Krasnow, R., Nishita, D., Michel, M., Conti, D. V., Liu, J., Lee, W., Edlund, C. K., Hall, S., et al. (2013) · Pharmacogenetics and Genomics · doi:10.1097/fpc.0b013e32835cdabd · Cited by 90
The sexual health, orientation, and activity of autistic adolescents and adults
Weir, E., Allison, C., & Baron‐Cohen, S. (2021) · Autism Research · doi:10.1002/aur.2604 · Cited by 79
Oral contraceptives and cognition: A role for ethinyl estradiol
Beltz, A. M., Hampson, E., & Berenbaum, S. A. (2015) · Hormones and Behavior · doi:10.1016/j.yhbeh.2015.06.012 · Cited by 93
Tsilidis, K. K., Papatheodorou, S. I., Evangelou, E., & Ioannidis, J. P. A. (2012) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djs437 · Cited by 93
Self-citationMuduma, G., Saunders, R., Odeyemi, I. A. O., & Pollock, R. F. (2016) · PLoS ONE · doi:10.1371/journal.pone.0160421 · Cited by 102
Environment, susceptibility windows, development, and child health
Wright, R. O. (2017) · Current Opinion in Pediatrics · doi:10.1097/mop.0000000000000465 · Cited by 96
Jordans, M. J. D., Luitel, N. P., Garman, E., Kohrt, B. A., Rathod, S. D., Shrestha, P., Komproe, I. H., Lund, C., & Patel, V. (2019) · British Journal of Psychiatry · doi:10.1192/bjp.2018.300 · Cited by 89
External validation of automated focal cortical dysplasia detection using morphometric analysis
David, B., Kröll‐Seger, J., Schuch, F., Wagner, J., Wellmer, J., Woermann, F. G., Oehl, B., Paesschen, W. V., Breyer, T., Becker, A. J., et al. (2021) · Epilepsia · doi:10.1111/epi.16853 · Cited by 89
Effect sizes of associations between neuroimaging measures and affective symptoms: A meta‐analysis
Feng, C., Thompson, W. K., & Paulus, M. P. (2021) · Depression and Anxiety · doi:10.1002/da.23215 · Cited by 97
“Moreover, the inflated effect size could be partly attributed to flexible analyses and selective reporting, known as “researcher degrees of freedom” (Ioannidis, 2008), an issue that could be efficiently handled with open science conventions of preregistration (Shrout & Rodgers, 2018).”
BackgroundHeterogeneity in effect size estimates
Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403490121 · Cited by 103
Human Fear Chemosignaling: Evidence from a Meta-Analysis
de Groot, J. H. B., & Smeets, M. A. M. (2017) · Chemical Senses · doi:10.1093/chemse/bjx049 · Cited by 88
Cluster failure or power failure? Evaluating sensitivity in cluster-level inference
Noble, S., Scheinost, D., & Constable, R. T. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116468 · Cited by 95
“Dovetailing with this, underpowered approaches often result in biased effect size estimates, which, in turn, leads to another type of bias whereby only effects inflated enough to pass the threshold are reported (Ioannidis, 2008; Lindquist and Mejia, 2015; Yarkoni, 2009).”
BackgroundManiadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 106
“In addition to proposing replicability strategies employed in other disciplines, it is important to examine whether it is possible to employ tools from the new discipline of meta-research (Ioannidis et al., 2015) to identify and explore the critical dimensions of the reproducibility of economic research. In this artic…”
Association of KIBRA with episodic and working memory: A meta‐analysis
Milnik, A., Heck, A., Vogler, C., Heinze, H., de Quervain, D. J. ‐., & Papassotiropoulos, A. (2012) · American Journal of Medical Genetics Part B: Neuropsychiatric Genetics · doi:10.1002/ajmg.b.32101 · Cited by 89
For and Against Methodologies: Some Perspectives on Recent Causal and Statistical Inference Debates
Greenland, S. (2017) · European Journal of Epidemiology · doi:10.1007/s10654-017-0230-6 · Cited by 86
Scientific inbreeding and same-team replication: Type D personality as an example
Ioannidis, J. P. A. (2012) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2012.09.014 · Cited by 94
“Analytical choices have been employed selectively [2], these choices lead to vibration of effects [3], and the best-looking results (among many possible results) are presented.”
BackgroundSelf-citationLi, N., Wu, X., Zhuang, W., Xia, L., Chen, Y., Zhao, R., Yi, M., Wan, Q., Du, L., & Zhou, Y. (2020) · Molecular Nutrition & Food Research · doi:10.1002/mnfr.201900751 · Cited by 80
The impact of sample size on the reproducibility of voxel-based lesion-deficit mappings
Lorca-Puls, D. L., Gajardo-Vidal, A., White, J., Seghier, M. L., Leff, A. P., Green, D. W., Crinion, J. T., Ludersdorfer, P., Hope, T. M. H., Bowman, H., et al. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.03.014 · Cited by 82
“Unreliable effect sizes in smaller samples 422 High variance in the results of our lesion-deficit mappings with smaller 423 samples (N = 30 and 60) demonstrates how effects can be over- as well as under-424 estimated (e.g., Cremers et al., 2017; Ioannidis, 2008).”
BackgroundResultInfluentialIoannidis, J. P. A. (2009) · Circulation: Cardiovascular Genetics · doi:10.1161/circgenetics.108.833392 · Cited by 85
Self-citationGenetic Cardiovascular Risk Prediction: Will We Get There?
Thanassoulis, G., & Vasan, R. S. (2010) · Circulation · doi:10.1161/circulationaha.109.909309 · Cited by 97
Mason, N. A., Shultz, A. J., & Burns, K. J. (2014) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2014.0967 · Cited by 98
“However, small sample sizes can lead to inflated effect sizes [51], suggesting that these correlations should be interpreted with caution.”
BackgroundEfficacy of pharmacotherapy for OSA in adults: A systematic review and network meta-analysis
Gaisl, T., Haile, S. R., Thiel, S., Osswald, M., & Kohler, M. (2019) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2019.04.009 · Cited by 84
The significance filter, the winner's curse and the need to shrink
van Zwet, E. W., & Cator, E. (2021) · Statistica Neerlandica · doi:10.1111/stan.12241 · Cited by 96
“by simulation, that the winner’s curse is especially severe when the power is low [10],[7].”
BackgroundInfluentialvan den Akker, O. R., van Assen, M. A. L. M., Bakker, M., Elsherif, M. M., Wong, T. K., & Wicherts, J. M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02277-0 · Cited by 99
“Researchers may therefore want to p -hack their way to a larger effect size in a similar way as they would to a positive result (Fanelli et al., 2017; Ioannidis, 2008).”
Resultvan Rijn, A. L., Roos, R., Dekker, F. W., Rotmans, J. I., & Feltkamp, M. (2022) · Reviews in Medical Virology · doi:10.1002/rmv.2393 · Cited by 84
Coyne, J. C., Thombs, B. D., & Hagedoorn, M. (2010) · Health Psychology · doi:10.1037/a0017633 · Cited by 80
“Even in the case of a true null, such a collection of published small studies would be expected to produce a robust effect size estimate (Howard et al., 2009; Ioannidis, 2008a; Kraemer et al., 1998).”
BackgroundResultAnimal models of Huntington's disease for translation to the clinic: Best practices
Menalled, L., & Brunner, D. (2014) · Movement Disorders · doi:10.1002/mds.26006 · Cited by 86
Bürkner, P.-C., Williams, D. R., Simmons, T. C., & Woolley, J. D. (2017) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx053 · Cited by 78
Chou, R., Atlas, S. J., Loeser, J. D., Rosenquist, R. W., & Stanos, S. P. (2011) · Journal of Pain · doi:10.1016/j.jpain.2011.04.012 · Cited by 85
Neurobiological substrates of personality: A critical overview.
Yarkoni, T. (2014) · American Psychological Association eBooks · doi:10.1037/14343-003 · Cited by 90
Hymas, R., Badcock, J. C., & Milne, E. (2022) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-022-00330-w · Cited by 89
“…influence of age, gender, population (child or adult), presence of intellectual disability (ID), publication status (given the existence of larger effects being found in published as compared with unpublished studies; Boland et al., 2014) and the methodological quality of studies (Ioannidis, 2008).”
Ingre, M. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.03.030 · Cited by 88
“…of these “effects” are large; it just shows that to “demonstrate” an effect, you have to be “lucky” and trust noise to produce a larger effect if your true effect would be smaller, inflating the reported size of “significant” effects in small underpowered studies (Ioannidis, 2008; Yarkoni, 2009).”
BackgroundResultVancraeyveldt, C., Verschueren, K., Wouters, S., Craeyevelt, S. V., Noortgate, W. V. D., & Colpin, H. (2014) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-014-9892-7 · Cited by 91
Ioannidis, J. P. A., & Karassa, F. B. (2010) · BMJ · doi:10.1136/bmj.c4875 · Cited by 78
“Secondly, early stopping: trials stopped early because of perceived effectiveness give inflated estimates of the treatment effect.”
BackgroundSelf-citationGoldberg, R., & Short, S. E. (2016) · AIDS Care · doi:10.1080/09540121.2016.1176684 · Cited by 81
“…is also limited by bias in the publication of articles (e.g., articles with statistically significant results) and in the effect sizes of newly discovered non-null associations (Chan, Hrobjartsson, Haahr, Gotzsche, & Altman, 2004; Ioannidis, 2008; Song, Eastwood, Gilbody, Duley, & Sutton, 2000).”
BackgroundConfidence crisis of results in biomechanics research
Knudson, D. V. (2017) · Sports Biomechanics · doi:10.1080/14763141.2016.1246603 · Cited by 87
“…to minimise cost of research often result in studies with small sample sizes that introduce a number of biases including low statistical power, inconsistent results (poor replicability), and inflated effect sizes (Button et al., 2013; Ioannidis, 2005, 2008; Maxwell, 2004; Schweizer & Furley, 2016).”
ResultRisk factors and interventions with statistically significant tiny effects
Siontis, G. C., & Ioannidis, J. P. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr099 · Cited by 80
Self-citationVul, E., Harris, C. R., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01132.x · Cited by 85
“On the other hand, if researchers test many hypotheses and report only the significant ones, the published literature will show a bias, even without biased analysis procedures (see also Ioannidis, 2008, for fascinating examples from epidemiology and medicine).”
BackgroundNew drug developments in psychosis: Challenges, opportunities and strategies
Keshavan, M. S., Lawler, A. N., Nasrallah, H. A., & Tandon, R. (2016) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2016.07.004 · Cited by 85
“The small effect size of genetic and biological findings in SCZ has been problematic making replications difficult (Ioannidis, 2008).”
Fast radio burst event rate counts – I. Interpreting the observations
Macquart, J., & Ekers, R. D. (2017) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stx2825 · Cited by 79
“This is a well known problem in some fields such as epidemiology, e.g. Ioannidis (2008).”
BackgroundLukas, C. A., & Berking, M. (2018) · Internet Interventions · doi:10.1016/j.invent.2017.07.002 · Cited by 68
“Given the small sample size, results of this study should be interpreted with caution as the small and homogenous sample may have led to an overestimation of effect sizes (Howard et al., 2009; Ioannidis, 2008).”
ResultHornshøj, H., Nielsen, M. M., Sinnott-Armstrong, N. A., Świtnicki, M. P., Juul, M., Madsen, T., Sallari, R., Kellis, M., Ørntoft, T., Hobolth, A., et al. (2018) · npj Genomic Medicine · doi:10.1038/s41525-017-0040-5 · Cited by 74
“Such drivers are unlikely to be consistently detected across sets, known as winner’s curse.(20) Overall 17 elements were recalled (Supplementary Table 2), including eight protein-coding genes (TP53, KRAS, FBXW7, PIK3CA, TMEM132C, CSMD1, BRINP3, and CDH10), one enhancer (associated with the known TERT promoter sites(8,…”
BackgroundIssues in Differential Response
Hughes, R. C., Rycus, J. S., Saunders-Adams, S., Hughes, L. K., & Hughes, K. N. (2013) · Research on Social Work Practice · doi:10.1177/1049731512466312 · Cited by 78
Working memory capacity and Stroop interference: Global versus local indices of executive control.
Meier, M. E., & Kane, M. J. (2012) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/a0029200 · Cited by 88
Questionable science and reproducibility in electrical brain stimulation research
Héroux, M. E., Loo, C. K. A., Taylor, J. L., & Gandevia, S. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175635 · Cited by 80
Increasing the statistical power of animal experiments with historical control data
Bonapersona, V., Hoijtink, H., RELACS, Sarabdjitsingh, R. A., & Joëls, M. (2021) · Nature Neuroscience · doi:10.1038/s41593-020-00792-3 · Cited by 88
Schmidt, R. J., Steeves, M. A., Bayrak‐Toydemir, P., Benson, K. A., Coe, B. P., Conlin, L. K., Ganapathi, M., Garcia, J., Gollob, M. H., Jobanputra, V., et al. (2023) · Genetics in Medicine · doi:10.1016/j.gim.2023.101036 · Cited by 85
Estimating Population Mean Power Under Conditions of Heterogeneity and Selection for Significance
Brunner, J., & Schimmack, U. (2020) · Meta-Psychology · doi:10.15626/mp.2018.874 · Cited by 76
“Information about the average power of studies is also useful because selection for significance increases the type-I error rate and inflates effect sizes (Ioannidis, 2008).”
BackgroundThe continuing misuse of null hypothesis significance testing in biological anthropology
Smith, R. J. (2018) · American Journal of Physical Anthropology · doi:10.1002/ajpa.23399 · Cited by 80
Gagné, N., & Franzen, L. (2023) · Swiss Psychology Open · doi:10.5334/spo.34 · Cited by 79
Lucotte, E. A., Laurent, R., Heyer, É., Ségurel, L., & Toupance, B. (2016) · Genome Biology and Evolution · doi:10.1093/gbe/evw090 · Cited by 79
Iacono, W. G., Malone, S. M., & Vrieze, S. I. (2017) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2016.07.516 · Cited by 68
“…of the variance in ERN amplitude in the initial study of Kramer et al., it only accounted for about 3% of variance in the subsequent study of Agam and colleagues (the “winner's curse”; Ioannidis, 2008), and this latter effect was statistically significant only because a one-tailed test was used.”
BackgroundResultInfluentialIllusions in Regression Analysis
Armstrong, J. S. (2011) · doi:10.2139/ssrn.1969740 · Cited by 73
“Ioannidis (2005, 2008) provides more reasons why regressions over-estimate changes.”
When decision heuristics and science collide
Yu, E. C., Sprenger, A. M., Thomas, R. P., & Dougherty, M. R. P. (2013) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0495-z · Cited by 85
“However, power analyses are imperfect, since effect size calculations tend to overestimate the population effect sizes when based on either small samples (e.g., pilot data; Fiedler, 2011; Ioannidis, 2008) or published work (i.”
Background
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- Epidemiology
- Published
- 2008-09
- Volume
- 19
- Issue
- 5
- Pages
- 640-648
- ISSN
- 1044-3983 (ISSN-L)
- OpenAlex
- W2150215887
- Semantic Scholar
- 0373f18622b7b64e99c7e56340ab2011dbd00b5d
- Cited by
- 1,495
- References
- 109
- Citations by year
- 2026: 91 · 2025: 87 · 2024: 78 · 2023: 112 · 2022: 94 · 2021: 99 · 2020: 103 · 2019: 131 · 2018: 108 · 2017: 101 · 2016: 87 · 2015: 77 · 2014: 72 · 2013: 60 · 2012: 50
- FWCI
- 15.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- effect size inflation, statistical significance thresholds, underpowered studies, selective reporting, conflicts of interest, reverse bias, measurement error, replication
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Humans, Sensitivity and Specificity, Models, Statistical, Linkage Disequilibrium, Molecular Epidemiology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.