Why Most Discovered True Associations Are Inflated
The DOI matches its Crossref record; 111 of the 111 references it cites were found in a registry or an index.
Abstract
Newly discovered true (non-null) associations often have inflated effects compared with the true effect sizes. I discuss here the main reasons for this inflation. First, theoretical considerations prove that when true discovery is claimed based on crossing a threshold of statistical significance and the discovery study is underpowered, the observed effects are expected to be inflated. This has been demonstrated in various fields ranging from early stopped clinical trials to genome-wide associations. Second, flexible analyses coupled with selective reporting may inflate the published discovered effects. The vibration ratio (the ratio of the largest vs. smallest effect on the same association approached with different analytic choices) can be very large. Third, effects may be inflated at the stage of interpretation due to diverse conflicts of interest. Discovered effects are not always inflated, and under some circumstances may be deflated-for example, in the setting of late discovery of associations in sequentially accumulated overpowered evidence, in some types of misclassification from measurement error, and in conflicts causing reverse biases. Finally, I discuss potential approaches to this problem. These include being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery ph…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Given that initial effect sizes are often inflated 61 and that we sought power to detect a difference between two groups, we doubled this estimate and aimed to test ~28 participants in each group with some attrition expected.”
From Adults with autism overestimate the volatility of the sensory environment (2017) · cites this paper as [61]
In the citing paper
“While this may reflect exaggeration of original effect sizes or publication bias (Ioannidis 2008), some degree of inflation is expected even in its absence, as our selection process filtered for original experiments with significant differences.”
From Estimating the replicability of Brazilian biomedical science (2025) · cites this paper as Ioannidis 2008
In the citing paper
“Further, in situations when an underpowered study detects a true effect, the estimate of this effect size is likely to be highly exaggerated, a phenomenon often referred to as the “winner’s curse” (15).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [15]
In the citing paper
“Researchers have strong incentives to interpret their results as supporting or extending the Bayesian framework, rather than challenging it—leading to a literature that systematically over-represents supportive evidence (Ioannidis 2008; Nosek et al. 2012).”
From The myth of the Bayesian brain (2025) · cites this paper as Ioannidis 2008
In the citing paper
“This flexibility in the synthesis of the evidence provides the assessors with the opportunity to inflate or disinflate the efficacy of a given technology, a phenomenon that was observed in scientific publications.[24]”
From Health technology assessments conducted in health care facilities: A strategic practice? Findings from a content analysis of HTA reports (2017) · cites this paper as [24]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationGene expression profiling predicts clinical outcome of breast cancer
van 't Veer, L. J., Dai, H., van de Vijver, M. J., He, Y. D., Hart, A. A. M., Mao, M., Peterse, H. L., van der Kooy, K., Marton, M. J., Witteveen, A. T., et al. (2002) · Nature · doi:10.1038/415530a · Cited by 9,536
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“This article will not deal with the debate on whether research findings are credible.(1) I will focus instead on the interesting subset of research findings that are true.”
BackgroundSelf-citationEffect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Nissen, S. E., & Wolski, K. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072761 · Cited by 4,675
The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2007) · Epidemiology · doi:10.1097/ede.0b013e3181577511 · Cited by 2,779
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544
Cited by
Items 101–200 of 1,004
Comparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
Frone, M. R., & Tidwell, M.-C. O. (2015) · Journal of Occupational Health Psychology · doi:10.1037/a0038700 · Cited by 231
“…of Study 2 was that it used a broad probability sample of employed adults in the U.S. Compared with most convenience samples, the present sample was large thereby providing adequate statistical power to detect the hypothesized effects and providing more accurate effect sizes (Ioannidis, 2008).”
Sena, E. S., Currie, G. L., McCann, S. K., Macleod, M. R., & Howells, D. W. (2014) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2014.28 · Cited by 264
Breckenridge, C. B., Berry, C., Chang, E. T., Sielken, R. L., & Mandel, J. S. (2016) · PLOS ONE · doi:10.1371/journal.pone.0151841 · Cited by 239
Schmaal, L., Pozzi, E., C. Ho, T., van Velzen, L. S., Veer, I. M., Opel, N., Van Someren, E. J. W., Han, L. K. M., Aftanas, L., Aleman, A., et al. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0842-6 · Cited by 245
Can Policies Stall the Fertility Fall? A Systematic Review of the (Quasi‐) Experimental Literature
Bergsvik, J., Fauske, A., & Hart, R. K. (2021) · Population and Development Review · doi:10.1111/padr.12431 · Cited by 259
TREM2 and Neurodegenerative Disease
(2013) · New England Journal of Medicine · doi:10.1056/nejmc1306509 · Cited by 254
Empirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2012.13444 · Cited by 257
Self-citationIoannidis, J. P. A. (2011) · JAMA · doi:10.1001/jama.2011.713 · Cited by 252
Self-citationFalse Alarms and Pseudo-Epidemics: The Limitations of Observational Epidemiology
Grimes, D. A., & Schulz, K. F. (2012) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e31826af61a · Cited by 254
An umbrella review of the evidence associating diet and cancer risk at 11 anatomical sites
Papadimitriou, N., Markozannes, G., Kanellopoulou, A., Critselis, E., Alhardan, S., Karafousia, V., Kasimis, J. C., Katsaraki, C., Papadopoulou, A., Zografou, M., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24861-8 · Cited by 234
Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials?
Ioannidis, J. P. (2008) · Philosophy, Ethics, and Humanities in Medicine · doi:10.1186/1747-5341-3-14 · Cited by 253
“In small trials, the same amount of manipulation will inflate the effect size more than in large trials, in other words the vibration of the effect size is larger [12,13].”
Self-citationThe Psychology of Replication and Replication in Psychology
Francis, G. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459520 · Cited by 238
“Ioannidis (2008) notes that effect size underestimation can happen for some sequential sampling situations when the true effect size is large because an experiment can reject the null hypothesis with an underestimated value of the effect size.”
BackgroundThe Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246
“Replication can occur when authors make their data available for reanalysis, but as described above, this is exceedingly rare (Ioannidis, 2005, 2008).”
BackgroundPublication bias and the failure of replication in experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0322-y · Cited by 213
“Thus, one side effect of a file drawer publication bias is that both the effect size and experimental power are overestimated (Hedges, 1984; Ioannidis, 2008; Lane & Dunlap, 1978).”
BackgroundEthical Standards in Sport and Exercise Science Research: 2022 Update
Harriss, D. J., Jones, C., & MacSween, A. (2022) · International Journal of Sports Medicine · doi:10.1055/a-1957-2356 · Cited by 220
Meaningful associations in the adolescent brain cognitive development study
Dick, A. S., Lopez, D. A., Watts, A. L., Heeringa, S., Reuter, C., Bartsch, H., Fan, C. C., Kennedy, D. N., Palmer, C., Marshall, A., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118262 · Cited by 241
Meta‐research: The art of getting it wrong
Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.19 · Cited by 227
“2010, 1 169--184 1 7 5 A number of tests have been developed by meta-analysts to detect and exclude publication bias, but I am afraid that they are used beyond their real capacity (Ioannidis, 2008a; Ioannidis and Trikalinos, 2007b; Lau et al., 2006).”
BackgroundInfluentialSelf-citationKharabian Masouleh, S., Eickhoff, S. B., Hoffstaedter, F., Genon, S., & Initiative, A. D. N. (2019) · eLife · doi:10.7554/elife.43464 · Cited by 230
Curses—Winner's and Otherwise—in Genetic Epidemiology
Kraft, P. (2008) · Epidemiology · doi:10.1097/ede.0b013e318181b865 · Cited by 225
Too good to be true: Publication bias in two prominent studies from experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0227-9 · Cited by 206
“In fact, the test used here is strongly inclined to not report a publication bias, because reported effect sizes tend to overestimate reality (Ioannidis, 2008).”
BackgroundCancer pharmacogenomics: strategies and challenges
Wheeler, H. E., Maitland, M. L., Dolan, M. E., Cox, N. J., & Ratain, M. J. (2013) · Nature Reviews Genetics · doi:10.1038/nrg3352 · Cited by 215
Everything You Never Wanted to Know about Circular Analysis, but Were Afraid to Ask
Kriegeskorte, N., Lindquist, M. A., Nichols, T. E., Poldrack, R. A., & Vul, E. (2010) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2010.86 · Cited by 214
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219
“…be based on the results of p-uniform or p-curve, rather than on fixed-effect and random-effects meta-analysis, because these traditional methods overestimate effect size in the presence of publication bias (e.g., Bakker et al., 2012; Ioannidis, 2008; Lane & Dunlap, 1978; van Assen et al., 2015).”
BackgroundMethodEvangelou, E., Chapman, K., Meulenbelt, I., Karassa, F. B., Loughlin, J., Carr, A., Doherty, M., Doherty, S., Gómez‐Reino, J. J., Gonzalez, A., et al. (2009) · Arthritis & Rheumatism · doi:10.1002/art.24524 · Cited by 201
“The more prominent effect sizes reported in earlier studies may have been inflated (41), and the large number of samples now analyzed provides an estimate of genetic effect in knee OA that is probably more realistic.”
BackgroundSelf-citationp-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210
“It may cause what is called a vibration of effects [17] and it is well known to be a key threat to the validity of inferences in observational research [14,18,19].”
BackgroundSelf-citationO'Keefe, D. J., & Nan, X. (2012) · Health Communication · doi:10.1080/10410236.2011.640974 · Cited by 197
McAllister, K., Mechanic, L. E., Amos, C., Aschard, H., Blair, I. A., Chatterjee, N., Conti, D., Gauderman, W. J., Hsu, L., Hutter, C. M., et al. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx227 · Cited by 208
Giannakou, K., Evangelou, E., & Papatheodorou, S. I. (2018) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.18959 · Cited by 201
Bibliometric Evidence for a Hierarchy of the Sciences
Fanelli, D., & Glänzel, W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066938 · Cited by 197
“Theory and empirical evidence, for example, suggest that the frequency of false positives and publication biases vary with the level of scholarly consensus [6,73,74,75].”
BackgroundFinkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 202
Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019) · Psychophysiology · doi:10.1111/psyp.13437 · Cited by 184
Justice, A. E., Winkler, T. W., Feitosa, M. F., Graff, M., Fisher, V. A., Young, K., Barata, L., Deng, X., Czajkowski, J., Hadley, D., et al. (2017) · Nature Communications · doi:10.1038/ncomms14977 · Cited by 206
Is everything we eat associated with cancer? A systematic cookbook review
Schoenfeld, J. D., & Ioannidis, J. P. (2013) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.112.047142 · Cited by 198
“Our findings support previous evidence suggesting that effect sizes are likely to trend closer to the null as more data are accumulated (55).”
ResultSelf-citationParacelsus Revisited: The Dose Concept in a Complex World
Grandjean, P. (2016) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/bcpt.12622 · Cited by 179
Growth factors for treating diabetic foot ulcers
Martí-Carvajal, A. J., Gluud, C., Nicola, S., Simancas-Racines, D., Reveiz, L., Oliva, P., & Cedeño-Taborda, J. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008548.pub2 · Cited by 192
The statistical significance filter leads to overoptimistic expectations of replicability
Vasishth, S., Mertzen, D., Jäger, L. A., & Gelman, A. (2018) · Journal of Memory and Language · doi:10.1016/j.jml.2018.07.004 · Cited by 204
Obesity and gynaecological and obstetric conditions: umbrella review of the literature
Kalliala, I., Markozannes, G., Gunter, M. J., Paraskevaidis, E., Gabra, H., Mitra, A., Terzidou, V., Bennett, P., Martin-Hirsch, P., Tsilidis, K. K., et al. (2017) · BMJ · doi:10.1136/bmj.j4511 · Cited by 199
Trusting Judgements: How to Get the Best out of Experts
Burgman, M. A. (2015) · Cambridge University Press eBooks · doi:10.1017/cbo9781316282472 · Cited by 239
Mapping brain asymmetry in health and disease through the ENIGMA consortium
Kong, X., Postema, M. C., Guadalupe, T., de Kovel, C., Boedhoe, P. S. W., Hoogman, M., Mathias, S. R., van Rooij, D., Schijven, D., Glahn, D. C., et al. (2022) · Human Brain Mapping · doi:10.1002/hbm.25033 · Cited by 186
“…for inclusion in publications that yield positive results (p-hacking) (Aarts, et al., 2015; Baker, 2016; Bakker, van Dijk, & Wicherts, 2012; Ioannidis, 2005; Ioannidis, 2008; Ioannidis, Munafo, Fusar-Poli, Nosek, & David, 2014; John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simonsohn, 2011).”
Statistical considerations for reporting and planning heart rate variability case‐control studies
Quintana, D. S. (2017) · Psychophysiology · doi:10.1111/psyp.12798 · Cited by 192
Gene annotation bias impedes biomedical research
Haynes, W. A., Tomczak, A., & Khatri, P. (2018) · Scientific Reports · doi:10.1038/s41598-018-19333-x · Cited by 192
Body Mass Index Categories in Observational Studies of Weight and Risk of Death
Flegal, K. M., Kit, B. K., & Graubard, B. I. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu111 · Cited by 176
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
“…with the filter of an academic review process that primarily permits statistically significant results to appear in the published record (Fanelli, 2010), the end result is a research literature filled with inaccurately estimated effect sizes as a result of publication bias (Ioannidis, 2005, 2008).”
BackgroundKemp, A. H., Brunoni, A. R., Santos, I. S., Nunes, M. A., Dantas, E. M., Carvalho de Figueiredo, R., Pereira, A. C., Ribeiro, A. L. P., Mill, J. G., Andreão, R. V., et al. (2014) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2014.13121605 · Cited by 189
The Song Is You: Preferences for Musical Attribute Dimensions Reflect Personality
Greenberg, D. M., Kosiński, M., Stillwell, D. J., Monteiro, B., Levitin, D. J., & Rentfrow, P. J. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616641473 · Cited by 238
“…et al., 2008; Dunn et al., 2011; George et al., 2007; Langmeyer, et al., 2012; Ren-tfrow & Gosling, 2003; Zweigenhaft, 2008), and considering the comparatively large sample size used, the correlation estimates we observed should be quite stable (Ioannidis, 2008; Scho¨nbrodt & Perugini, 2013).”
MethodFluid therapy in neurointensive care patients: ESICM consensus and clinical practice recommendations
Oddo, M., Poole, D., Helbok, R., Meyfroidt, G., Stocchetti, N., Bouzat, P., Cecconi, M., Geeraerts, T., Martin-Loeches, I., Quintard, H., et al. (2018) · Intensive Care Medicine · doi:10.1007/s00134-018-5086-z · Cited by 179
“The body of evidence was downgraded because of the high degree of imprecision due to small sample size and the risk of inflated effect [14].”
BackgroundThe role of neuroscience within psychology: A call for inclusiveness over exclusiveness.
Schwartz, S. J., Lilienfeld, S. O., Meca, A., & Sauvigné, K. C. (2016) · American Psychologist · doi:10.1037/a0039678 · Cited by 180
Genetics of schizophrenia: new findings and challenges
Gejman, P. V., Sanders, A. R., & Kendler, K. S. (2011) · Annual Review of Genomics and Human Genetics · doi:10.1146/annurev-genom-082410-101459 · Cited by 183
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016) · PLOS ONE · doi:10.1371/journal.pone.0157732 · Cited by 161
Kent, D. M., van Klaveren, D., Paulus, J. K., D'Agostino, R., Goodman, S., Hayward, R., Ioannidis, J. P. A., Patrick-Lake, B., Morton, S., Pencina, M., et al. (2020) · Annals of Internal Medicine · doi:10.7326/m18-3668 · Cited by 179
Self-citationEnteral lactoferrin supplementation for very preterm infants: a randomised placebo-controlled trial
Griffiths, J., Jenkins, P., Vargova, M., Bowler, U., Juszczak, E., King, A., Linsell, L., Murray, D., Partlett, C., Patel, M., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(18)32221-9 · Cited by 171
Cognitive neuroscience 2.0: building a cumulative science of human brain function
Yarkoni, T., Poldrack, R. A., Van Essen, D. C., & Wager, T. D. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.08.004 · Cited by 174
Poor performance of clinical prediction models: the harm of commonly applied methods
Steyerberg, E. W., Uno, H., Ioannidis, J. P. A., van Calster, B., Ukaegbu, C., Dhingra, T., Syngal, S., & Kastrinos, F. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.013 · Cited by 168
Self-citationKohl, S. H., Mehler, D. M. A., Lührs, M., Thibault, R. T., Konrad, K., & Sorger, B. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00594 · Cited by 161
“…a lacuna in reporting regulation success is evident as well, which, together with a high level of degrees of freedom on the part of the researchers as outlined above, raises suspicions of selectively reporting positive results (Simmons et al., 2011) and inflated effect sizes (Ioannidis, 2008).”
Dubois, J., Galdi, P., Han, Y., Paul, L. K., & Adolphs, R. (2018) · Personality Neuroscience · doi:10.1017/pen.2018.8 · Cited by 172
“The “ vibration of effects ” due to analytical flexibility (Ioannidis 2008; Varoquaux 2017) should be reported rather than exploited .”
Elosua, R., Sayols-Baixeras, S., Lluís-Ganella, C., & Lucas, G. (2014) · The Application of Clinical Genetics · doi:10.2147/tacg.s35301 · Cited by 162
Economic Significance in Corporate Finance
Mitton, T. (2022) · The Review of Corporate Finance Studies · doi:10.1093/rcfs/cfac008 · Cited by 184
Taylor, A. E., Davies, N. M., Ware, J. J., VanderWeele, T., Smith, G. D., & Munafò, M. R. (2014) · Economics & Human Biology · doi:10.1016/j.ehb.2013.12.002 · Cited by 166
Philosophy of science and the replicability crisis
Romero, F. (2019) · Philosophy Compass · doi:10.1111/phc3.12633 · Cited by 185
Heritability in the genome-wide association era
Zaitlen, N., & Kraft, P. (2012) · Human Genetics · doi:10.1007/s00439-012-1199-6 · Cited by 157
“The winner's curse (Ioannidis 2007, 2008; Kraft 2008) and the inclusion of false-positive markers in the bottom-up estimate of genetic variance could in principle lead to an overestimate of heritability.”
Deep phenotyping of post-infectious myalgic encephalomyelitis/chronic fatigue syndrome
Walitt, B. T., Singh, K., LaMunion, S. R., Hallett, M., Jacobson, S., Chen, K. Y., Enose‐Akahata, Y., Apps, R., Barb, J. J., Bédard, P., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45107-3 · Cited by 168
Use of the “Exposome” in the Practice of Epidemiology: A Primer on -Omic Technologies
DeBord, D. G., Carreón, T., Lentz, T. J., Middendorf, P. J., Hoover, M. D., & Schulte, P. A. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kwv325 · Cited by 176
“Improving the accuracy of research findings will require better powered studies, diminishing bias, and improvement in our understanding of R values (62, 63).”
Pereira, T. V., & Ioannidis, J. P. A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.12.012 · Cited by 160
Self-citationFrone, M. R. (2015) · Journal of Occupational Health Psychology · doi:10.1037/a0038375 · Cited by 136
Medina, J., & Cason, S. (2017) · Cortex · doi:10.1016/j.cortex.2017.06.021 · Cited by 168
“However, given that underpowered studies can only detect large effects (the “winner's curse”, Button et al., 2013; Ioannidis, 2008), it is imperative that these replications be done to examine whether published findings have evidential value.”
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 169
Larson, M. J., & Carbine, K. A. (2016) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2016.06.015 · Cited by 162
“…is to complete a priori sample size calculations to ensure that studies are suf fi ciently powered, to reduce the probability of false negative fi ndings, and to ensure that effect sizes are not in fl ated (Button et al., 2013; Ioannidis, 2005, 2008; Sawyer and Ball, 1981; Sterne and Smith, 2001).”
BackgroundChing, C. R. K., Hibar, D. P., Gurholt, T. P., Nunes, A., Thomopoulos, S. I., Abé, C., Agartz, I., Brouwer, R. M., Cannon, D. M., de Zwarte, S. M. C., et al. (2020) · Human Brain Mapping · doi:10.1002/hbm.25098 · Cited by 165
“These core challenges have contributed to the “replication crisis” affecting much of biomedical research (Button et al., 2013; DumasMallet, Button, Boraud, Gonon, & Munafo, 2017; Ioannidis, 2008, 2011, 2017).”
BackgroundBeyond odds ratios — communicating disease risk based on genetic profiles
Kraft, P. L., Wacholder, S., Cornelis, M. C., Hu, F. B., Hayes, R. B., Thomas, G. D., Hoover, R. N., Hunter, D. J., & Chanock, S. J. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2516 · Cited by 158
“Second, data from GWA studies can be used to estimate some measures of clinical validity, although these data are potentially subject to bias owing to design artefact or to the 'winner's curse' (the inflation in measures of association that can occur when discovered loci have to pass stringent significance threshold”
MethodFrom Methodology to Practice: Inductive Iteration in Comparative Research
Yom, S. (2015) · Comparative Political Studies · doi:10.1177/0010414014554685 · Cited by 110
“Such last-minute tweaking inflates reported causal effects (Ioannidis, 2008; Vul, Harris, Winkielman, & Pashler, 2009).”
BackgroundPatel, C. J., Cullen, M. R., Ioannidis, J. P. A., & Butte, A. J. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys003 · Cited by 154
Self-citationBernardi, F., Chakhaia, L., & Leopold, L. (2016) · European Sociological Review · doi:10.1093/esr/jcw047 · Cited by 147
Khoury, M. J., Bertram, L., Boffetta, P., Butterworth, A. S., Chanock, S. J., Dolan, S. M., Fortier, I., Garcia-Closas, M., Gwinn, M., Higgins, J. P. T., et al. (2009) · American Journal of Epidemiology · doi:10.1093/aje/kwp119 · Cited by 146
“These estimates may be inflated in comparison with the true effects, because of the ‘‘winner’s curse’’ phenomenon (inflation of effects selected based on significance thresholds) (41, 42).”
BackgroundSelf-citationShin, A., Camilleri, M., Kolar, G., Erwin, P., West, C. P., & Murad, M. H. (2014) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.12571 · Cited by 145
“Although this evidence is derived from RCTs, it is considered to be of moderate quality, due to heterogeneity and the fact that novel therapies usually have an exaggerated effect size in their earlier publications, the so-called, Proteus phenomenon.(39) Continued development of this class of drugs and ongoing investig…”
MethodAssociation of Apathy With Risk of Incident Dementia: A Systematic Review and Meta-analysis
van Dalen, J. W., van Wanrooij, L. L., Moll van Charante, E. P., Brayne, C., van Gool, W. A., & Richard, E. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.1877 · Cited by 132
Influence of land tenure interventions on human well-being and environmental outcomes
Tseng, T.-W. J., Robinson, B. E., Bellemare, M. F., BenYishay, A., Blackman, A., Boucher, T., Childress, M., Holland, M. B., Kroeger, T., Linkow, B., et al. (2020) · Nature Sustainability · doi:10.1038/s41893-020-00648-5 · Cited by 140
Steblay, N. K., Wells, G. L., & Douglass, A. B. (2014) · Psychology, Public Policy, and Law · doi:10.1037/law0000001 · Cited by 125
“…in recent years about the tendency of some research effects—that appear real based on initial studies—to lessen or even disappear over time as researchers attempt to replicate and refine the original findings or to ascertain the boundaries of the effect (e.g., Ionnadis, 2008; Schooler, 2011).”
ResultEnsuring the quality and specificity of preregistrations
Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000937 · Cited by 138
Translating biomarkers to clinical practice
Perlis, R. H. (2011) · Molecular Psychiatry · doi:10.1038/mp.2011.63 · Cited by 133
Tonelli, A. R., Zein, J., Adams, J., & Ioannidis, J. P. A. (2014) · Intensive Care Medicine · doi:10.1007/s00134-014-3272-1 · Cited by 133
“Small trials are likely more susceptible to selective reporting of analyses and outcomes, and results may become even more confusing with emphasis on subgroup analyses and other secondary explorations of the data [8, 84].”
BackgroundSelf-citationUS studies may overestimate effect sizes in softer research
Fanelli, D., & Ioannidis, J. P. A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1302997110 · Cited by 130
“The publication of false, exaggerated, and falsified findings is believed to be more common in research fields where replication is difficult, theories are less clear, and methods are less standardized, because researchers have more “degrees of freedom” to produce the results they expect (6, 14).”
BackgroundSelf-citationReplicating Studies in Which Samples of Participants Respond to Samples of Stimuli
Westfall, J., Judd, C. M., & Kenny, D. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691614564879 · Cited by 119
“It is, of course, important for any study to have high statistical power—a fact of which there is a growing awareness in psychology (Asendorpf et al., 2013; Bakker, van Dijk, & Wicherts, 2012; Button et al., 2013; Ioannidis, 2008; Schimmack, 2012).”
BackgroundEscaping Paternalism: Rationality, Behavioral Economics, and Public Policy
Rizzo, M. J., & Whitman, G. (2019) · Cambridge University Press eBooks · doi:10.1017/9781139061810 · Cited by 184
Kourtesis, P., Argelaguet, F., Vizcay, S., Marchal, M., & Pacchierotti, C. (2022) · IEEE Transactions on Haptics · doi:10.1109/toh.2022.3189866 · Cited by 144
“Under-powered statistical analyses may inflate or deflate considerably the observed effect sizes [135].”
BackgroundBarling, J., & Frone, M. R. (2017) · Stress and Health · doi:10.1002/smi.2697 · Cited by 130
“Further, compared with typical convenience samples, the present sample provided adequate statistical power to detect the hypothesized effects and provide more accurate estimates of population effect sizes (e.g. Ioannidis, 2005, 2008).”
Vive les differences! Individual variation in neural mechanisms of executive control
Braver, T. S., Cole, M. W., & Yarkoni, T. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.03.002 · Cited by 130
“The problem is twofold: first, studies with small sample sizes will tend to identify only a small fraction of populationlevel effects in any given sample; and second, because of increased sampling error, effect sizes derived from small samples will tend to be inflated — often grossly so [15].”
BackgroundFortune, M. D., Guo, H., Burren, O., Schofield, E., Walker, N. M., Ban, M., Sawcer, S. J., Bowes, J., Worthington, J., Barton, A., et al. (2015) · Nature Genetics · doi:10.1038/ng.3330 · Cited by 133
Patel, C. J., Rehkopf, D. H., Leppert, J. T., Bortz, W. M., Cullen, M. R., Chertow, G. M., & Ioannidis, J. P. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dyt208 · Cited by 127
Self-citationGenome‐wide screening for DNA variants associated with reading and language traits
Gialluisi, A., Newbury, D. F., Willcutt, E. G., Olson, R. K., DeFries, J. C., Brandler, W. M., Pennington, B. F., Smith, S. D., Scerri, T. S., Simpson, N. H., et al. (2014) · Genes Brain & Behavior · doi:10.1111/gbb.12158 · Cited by 141
“is the first report of these associations (Ioannidis 2008).”
BackgroundReplication, statistical consistency, and publication bias
Francis, G. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.02.003 · Cited by 138
“Thus, experiments based on optional stopping can simultaneously underestimate the magnitude of true e↵ects and exaggerate the rejection rate of experiments (Ioannidis, 2008; Francis, 2012g).”
BackgroundPre‐eclampsia and long‐term health outcomes for mother and infant: an umbrella review
Pittara, T., Vyrides, A., Lamnisos, D., & Giannakou, K. (2021) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16683 · Cited by 125
Consortium for Transparency in Exercise Science (COTES) Collaborators, Caldwell, A. R., Vigotsky, A. D., Tenan, M. S., Radel, R., Mellor, D. T., Kreutzer, A., Lahart, I. M., Mills, J. P., & Boisgontier, M. P. (2020) · Sports Medicine · doi:10.1007/s40279-019-01227-1 · Cited by 114
“In addition to false positives, more extreme observations or larger effect sizes are more often published because small studies have to report a large effect size to reach statistical significance thresholds [39, 40].”
BackgroundA meta-analytic review of psychodynamic therapies for anxiety disorders
Keefe, J. R., McCarthy, K. S., Dinger, U., Zilcha‐Mano, S., & Barber, J. P. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.03.004 · Cited by 138
“Due to the relatively small sample size (K = 14 studies) for our meta-analysis, significant moderator findings are considered exploratory (Hedges & Pigott, 2004; Ioannidis, 2008) and should be interpreted in context of their contributing samples (Tables 2 & 3).”
BackgroundCohen, M. X. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118809 · Cited by 128
Effect Size Estimation in Neuroimaging
Reddan, M. C., Lindquist, M. A., & Wager, T. D. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.3356 · Cited by 123
Making Prospective Registration of Observational Research a Reality
Dal-Ré, R., Ioannidis, J. P., Bracken, M. B., Buffler, P. A., Chan, A.-W., Franco, E. L., La Vecchia, C., & Weiderpass, E. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3007513 · Cited by 123
“Moreover, hypothesis-based research is more likely to be appropriately powered and thus less likely to produce inflated associations (15).”
BackgroundSelf-citationWork stress and alcohol use: developing and testing a biphasic self-medication model
Frone, M. R. (2016) · Work & Stress · doi:10.1080/02678373.2016.1252971 · Cited by 134
“Also, compared to most convenience samples, the present sample provided adequate statistical power to detect the hypothesized effects and provided more accurate estimates of population effect sizes (Ioannidis, 2008; Schmidt, 1992).”
ResultYi, M., Wu, X., Zhuang, W., Xia, L., Chen, Y., Zhao, R., Wan, Q., Du, L., & Zhou, Y. (2019) · Molecular Nutrition & Food Research · doi:10.1002/mnfr.201900389 · Cited by 116
WHAT'S TO KNOW ABOUT THE CREDIBILITY OF EMPIRICAL ECONOMICS?
Ioannidis, J., & Doucouliagos, C. (2013) · Journal of Economic Surveys · doi:10.1111/joes.12032 · Cited by 118
“…possible selective reporting biases remain common in randomized trials in other disciplines (Dwan et al., 2008).4 Similarly, we do not know how vulnerable these studies are to the winner’s curse, i.e. inflated effect sizes when effects are discovered by relatively small studies (Ioannidis, 2008).”
BackgroundSelf-citation
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- Epidemiology
- Published
- 2008-09
- Volume
- 19
- Issue
- 5
- Pages
- 640-648
- ISSN
- 1044-3983 (ISSN-L)
- OpenAlex
- W2150215887
- Semantic Scholar
- 0373f18622b7b64e99c7e56340ab2011dbd00b5d
- Cited by
- 1,495
- References
- 109
- Citations by year
- 2026: 91 · 2025: 87 · 2024: 78 · 2023: 112 · 2022: 94 · 2021: 99 · 2020: 103 · 2019: 131 · 2018: 108 · 2017: 101 · 2016: 87 · 2015: 77 · 2014: 72 · 2013: 60 · 2012: 50
- FWCI
- 15.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- effect size inflation, statistical significance thresholds, underpowered studies, selective reporting, conflicts of interest, reverse bias, measurement error, replication
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Humans, Sensitivity and Specificity, Models, Statistical, Linkage Disequilibrium, Molecular Epidemiology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.