Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects
Abstract
Objective To assess whether nominally statistically significant effects in meta-analyses of clinical trials are true and whether their magnitude is inflated.Study design and setting Data from the Cochrane Database of Systematic Reviews 2005 (issue 4) and 2010 (issue 1) were used. We considered meta-analyses with binary outcomes and four or more trials in 2005 with P<0.05 for the random-effects odds ratio (OR). We examined whether any of these meta-analyses had updated counterparts in 2010. We estimated the credibility (true-positive probability) under different prior assumptions and inflation in OR estimates in 2005.Results Four hundred sixty-one meta-analyses in 2005 were eligible, and 80 had additional trials included by 2010. The effect sizes (ORs) were smaller in the updating data (2005-2010) than in the respective meta-analyses in 2005 (median 0.85-fold, interquartile range [IQR]: 0.66-1.06), even more prominently for meta-analyses with less than 300 events in 2005 (median 0.67-fold, IQR: 0.54-0.96). Mean credibility of the 461 meta-analyses in 2005 was 63-84% depending on the assumptions made. Credibility estimates changed >20% in 19-31 (24-39%) of the 80 updated meta-analyses.Conclusions Most meta-analyses with nominally significant results pertain to truly nonnull effects, but exceptions are not uncommon. The magnitude of observed effects, especially in meta-analyses with limited evidence, is often inflated.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Summary effects from meta-analyses, including power estimates calculated from meta-analysis results, may be modestly inflated.
“If anything, summary effects from meta-analyses, including power estimates calculated from meta-analysis results, may also be modestly inflated 22 .”
From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [22]
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) · doi:10.1201/b16018 · Cited by 27,276 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Self-citationMoher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826 · see citing works
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Bayesian Data Analysis, 2nd edn
Brix, A. (2005) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.00347_4.x · Cited by 2,588 · see citing works
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538 · see citing works
Quantitative Synthesis in Systematic Reviews
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-9-199711010-00008 · Cited by 2,517 · cited 1× · see citing works
“The ability to arrive at more accurate treatment effects with reduced uncertainty (tight CIs) is a commonly stated advantage of metaanalysis [31,32].”
BackgroundSelf-citationPiaggio, G., Elbourne, D. R., Pocock, S. J., Evans, S. J. W., Altman, D. G., & CONSORT Group, F. T. (2012) · JAMA · doi:10.1001/jama.2012.87802 · Cited by 2,428 · see citing works
Gluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,262 · see citing works
Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis
Wetterslev, J., Thorlund, K., Brok, J., & Gluud, C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.013 · Cited by 1,881 · see citing works
Bayesian Approaches to Clinical Trials and Health‐Care Evaluation
Spiegelhalter, D. J., Abrams, K. R., & Myles, J. P. (2003) · Wiley eBooks · doi:10.1002/0470092602 · Cited by 1,675 · see citing works
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · see citing works
Self-citationContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · see citing works
Self-citationMeta-Analyses of Randomized Controlled Trials
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A., & Chalmers, T. C. (1987) · New England Journal of Medicine · doi:10.1056/nejm198702193160806 · Cited by 1,253 · see citing works
Users' guides to the medical literature : a manual for evidence-based clinical practice
Guyatt, G. H., Rennie, D., Meade, M. O., & Cook, D. J. (2002) · Cited by 1,196 · see citing works
Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,188 · see citing works
Discrepancies between Meta-Analyses and Subsequent Large Randomized, Controlled Trials
LeLorier, J., Grégoire, G., Benhaddad, A. A., Lapierre, J., & Derderian, F. (1997) · New England Journal of Medicine · doi:10.1056/nejm199708213370806 · Cited by 1,171 · see citing works
Reporting of Noninferiority and Equivalence Randomized Trials: An Extension of the CONSORT Statement
Piaggio, G., Elbourne, D. R., Altman, D. G., Pocock, S. J., Evans, S. J. W., & CONSORT Group, F. T. (2006) · JAMA · doi:10.1001/jama.295.10.1152 · Cited by 1,153 · see citing works
Patsopoulos, N. A., Εvangelou, Ε., & Ioannidis, J. P. A. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn065 · Cited by 1,147 · see citing works
Self-citationToward Evidence-Based Medical Statistics. 2: The Bayes Factor
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00019 · Cited by 913 · see citing works
Epidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891 · see citing works
Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?
Thorlund, K., Devereaux, P. J., Wetterslev, J., Guyatt, G. H., Ioannidis, J. P. A., Thabane, L., Gluud, L. L., Als-Nielsen, B. E. T., & Gluud, C. N. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn179 · Cited by 835 · see citing works
Self-citationSteyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8 · Cited by 784 · see citing works
Ioannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.4.281 · Cited by 632 · see citing works
Self-citationBassler, D. (2010) · JAMA · doi:10.1001/jama.2010.310 · Cited by 613 · see citing works
Statistical aspects of prognostic factor studies in oncology
Simon, R. M., & Altman, D. (1994) · British Journal of Cancer · doi:10.1038/bjc.1994.192 · Cited by 589 · see citing works
Molecular Dissection of Complex Traits
Lee, M. J. (1998) · Crop Science · doi:10.2135/cropsci1998.0011183x003800040039x · Cited by 576 · see citing works
Relative Citation Impact of Various Study Designs in the Health Sciences
Patsopoulos, N. A. (2005) · JAMA · doi:10.1001/jama.293.19.2362 · Cited by 498 · see citing works
Stepwise Selection in Small Data Sets A Simulation Study of Bias in Logistic Regression Analysis
Steyerberg, E. W. (1999) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00103-1 · Cited by 489 · see citing works
Large Upward Bias in Estimation of Locus-Specific Effects from Genomewide Scans
Göring, H. H. H., Terwilliger, J. D., & Blangero, J. (2001) · The American Journal of Human Genetics · doi:10.1086/324471 · Cited by 483 · see citing works
Why Current Publication Practices May Distort Science
Young, N., Ioannidis, J., & Al-Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 478 · see citing works
Self-citationWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 473 · see citing works
Self-citationReasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457 · see citing works
Self-citationJørgensen, A. W., Hilden, J., & Gøtzsche, P. C. (2006) · BMJ · doi:10.1136/bmj.38973.444699.0b · Cited by 439 · see citing works
Overcoming the limitations of current meta-analysis of randomised controlled trials
Pogue, J., & Yusuf, S. (1998) · The Lancet · doi:10.1016/s0140-6736(97)08461-4 · Cited by 428 · see citing works
Overcoming the Winner’s Curse: Estimating Penetrance Parameters from Case-Control Data
Zöllner, S. K., & Pritchard, J. K. (2007) · The American Journal of Human Genetics · doi:10.1086/512821 · Cited by 382 · see citing works
Molecular Dissection of Complex Traits
Paterson, A. H. (2010) · CRC Press eBooks · doi:10.1201/9781420049381 · Cited by 345 · see citing works
Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice
Kleinbart, J., & Williams, M. V. (2002) · ACP Journal Club · doi:10.7326/acpjc-2002-136-2-a17 · Cited by 268 · see citing works
META-ANALYSIS IN THE DESIGN AND MONITORING OF CLINICAL TRIALS
DerSimonian, R. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960630)15:12 · Cited by 257 · see citing works
The strengths and limitations of meta-analyses based on aggregate data
Lyman, G. H., & Kuderer, N. M. (2005) · BMC Medical Research Methodology · doi:10.1186/1471-2288-5-14 · Cited by 250 · see citing works
Jennions, M. D., & Møller, A. P. (2002) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2001.1832 · Cited by 250 · see citing works
Time to publication for results of clinical trials
Hopewell, S., Clarke, M. J., Stewart, L., & Tierney, J. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000011.pub2 · Cited by 243 · see citing works
Moseley, A. M., Elkins, M. R., Herbert, R. D., Maher, C. G., & Sherrington, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.09.018 · Cited by 202 · see citing works
Methods to elicit beliefs for Bayesian priors: a systematic review
Johnson, S. R., TOMLINSON, G. A., HAWKER, G. A., Granton, J. T., & Feldman, B. M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.003 · Cited by 183 · see citing works
Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time
Trikalinos, T. A., Churchill, R., Ferri, M., Leucht, S., Tuunainen, A., Wahlbeck, K., & Ioannidis, J. P. A. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.018 · Cited by 177 · see citing works
Self-citationPractical problems in interim analyses, with particular regard to estimation
Pocock, S., & Hughes, M. D. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90059-7 · Cited by 160 · see citing works
Leimu, R., & Koricheva, J. (2004) · Proceedings of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rspb.2004.2828 · Cited by 156 · see citing works
When and how to update systematic reviews
Moher, D., Tsertsvadze, A., Tricco, A. C., Eccles, M., Grimshaw, J., Sampson, M., & Barrowman, N. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000023.pub3 · Cited by 152 · see citing works
Rennie, D. (2004) · JAMA · doi:10.1001/jama.292.11.1359 · Cited by 136 · see citing works
Acetylcysteine in the Prevention of Contrast-Induced Nephropathy
Bagshaw, S. M., McAlister, F. A., Manns, B., & Ghali, W. A. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.2.161 · Cited by 134 · see citing works
Systematic reviews: when is an update an update?
Moher, D., & Tsertsvadze, A. (2006) · The Lancet · doi:10.1016/s0140-6736(06)68358-x · Cited by 110 · see citing works
Djulbegović, B. (2008) · Archives of Internal Medicine · doi:10.1001/archinte.168.6.632 · Cited by 108 · see citing works
The impact of high-risk patients on the results of clinical trials
Ioannidis, J. P. A., & Lau, J. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00149-2 · Cited by 108 · see citing works
Self-citationUpward bias in odds ratio estimates from genome‐wide association studies
Garner, C. (2007) · Genetic Epidemiology · doi:10.1002/gepi.20209 · Cited by 103 · see citing works
Moher, D., Tsertsvadze, A., Tricco, A. C., Eccles, M., Grimshaw, J., Sampson, M., & Barrowman, N. J. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.008 · Cited by 97 · see citing works
Tricco, A. C., Tetzlaff, J., Pham, B., Brehaut, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.08.008 · Cited by 93 · see citing works
The fading of reported effectiveness. A meta-analysis of randomised controlled trials
Gehr, B., Weiß, C., & Porzsolt, F. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-25 · Cited by 89 · see citing works
Evaluation of New Treatments in Radiation Oncology
Soares, H. P., Kumar, A., Daniels, S., Swann, S., Cantor, A. B., Hozo, I., Clark, M., Serdarević, F., Gwede, C. K., Trotti, A., et al. (2005) · JAMA · doi:10.1001/jama.293.8.970 · Cited by 85 · see citing works
Kumar, A., Soares, H. P., Wells, R. J., Clarke, M. J., Hozo, I., Bleyer, A., Reaman, G., Chalmers, I., & Djulbegović, B. (2005) · BMJ · doi:10.1136/bmj.38628.561123.7c · Cited by 70 · see citing works
Allison, D. B., Fernandez, J. R., Heo, M., Zhu, S., Etzel, C. J., Beasley, T. M., & Amos, C. I. (2002) · The American Journal of Human Genetics · doi:10.1086/339273 · Cited by 67 · see citing works
Effect of Formal Statistical Significance on the Credibility of Observational Associations
Ioannidis, J. P. A. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn156 · Cited by 63 · see citing works
Self-citationDiscovery Properties of Genome-wide Association Signals From Cumulatively Combined Data Sets
Pereira, T., Patsopoulos, N. A., Salanti, G., & Ioannidis, J. P. A. (2009) · American Journal of Epidemiology · doi:10.1093/aje/kwp262 · Cited by 60 · see citing works
Self-citationPublication bias in clinical trials
Hopewell, S., Dickersin, K., Clarke, M. J., Oxman, A. D., & Loudon, K. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub2 · Cited by 58 · see citing works
Invited Commentary: Evidence-based Evaluation of p Values and Bayes Factors
Katki, H. A. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn148 · Cited by 34 · see citing works
Calibration of credibility of agnostic genome‐wide associations
Ioannidis, J. P. A. (2008) · American Journal of Medical Genetics Part B Neuropsychiatric Genetics · doi:10.1002/ajmg.b.30721 · Cited by 32 · see citing works
Self-citationThe use of older studies in meta-analyses of medical interventions: a survey.
Patsopoulos, N. A., & Ioannidis, J. P. A. (2009) · PubMed Central · Cited by 30 · see citing works
Self-citationKrum, H., & Tonkin, A. (2003) · Journal of Cardiac Failure · doi:10.1054/s1071-9164(03)00018-6 · Cited by 29 · see citing works
Upward Bias in Estimation of Genetic Effects
Siegmund, D. (2002) · The American Journal of Human Genetics · doi:10.1086/343819 · Cited by 27 · see citing works
Randomized Trials Stopped Early for Benefit: A Systematic Review
Montori, V. M., Devereaux, P. J., Adhikari, N. K. J., Burns, K. E. A., Eggert, C. H., Briel, M., Lacchetti, C., Leung, T. W., Darling, E., Bryant, D. M., et al. (2005) · JAMA · doi:10.1001/jama.294.17.2203 · Cited by 26 · see citing works
Ranking Bias in Association Studies
Jeffries, N. O. (2009) · Human Heredity · doi:10.1159/000194979 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · see citing works
Self-citationIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · see citing works
Self-citationHow to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436 · see citing works
Self-citationThe Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · see citing works
“However, meta-analyses are only as good as the data they use, and a recent study estimated that up to 37% of meta-analyses of clinical trials reporting a significant mean effect size represent false positives [34].”
BackgroundTrial Sequential Analysis in systematic reviews with meta-analysis
Wetterslev, J., Jakobsen, J. C., & Gluud, C. N. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0315-7 · Cited by 1,207 · see citing works
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634 · see citing works
Self-citationNaci, H., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f5577 · Cited by 630 · see citing works
Self-citationXu, L., Freeman, G., Cowling, B. J., & Schooling, C. M. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-108 · Cited by 562 · see citing works
Goldberg, S. B., Riordan, K. M., Sun, S., & Davidson, R. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620968771 · Cited by 520 · see citing works
“…of four studies for effect sizes to be included resulted in some estimates being based on only four studies (and likely less reliable as a result; Pereira & Ioannidis, 2011) and other estimates being excluded because of an insufficient number of available studies (and therefore not represented…”
ResultAndreae, M. H., & Andreae, D. A. (2013) · British Journal of Anaesthesia · doi:10.1093/bja/aet213 · Cited by 506 · see citing works
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
Kim, M. S., Kim, W. J., Khera, A. V., Kim, J. Y., Yon, D. K., Lee, S. W., Shin, J. I., & Won, H. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab454 · Cited by 381 · see citing works
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 345 · see citing works
“Even though summary effects from meta-analyses can be inflated due to various sources of bias (8), these analyses provide the best estimates of the true effect size.”
BackgroundThorlund, K., Imberger, G. L., Walsh, M. W., Chu, R., Gluud, C. N., Wetterslev, J., Guyatt, G. H., Devereaux, P. J., & Thabane, L. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025491 · Cited by 320 · see citing works
Silverman, M. H., Jedd, K., & Luciana, M. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.07.083 · Cited by 277 · see citing works
Imberger, G. L., Thorlund, K., Gluud, C. N., & Wetterslev, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011890 · Cited by 278 · see citing works
Inhaled Cannabis for Chronic Neuropathic Pain: A Meta-analysis of Individual Patient Data
Andreae, M. H., Carter, G. M., Shaparin, N., Suslov, K., Ellis, R. J., Ware, M. A., Abrams, D. I., Prasad, H., Wilsey, B. L., Indyk, D., et al. (2015) · Journal of Pain · doi:10.1016/j.jpain.2015.07.009 · Cited by 268 · see citing works
Empirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2012.13444 · Cited by 257 · see citing works
Self-citationAssociation Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses
Dechartres, A., Altman, D. G., Trinquart, L., Boutron, I., & Ravaud, P. (2014) · JAMA · doi:10.1001/jama.2014.8166 · Cited by 233 · see citing works
Meta‐research: The art of getting it wrong
Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.19 · Cited by 231 · see citing works
Self-citationLow statistical power in biomedical science: a review of three human research domains
Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F., & Munafò, M. R. (2017) · Royal Society Open Science · doi:10.1098/rsos.160254 · Cited by 208 · see citing works
Hemkens, L. G., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2016) · BMJ · doi:10.1136/bmj.i493 · Cited by 220 · see citing works
Self-citationTrial sequential analysis: novel approach for meta-analysis
Kang, H. (2021) · Anesthesia and Pain Medicine · doi:10.17085/apm.21038 · Cited by 213 · see citing works
Naci, H., & Ioannidis, J. P. A. (2015) · British Journal of Sports Medicine · doi:10.1136/bjsports-2015-f5577rep · Cited by 206 · see citing works
Self-citationSample size and power considerations in network meta-analysis
Thorlund, K., & Mills, E. J. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-41 · Cited by 197 · see citing works
Ioannidis, J. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.03.002 · Cited by 134 · see citing works
Self-citation(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 135 · see citing works
Clarke, M., Brice, A., & Chalmers, I. (2014) · PLoS ONE · doi:10.1371/journal.pone.0102670 · Cited by 120 · see citing works
Booksmythe, I., Mautz, B. S., Davis, J., Nakagawa, S., & Jennions, M. D. (2015) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12220 · Cited by 105 · see citing works
Systematic Reviews of Anesthesiologic Interventions Reported as Statistically Significant
Imberger, G. L., Gluud, C. N., Boylan, J. F., & Wetterslev, J. (2015) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000000892 · Cited by 101 · see citing works
A Call for Examining Replication and Bias in Special Education Research
Cook, B. G. (2014) · Remedial and Special Education · doi:10.1177/0741932514528995 · Cited by 96 · see citing works
Panagiotou, O. A., Contopoulos-Ioannidis, D. G., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f707 · Cited by 89 · see citing works
Self-citationThe Necessity for Clinical Reasoning in the Era of Evidence-Based Medicine
Sniderman, A. D., Lachapelle, K. J., Rachon, N. A., & Furberg, C. D. (2013) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2013.07.012 · Cited by 96 · see citing works
Mathur, M. B., & VanderWeele, T. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1464 · Cited by 59 · see citing works
Ukai, T., Shikata, S., Takeda, H., Dawes, L., Noguchi, Y., Nakayama, T., & Takemura, Y. C. (2016) · BMC Gastroenterology · doi:10.1186/s12876-016-0453-0 · Cited by 71 · see citing works
Naci, H., & Ioannidis, J. P. A. (2014) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-010814-124614 · Cited by 68 · see citing works
Self-citationO'Connor, E. J., Gaynes, B. N., Bu, B., Williams, C., & Ep, W. (2013) · Cited by 64 · see citing works
Henssler, J. F., Bschor, T., & Baethge, C. (2016) · The Canadian Journal of Psychiatry · doi:10.1177/0706743715620411 · Cited by 64 · see citing works
London, A. J., & Kimmelman, J. (2019) · The Hastings Center Report · doi:10.1002/hast.1034 · Cited by 62 · see citing works
Chen, Y., Chaudhary, S., & Li, C. (2022) · bioRxiv · doi:10.1016/j.neuroimage.2022.119764 · Cited by 54 · see citing works
Sivakumar, H., & Peyton, P. (2016) · British Journal of Anaesthesia · doi:10.1093/bja/aew170 · Cited by 58 · see citing works
Can systematic reviews with sparse data be trusted?
Afshari, A., Wetterslev, J., & Smith, A. F. (2016) · Anaesthesia · doi:10.1111/anae.13730 · Cited by 56 · see citing works
Lamberink, H. J., Otte, W. M., Sinke, M. R. T., Lakens, D., Glasziou, P., Tijdink, J. K., & Vinkers, C. H. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.06.014 · Cited by 53 · see citing works
Spielmans, G. I., Benish, S. G., Marin, C., Bowman, W. M., Menster, M., & Wheeler, A. J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.01.008 · Cited by 53 · see citing works
Mierop, A., Mikolajczak, M., Stahl, C., Béna, J., Luminet, O., Lane, A., & Corneille, O. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620921525 · Cited by 53 · see citing works
Hengartner, M. P. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00256 · Cited by 52 · see citing works
Frost, S. A., Hou, Y. C., Lombardo, L., Metcalfe, L., Lynch, J. M., Hunt, L., Alexandrou, E., Brennan, K. J., Sanchez, D., Åneman, A., et al. (2018) · BMC Infectious Diseases · doi:10.1186/s12879-018-3521-y · Cited by 53 · see citing works
Martí-Carvajal, A. J., Ramón-Pardo, P., Javelle, É., Simon, F., Aldighieri, S., Horváth, H., Rodríguez-Abreu, J., & Revéiz, L. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179028 · Cited by 50 · see citing works
Panagiotou, O. A., & Ioannidis, J. P. A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.01.008 · Cited by 49 · see citing works
Self-citationJanus kinase-1 and Janus kinase-2 inhibitors for treating myelofibrosis
Martí-Carvajal, A. J., Anand, V., & Solà, I. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010298.pub2 · Cited by 46 · see citing works
When may systematic reviews and meta-analyses be considered reliable?
Afshari, A., & Wetterslev, J. (2015) · European Journal of Anaesthesiology · doi:10.1097/eja.0000000000000186 · Cited by 45 · see citing works
Bai, X.-L., Li, Y., Feng, Z.-F., Cao, F., Wang, D.-D., Ma, J., Yang, D., Li, D.-R., Fang, Q., Wang, Y., et al. (2025) · British Journal of Sports Medicine · doi:10.1136/bjsports-2024-109392 · Cited by 45 · see citing works
Evangelou, E., Siontis, K. C. M., Pfeiffer, T., & Ioannidis, J. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.009 · Cited by 39 · see citing works
Self-citationBogg, T., & Lasecki, L. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01589 · Cited by 41 · see citing works
Clinical judgment in psychiatry. Requiem or reveille?
Fava, G. A. (2012) · Nordic Journal of Psychiatry · doi:10.3109/08039488.2012.701665 · Cited by 38 · see citing works
Are Meta-Analyses a Form of Medical Fake News?
Packer, M. (2017) · Circulation · doi:10.1161/circulationaha.117.030209 · Cited by 38 · see citing works
Albalawi, Z., McAlister, F. A., Thorlund, K., Wong, M. M., & Wetterslev, J. (2012) · International Journal of Cardiology · doi:10.1016/j.ijcard.2012.11.048 · Cited by 36 · see citing works
Testosterone, myocardial function, and mortality
Bianchi, V. E. (2018) · Heart Failure Reviews · doi:10.1007/s10741-018-9721-0 · Cited by 36 · see citing works
Meta-analyses triggered by previous (false-)significant findings: problems and solutions
Schuit, E., Roes, K. C. B., Mol, B. W., Kwee, A., Moons, K. G. M., & Groenwold, R. H. H. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0048-9 · Cited by 35 · see citing works
Pedroza, C., Han, W., Truong, V. T. T., Green, C., & Tyson, J. E. (2015) · Statistical Methods in Medical Research · doi:10.1177/0962280215620828 · Cited by 29 · see citing works
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 30 · see citing works
Dumas-Mallet, E., Button, K., Boraud, T., Munafo, M., & Gonon, F. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158064 · Cited by 24 · see citing works
Xu, X., Liu, X., Li, J., Deng, X., Dai, T., Ji, Q., Xiong, D., & Xie, H. (2023) · Clinical Reviews in Allergy & Immunology · doi:10.1007/s12016-023-08964-2 · Cited by 23 · see citing works
Neez, E., Hwang, T. J., Sahoo, S. A., & Naci, H. (2019) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1669 · Cited by 23 · see citing works
Limitations of Medical Research and Evidence at the Patient-Clinician Encounter Scale
Morris, A. H., & Ioannidis, J. P. A. (2013) · CHEST Journal · doi:10.1378/chest.12-1908 · Cited by 22 · see citing works
Self-citationAccumulation Bias in meta-analysis: the need to consider time in error control
ter Schure, J., & Grünwald, P. (2019) · F1000Research · doi:10.12688/f1000research.19375.1 · Cited by 21 · see citing works
Hodkinson, A., & Kontopantelis, E. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211022385 · Cited by 20 · see citing works
Prostaglandins for adult liver transplanted patients
Cavalcanti, A. B., de Vasconcelos, C. P., de Oliveira, M. P., Rother, E. T., & Ferraz, L. J. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006006.pub2 · Cited by 20 · see citing works
Bikdeli, B., McAndrew, T., Crowley, A., Chen, S., Mehdipoor, G., Redfors, B., Liu, Y., Zhang, Z., Liu, M., Zhang, Y., et al. (2019) · Thrombosis and Haemostasis · doi:10.1055/s-0039-1700872 · Cited by 19 · see citing works
Meta-Analyses and Conflict of Interest
Fava, G. A. (2011) · CNS Drugs · doi:10.2165/11587940-000000000-00000 · Cited by 19 · see citing works
Anjum, R. L., Kerry, R., & Mumford, S. D. (2015) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12493 · Cited by 17 · see citing works
Burning issues in the meta-analysis of pharmaceutical trials for depression
Fountoulakis, K. Ν., Samara, M. T., & Siamouli, M. (2013) · Journal of Psychopharmacology · doi:10.1177/0269881113504014 · Cited by 17 · see citing works
Fuller, J. (2020) · Studies in History and Philosophy of Science Part C Studies in History and Philosophy of Biological and Biomedical Sciences · doi:10.1016/j.shpsc.2020.101270 · Cited by 17 · see citing works
The Unknown Prevalence of Postrandomization Bias in 15 Physical Therapy Journals: A Methods Review
Riley, S. P., Swanson, B. T., Shaffer, S. M., Sawyer, S. F., & Cleland, J. A. (2021) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2021.10491 · Cited by 16 · see citing works
Ling, X., Sun, X., Kong, H., Peng, S., Zheng, Y., Wen, J., & Yuan, B. (2021) · Frontiers in Pharmacology · doi:10.3389/fphar.2021.678631 · Cited by 16 · see citing works
Expanding our perspectives on research in musculoskeletal science and practice
Kerry, R. (2017) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2017.10.004 · Cited by 16 · see citing works
Yu, C., Carvalho, A. F., Thompson, T., Tsai, T., Tseng, P., Tu, Y., Yang, S., Yang, F., Chang, C., Hsu, C., et al. (2023) · Journal of Pineal Research · doi:10.1111/jpi.12857 · Cited by 15 · see citing works
Fountoulakis, K. Ν., McIntyre, R. S., & Carvalho, A. F. (2015) · Current Neuropharmacology · doi:10.2174/1570159x13666150630174343 · Cited by 15 · see citing works
“Transit makes you short”: On health impact assessment of transportation and the built environment
Ermagun, A., & Levinson, D. M. (2016) · Journal of Transport & Health · doi:10.1016/j.jth.2016.08.004 · Cited by 15 · see citing works
Triclosan-Containing Sutures for the Prevention of Surgical Site Infection
Jalalzadeh, H., Timmer, A. S., Buis, D. R., Dreissen, Y. E. M., Goosen, J. H. M., Graveland, H., Griekspoor, M., IJpma, F. F. A., van der Laan, M. J., Schaad, R. R., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.0306 · Cited by 14 · see citing works
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 15 · see citing works
Meta-Analyses Do Not Establish Improved Mortality With Ivermectin Use in COVID-19
Rothrock, S. G., Weber, K. D., Giordano, P. A., & Barneck, M. D. (2021) · American Journal of Therapeutics · doi:10.1097/mjt.0000000000001461 · Cited by 13 · see citing works
An umbrella review of effect size, bias, and power across meta‐analyses in emergency medicine
Parish, A. J., Yuan, D. M. K., Raggi, J. R., Omotoso, O. O., West, J. R., & Ioannidis, J. P. A. (2021) · Academic Emergency Medicine · doi:10.1111/acem.14312 · Cited by 13 · see citing works
Self-citationEpistemic Functions of Replicability in Experimental Sciences: Defending the Orthodox View
Sikorski, M., & Andreoletti, M. (2023) · Foundations of Science · doi:10.1007/s10699-023-09901-4 · Cited by 13 · see citing works
Awareness and use of systematic literature reviews and meta‐analyses by ministerial policy analysts
Bédard, P., & Ouimet, M. (2017) · Canadian Public Administration · doi:10.1111/capa.12215 · Cited by 14 · see citing works
Payne, T., Moran, B., Loadsman, J. A., Marschner, I. C., McCulloch, T. J., & Sanders, R. D. (2023) · British Journal of Anaesthesia · doi:10.1016/j.bja.2023.01.011 · Cited by 12 · see citing works
The quest for the holy volume therapy
ROBERTIS, E. D., Afshari, A., & Longrois, D. (2016) · European Journal of Anaesthesiology · doi:10.1097/eja.0000000000000441 · Cited by 12 · see citing works
Mathur, M. B., & VanderWeele, T. J. (2019) · doi:10.31219/osf.io/p3xyd · Cited by 13 · see citing works
Liu, Y., Lai, L., Wilhelm, S., Phillips, K. A., Guo, Y., Greenberg, J. L., & Ren, Z. (2024) · Psychological Medicine · doi:10.1017/s0033291724002733 · Cited by 10 · see citing works
Warraich, N., Sá, M. P., Jacquemyn, X., Kuno, T., Serna-Gallegos, D. R., & Sultan, I. S. (2025) · Journal of the American Heart Association · doi:10.1161/jaha.124.038869 · Cited by 10 · see citing works
Jacquemyn, X., Sá, M. P., Rega, F., Verbrugghe, P., Meuris, B., Serna-Gallegos, D. R., Brown, J. A., Clavel, M.-A. A., Pîbarot, P., & Sultan, I. S. (2024) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2024.04.007 · Cited by 9 · see citing works
Saunders, B., & Inzlicht, M. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.01.018 · Cited by 9 · see citing works
Towards a genetically informed approach in the social sciences: Strengths and an opportunity
Bates, T. C., & Lewis, G. J. (2012) · Personality and Individual Differences · doi:10.1016/j.paid.2012.03.002 · Cited by 9 · see citing works
Shah, S., Priyanka, P., & Sharma, S. (2022) · Indian Journal of Clinical Biochemistry · doi:10.1007/s12291-022-01091-3 · Cited by 8 · see citing works
The Rollercoaster of Paclitaxel in the Lower Limbs and Skeletons in the Closet: An Opinion Review
Katsanos, K. H., Kitrou, P. M., & Spiliοpoulos, S. (2021) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2021.03.537 · Cited by 7 · see citing works
Zanghelini, F., Zimmermann, I. R., Andrade, C. A. S., & Wu, O. (2018) · Journal of Palliative Care & Medicine · doi:10.4172/2165-7386.1000343 · Cited by 7 · see citing works
Malik, I. S., Khan, Z. I., Ahmad, K., Nadeem, M., Akhtar, S., Ashfaq, A., Faisal, M., Alatar, A. A., Munir, M., & Ullah, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-13628-6 · Cited by 6 · see citing works
Zheng, S.-W., Lin, X., Xie, N.-S., Zhang, X., Deng, F., Zou, H.-Q., Zhan, X.-L., & Tang, G.-Y. (2025) · World Journal of Diabetes · doi:10.4239/wjd.v16.i11.111578 · Cited by 8 · see citing works
Systematic Reviews With Meta-analysis of Clinical Trials: Is There Enough Evidence?
Garcia-Alamino, J. M., & Cano, M. L. (2020) · Cirugía Española (English Edition) · doi:10.1016/j.cireng.2019.12.009 · Cited by 6 · see citing works
- Authors
- Pereira, T. V., & Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Tufts University (US)
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2011-10
- Volume
- 64
- Issue
- 10
- Pages
- 1060-1069
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2004108172
- Cited by
- 160
- References
- 71
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 8 · 2023: 7 · 2022: 5 · 2021: 14 · 2020: 25 · 2019: 7 · 2018: 7 · 2017: 7 · 2016: 11 · 2015: 14 · 2014: 8 · 2013: 13 · 2012: 9
- FWCI
- 2.72
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministério da Educação
- Fundação de Amparo à Pesquisa do Estado de São Paulo
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- meta-analysis, statistical significance, effect size inflation, prior assumptions, binary outcomes, odds ratio, limited evidence, Cochrane Database of Systematic Reviews
- MeSH terms
- Clinical Trials as Topic, Health Services Research, Humans, Meta-Analysis as Topic, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
