Cited by 160Meta-analysis and systematic reviewsStatistical Methods in Clinical TrialsStatistical Methods in Epidemiology

The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.

Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects

Pereira, Tiago V., Ioannidis, John P.A.·2011·Journal of Clinical Epidemiology

DOI 10.1016/j.jclinepi.2010.12.012

Open this paper in CoChat — free account

Abstract

Objective To assess whether nominally statistically significant effects in meta-analyses of clinical trials are true and whether their magnitude is inflated.Study design and setting Data from the Cochrane Database of Systematic Reviews 2005 (issue 4) and 2010 (issue 1) were used. We considered meta-analyses with binary outcomes and four or more trials in 2005 with P<0.05 for the random-effects odds ratio (OR). We examined whether any of these meta-analyses had updated counterparts in 2010. We estimated the credibility (true-positive probability) under different prior assumptions and inflation in OR estimates in 2005.Results Four hundred sixty-one meta-analyses in 2005 were eligible, and 80 had additional trials included by 2010. The effect sizes (ORs) were smaller in the updating data (2005-2010) than in the respective meta-analyses in 2005 (median 0.85-fold, interquartile range [IQR]: 0.66-1.06), even more prominently for meta-analyses with less than 300 events in 2005 (median 0.67-fold, IQR: 0.54-0.96). Mean credibility of the 461 meta-analyses in 2005 was 63-84% depending on the assumptions made. Credibility estimates changed >20% in 19-31 (24-39%) of the 80 updated meta-analyses.Conclusions Most meta-analyses with nominally significant results pertain to truly nonnull effects, but exceptions are not uncommon. The magnitude of observed effects, especially in meta-analyses with limited evidence, is often inflated.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2010.12.012

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Bayesian Data Analysis
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citation
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Bayesian Data Analysis, 2nd edn
Brix, A. (2005) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.00347_4.x · Cited by 2,588
Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Quantitative Synthesis in Systematic Reviews
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-9-199711010-00008 · Cited by 2,517 · cited 1×
“The ability to arrive at more accurate treatment effects with reduced uncertainty (tight CIs) is a commonly stated advantage of metaanalysis [31,32].”
BackgroundSelf-citation
Reporting of Noninferiority and Equivalence Randomized Trials: Extension of the CONSORT 2010 Statement
Piaggio, G., Elbourne, D. R., Pocock, S. J., Evans, S. J. W., Altman, D. G., & CONSORT Group, F. T. (2012) · JAMA · doi:10.1001/jama.2012.87802 · Cited by 2,428
Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses
Gluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,266

Show all 71 references

Sorted by citations

Items 101–160 of 160

Association between long non-coding RNA (lncRNA) GAS5 polymorphism rs145204276 and cancer risk
Zhao, S., Liu, P., Ruan, Z., Li, J., Zeng, S., Zhong, M., & Tang, L. (2021) · Journal of International Medical Research · doi:10.1177/03000605211039798 · Cited by 5
“Because of systematic errors and random errors caused by sparse data and repetitive testing, conventional meta-analyses of cumulative trails may include false positive results (type I errors) and false negative results (type II errors).(28) Trial sequential analysis (TSA) is a tool for quantifying the statistical reli…”
Background
Subcutaneous fast-acting insulin analogues, alone or in combination with long-acting insulin, versus intravenous regular insulin infusion in patients with diabetic ketoacidosis: protocol for an updated systematic review and meta-analysis of randomised trials
Lim, B. L., Lee, W. F., Lee, B., Chung, Y. E. L., & Loo, K. V. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-070131 · Cited by 5
Colchicine for the primary prevention of cardiovascular events
Martí-Carvajal, A. J., Gemmato-Valecillos, M. A., Martín, D. M., Sanctis, J. B. D., Martí-Amarista, C. E., Hidalgo, R., Alegría‐Barrero, E., Lizardo, R. J. R., & Correa-Pérez, A. (2025) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015003.pub2 · Cited by 8
The fragility of statistically significant results from clinical nutrition randomized controlled trials
Pędziwiatr, M., Mizera, M., Wysocki, M., Małczak, P., Stefura, T., Gajewska, N., Torbicz, G., Droś, J., Kacprzyk, A., Major, P., et al. (2019) · Clinical Nutrition · doi:10.1016/j.clnu.2019.05.024 · Cited by 5
Revisiones sistemáticas con metaanálisis de ensayos clínicos: ¿es evidencia suficiente?
Garcia-Alamino, J. M., & Cano, M. L. (2019) · Cirugía Española · doi:10.1016/j.ciresp.2019.08.006 · Cited by 5
Reliability of Inference: Analogs of Replication in Qualitative Research
Fairfield, T., & Charman, A. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.012 · Cited by 6
Introduction
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.001 · Cited by 6
Replication for Quantitative Research
Jeremy Freese, A., & Peterson, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.010 · Cited by 4
Trial sequential analysis: A simple guide for judging the conclusiveness of the effect
Mheissen, S. (2024) · Journal of Orthodontics · doi:10.1177/14653125241245140 · Cited by 5
“However, a type I error may arise as a result of this approach, particularly when the analysis includes a large number of trials (Pereira and Ioannidis, 2011).”
Transparency and Reproducibility: Conceptualizing the Problem
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.006 · Cited by 4
Measurement Replication in Qualitative and Quantitative Studies
Reiter, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.011 · Cited by 4
Interleukin-receptor antagonist and tumour necrosis factor inhibitors for the primary and secondary prevention of atherosclerotic cardiovascular diseases
Martí-Carvajal, A. J., Gemmato-Valecillos, M. A., Martín, D. M., Dayer, M., Alegría‐Barrero, E., Sanctis, J. B. D., Parise-Vasco, J. M., Lizardo, R. J. R., Nicola, S., Martí-Amarista, C. E., et al. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd014741.pub2 · Cited by 3
Analysis of scientific truth status in controlled rehabilitation trials
Kerry, R., Madouasse, A., Arthur, A. J., & Mumford, S. D. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2012.01855.x · Cited by 4
Comment lire une méta-analyse ?
Moulis, G., & Sommet, A. (2013) · La Revue de Médecine Interne · doi:10.1016/j.revmed.2013.07.011 · Cited by 5
Introduction to Intervention Research
Sidani, S., & Braden, C. J. (2011) · Wiley eBooks · doi:10.1002/9781118785553.ch1 · Cited by 4
Statistical power analysis in exercise science
Uehata, A., & Rein, R. (2025) · Journal of Sports Sciences · doi:10.1080/02640414.2025.2571841 · Cited by 5
“The validity of this approach is supported by evidence that pairs of meta-analyses published five years apart on the same topic consistently report very similar effect size (Pereira & Ioannidis, 2011).”
Decisive gene strategy on osteoarthritis: a comprehensive whole-literature based approach for conclusive gene targets
Chen, Y., Wang, Y.-C., Lee, M.-C., Chen, Y.-H., Su, W., Ko, P.-S., Chen, C., & Su, S. (2024) · Aging · doi:10.18632/aging.206094 · Cited by 3
Decisive gene strategy on osteoporosis: a comprehensive whole-literature-based approach for conclusive candidate gene targets
Chen, Y.-C., Tsai, Y.-J., Wang, C., Ko, P.-S., Su, W., & Su, S. (2022) · Aging · doi:10.18632/aging.204026 · Cited by 3
Unveiling Statistical Significance of Online Regression Over Multiple Datasets
Abu-Shaira, M., & Shi, W. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00051 · Cited by 3
“Statistical significance not only validates the significance of predictor variables in explaining the variability of the response variable but also aids in discerning spurious relationships from genuine ones [2].”
Trial Sequential Analysis and Updated Meta-Analysis of Fluvoxamine on Clinical Deterioration in Adult Patients with Symptomatic COVID-19 Infection
Yu, C., Carvalho, A. F., Thompson, T., Tsai, T., Tseng, P., Hsu, C., Tu, Y., Yang, S., Hsu, T., Yeh, T., et al. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20054088 · Cited by 3
“However, statistically significant meta-analyses that have a few trials or participants have poor credibility, and the intervention effects may be spuriously over(type I error) or underestimated (type II error) [11].”
Background
The effect of prone position for ventilator-associated pneumonia in adult patients: a systematic review and meta-analysis
Zhu, X., Lu, Z., Xiao, W., Zhang, J., Di, J., & Yang, M. (2021) · Emergency and Critical Care Medicine · doi:10.1097/ec9.0000000000000006 · Cited by 3
“However, the risk of type I error (false-positive result) from repetitive statistical testing could increase when the overall sample size is not sufficient.([15]) In order to reduce this risk, the P values and corresponding confidence interval in each statistical testing need to be repeatedly corrected, and analogous…”
Method
Multivariate and correlation analysis of cadmium contamination in the food chain across spatio-temporal scales
Raza, H., Tahir, M. A., Sabah, N. U., Iftikhar, R., Rashid, M., Alatar, A. A., & Faisal, M. (2026) · Agricultural Water Management · doi:10.1016/j.agwat.2026.110255 · Cited by 2
Transparency and Reproducibility: Potential Solutions
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.007 · Cited by 2
Transparent Science
Mellor, D. T., Vazire, S., & Lindsay, D. S. (2018) · Cambridge University Press eBooks · doi:10.1017/9781108304443.020 · Cited by 2
Impact Metrics
Gerring, J., Karcher, S., & Apfeld, B. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.015 · Cited by 3
Einführung: Forschungs- und Innovationspotenziale in den Gesundheitsfachberufen
Voigt-Radloff, S., Lang, B., & Antes, G. (2014) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2014.09.011 · Cited by 2
What’s Wrong with Replicating the Old Boys’ Networks?
Teele, D. L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.016 · Cited by 2
Coordinating Reappraisals
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.013 · Cited by 2
Research Cycles
Lieberman, E. S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.003 · Cited by 2
Insufficient information size and potentially false results in orthodontic meta-analyses using trial sequential analysis
Mheissen, S., Aldandan, M., & Khan, H. (2023) · European Journal of Orthodontics · doi:10.1093/ejo/cjad049 · Cited by 2
Prophylactic antibiotics in laparoscopic cholecystectomy: Reducing postoperative infection risk–A meta-analysis and trial sequential analysis
Wang, S., Gao, Q., Qi, X.-G., Hong, L., & Huang, H. (2025) · Heliyon · doi:10.1016/j.heliyon.2025.e41982 · Cited by 1
“The repeated significance tests can lead to both type I errors (overestimated) and type II errors (underestimated) [21].”
Absence of a meaningful effect of intranasal oxytocin on trusting behavior: a registered report with pooled equivalence testing
Kroll, C., Schruers, K. R. J., Viechtbauer, W., Vingerhoets, C., Seidel, L., Riedl, A. M., & Hernaus, D. (2026) · Cortex · doi:10.1016/j.cortex.2026.03.006 · Cited by 1
Cerebral Embolic Protection Devices (CEPDs) During Transcatheter Aortic Valve Implantation (TAVI): A Meta‐Analysis of Randomized Controlled Trials
Gbreel, M. I., Badran, A., Hassan, M. N. M., & Balata, M. (2026) · Clinical Cardiology · doi:10.1002/clc.70309 · Cited by 1
“We performed Trial Sequential Analysis (TSA) for the primary outcomes, using TSA software (Version 0.9.5.10 Beta, Copenhagen Trial Unit, Copenhagen) [15] to avoid false‐ positive results and to draw conclusive evidence [16].”
Influential
Trial sequential analysis in anesthesiology and intensive care: a systematic review
Yadgarov, M. Y., Бондоевич, Б. Л., Polyakov, P. A., Kadantseva, K., Smirnova, A., Kuznetsov, I. V., Yakovlev, A. А., & Likhvantsev, V. V. (2024) · Patologiya krovoobrashcheniya i kardiokhirurgiya · doi:10.21688/1681-3472-2024-1-26-40
Subcutaneous fast-acting insulin analogues ± long-acting insulin vs IV insulin infusion in DKA: updated meta-analysis of randomised trials
Lim, B. L., Lee, W. F., Chung, Y. E. L., Lee, B., & Loo, K. V. (2024) · Endocrine · doi:10.1007/s12020-024-04071-0 · Cited by 2
Efficacy and safety of Pueraria lobata radix-containing traditional Chinese medicine in the treatment of essential hypertension: a systematic review
Deng, X., Xue, Q., Feng, X., Wang, H. C., Luo, P., Tian, Y., Zhang, R., Zhang, G., Yan, D., & Zhou, X. (2025) · Journal of Herbal Medicine · doi:10.1016/j.hermed.2025.101010 · Cited by 1
Proposals
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.018 · Cited by 1
Length Limits
Gerring, J., & Cojocaru, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.005 · Cited by 1
Comprehensive Appraisal
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.014 · Cited by 1
Ideological Diversity
Gross, N., & Robertson, C. T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.017 · Cited by 1
Multiplicity and sparse data in systematic reviews of anaesthesiological interventions cause increased risk of random errors and lack of reliability of conclusions
Imberger, G. L. (2014) · Cited by 1
Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological SciencePreprint
Nuijten, M. B. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/qtk7e · Cited by 1
Antibiotic Prophylaxis with Quinolones in Afebrile Neutropenic Patients: Is there Enough Evidence for Mortality Effect?
Cota, G., Fereguetti, T., & de Sousa, M. R. (2012) · The Open Infectious Diseases Journal · doi:10.2174/1874279301206010015
“Pereira and Ioannidis [23] evaluated 461 meta-analyses of clinical trials on diverse interventions.”
Background
Contributions to estimation and interpretation of intervention effects and heterogeneity in meta-analysis
Thorlund, K. (2011) · doi:10.71548/21019
testosterone, cardiovascular, men, trial
Xu, L., Freeman, G. H., Cowling, B. J., & Schooling, M. (2013) · CUNY Academic Works (City University of New York)
How to read a scientific article?
de Sousa, M. R., & Cota, G. (2016) · Revista Médica de Minas Gerais · doi:10.5935/2238-3182.20160081
Making the most of a few small population clinical trials
Pateras, K. (2020) · doi:10.33540/53 · Cited by 1
Adolescents' Decisions to be (In)Active: Relationships between Automatic Associations, Affective Reactions, and Deliberate Reflections
McFadden, K. (2021) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3-32yn-1m82
Peer Review
Liao, T. F. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.004
Index
Elman, C., Gerring, J., & Mahoney, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.019
Introduction to Intervention Research
(2021) · Wiley eBooks · doi:10.1002/9781119610113.ch1
Effect of Fluvoxamine on Clinical Deterioration in Adult Patients with Symptomatic COVID-19 Infection: A Trial Sequential AnalysisPreprint
Yu, C., Carvalho, A. F., Thompson, T., Tsai, T., Tseng, P., Hsu, C., Tu, Y., Yang, S., Hsu, T., & Liang, C. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4184790
Letter to the Editor Regarding “Radiosurgery for Benign Vertebral Body Hemangiomas of the Spine: A Systematic Review and Meta-Analysis”
Xiao, J., & Hao, L. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2022.06.069
The trials of interpreting clinical trials - A Bayesian perspective Colchicine in secondary cardiovascular preventionPreprint
Brophy, J. M. (2025) · medRxiv · doi:10.1101/2025.05.04.25326946
The Trials of Interpreting Clinical Trials—A Bayesian Perspective on Colchicine After an Acute Coronary Syndrome
Brophy, J. M. (2025) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2025.06.008
Statistical Power in Musculoskeletal Research: A Meta-Review of 266 Randomised Controlled Trials
Bleakley, C., Klempel, N., Wagemans, J., Netterström-Wedin, F., & Smoliga, J. M. (2025) · Sports Medicine - Open · doi:10.1186/s40798-025-00908-8 · Cited by 2
“However, the random effects model assigns more weight to smaller studies, a property that has been criticized due to the tendency of smaller studies to report inflated effect sizes [38].”
Intravascular imaging-guided versus coronary angiography-guided percutaneous coronary intervention: Meta-analysis and trial sequential analysis of randomized controlled trials
Li, X., Zhong, D., Yin, Z. H. I. Y. I., Liu, H., Xie, W., Liao, Z., Tian, J., Kong, L., Chen, K., & Wang, C. (2026) · Medicine · doi:10.1097/md.0000000000047193
“[13] Trial sequential analysis (TSA) can reduce the risk of these random errors.”
The Statistical Tests Used in the Literature on Conditioned Place Preference Induced by Nicotine in Mice Are Underpowered
Léonard, F., & Tirelli, E. (2026) · Journal of Trial and Error · doi:10.36850/34b0-42b1 · Cited by 1
Toward Context‐Appropriate Feed Processing in Aquaculture: Evidence From a Meta‐Analysis of Extrusion and Pelleting
Kovac, Z., Bayeto, S. G., Nole, B. E. C., Donayre, B. A. D., Lopez, M. S. T., & Espinoza, C. (2026) · Aquaculture Research · doi:10.1155/are/5476546
Anti‐inflammatory agents in atherosclerosis—and a need for reform: Extraordinary claims require extraordinary evidence
Williams, K. J. (2026) · Journal of Internal Medicine · doi:10.1111/joim.70128

Authors: Pereira, T. V., & Ioannidis, J. P. A.
Affiliations: University of Ioannina (GR)
Tufts University (US)
ORCID: Pereira, Tiago V.
Ioannidis, John P.A.
Venue: Journal of Clinical Epidemiology
Publisher: Elsevier BV
Published: 2011-10
Volume: 64
Issue: 10
Pages: 1060-1069
ISSN: 0895-4356 (ISSN-L)
DOI: 10.1016/j.jclinepi.2010.12.012
OpenAlex: W2004108172
Cited by: 160
References: 71
Citations by year: 2026: 7 · 2025: 12 · 2024: 8 · 2023: 7 · 2022: 5 · 2021: 14 · 2020: 25 · 2019: 7 · 2018: 7 · 2017: 7 · 2016: 11 · 2015: 14 · 2014: 8 · 2013: 13 · 2012: 9
FWCI: 2.72
Type: journal article
Language: en
Open access: closed
Licence: www.elsevier.com/tdm/userlicense/1.0/ (tdm)
www.elsevier.com/legal/tdmrep-license (tdm)
Locations: Journal of Clinical Epidemiology (publishedVersion)
PubMed (publishedVersion)
Funders: Ministério da Educação
Fundação de Amparo à Pesquisa do Estado de São Paulo
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords: meta-analysis, statistical significance, effect size inflation, prior assumptions, binary outcomes, odds ratio, limited evidence, Cochrane Database of Systematic Reviews
MeSH terms: Clinical Trials as Topic, Health Services Research, Humans, Meta-Analysis as Topic, Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks