The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects
Abstract
Objective To assess whether nominally statistically significant effects in meta-analyses of clinical trials are true and whether their magnitude is inflated.Study design and setting Data from the Cochrane Database of Systematic Reviews 2005 (issue 4) and 2010 (issue 1) were used. We considered meta-analyses with binary outcomes and four or more trials in 2005 with P<0.05 for the random-effects odds ratio (OR). We examined whether any of these meta-analyses had updated counterparts in 2010. We estimated the credibility (true-positive probability) under different prior assumptions and inflation in OR estimates in 2005.Results Four hundred sixty-one meta-analyses in 2005 were eligible, and 80 had additional trials included by 2010. The effect sizes (ORs) were smaller in the updating data (2005-2010) than in the respective meta-analyses in 2005 (median 0.85-fold, interquartile range [IQR]: 0.66-1.06), even more prominently for meta-analyses with less than 300 events in 2005 (median 0.67-fold, IQR: 0.54-0.96). Mean credibility of the 461 meta-analyses in 2005 was 63-84% depending on the assumptions made. Credibility estimates changed >20% in 19-31 (24-39%) of the 80 updated meta-analyses.Conclusions Most meta-analyses with nominally significant results pertain to truly nonnull effects, but exceptions are not uncommon. The magnitude of observed effects, especially in meta-analyses with limited evidence, is often inflated.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“If anything, summary effects from meta-analyses, including power estimates calculated from meta-analysis results, may also be modestly inflated 22 .”
From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [22]
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationMoher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Bayesian Data Analysis, 2nd edn
Brix, A. (2005) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.00347_4.x · Cited by 2,588
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Quantitative Synthesis in Systematic Reviews
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-9-199711010-00008 · Cited by 2,517 · cited 1×
“The ability to arrive at more accurate treatment effects with reduced uncertainty (tight CIs) is a commonly stated advantage of metaanalysis [31,32].”
BackgroundSelf-citationPiaggio, G., Elbourne, D. R., Pocock, S. J., Evans, S. J. W., Altman, D. G., & CONSORT Group, F. T. (2012) · JAMA · doi:10.1001/jama.2012.87802 · Cited by 2,428
Gluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,266
Cited by
Sorted by citations
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationHow to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436
Self-citationThe Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
“However, meta-analyses are only as good as the data they use, and a recent study estimated that up to 37% of meta-analyses of clinical trials reporting a significant mean effect size represent false positives [34].”
BackgroundTrial Sequential Analysis in systematic reviews with meta-analysis
Wetterslev, J., Jakobsen, J. C., & Gluud, C. N. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0315-7 · Cited by 1,216
“later [2]; the estimated intervention effects, when further trials are included in the meta-analysis update, become considerably lower than previously estimated [2].”
BackgroundInfluentialThe Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
“Its results may have only modest credibility, especially when they come from largely small and/or biased studies (Pereira and Ioannidis, 2011).”
BackgroundSelf-citationNaci, H., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f5577 · Cited by 630
Self-citationXu, L., Freeman, G., Cowling, B. J., & Schooling, C. M. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-108 · Cited by 569
“Metaanalysis may overstate the benefits of treatment [98] however, they are less prone to overstate the harms [98].”
Goldberg, S. B., Riordan, K. M., Sun, S., & Davidson, R. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620968771 · Cited by 520
“…of four studies for effect sizes to be included resulted in some estimates being based on only four studies (and likely less reliable as a result; Pereira & Ioannidis, 2011) and other estimates being excluded because of an insufficient number of available studies (and therefore not represented…”
ResultAndreae, M. H., & Andreae, D. A. (2013) · British Journal of Anaesthesia · doi:10.1093/bja/aet213 · Cited by 512
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 471
Association between Adiposity and Cardiovascular Outcomes: An Umbrella Review and Meta-Analysis
Kim, M. S., Kim, W. J., Khera, A. V., Kim, J. Y., Yon, D. K., Lee, S. W., Shin, J. I., & Won, H. (2020) · European Heart Journal · doi:10.1093/eurheartj/ehab454 · Cited by 400
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 347
“Even though summary effects from meta-analyses can be inflated due to various sources of bias (8), these analyses provide the best estimates of the true effect size.”
BackgroundThorlund, K., Imberger, G. L., Walsh, M. W., Chu, R., Gluud, C. N., Wetterslev, J., Guyatt, G. H., Devereaux, P. J., & Thabane, L. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025491 · Cited by 320
“As demonstrated in empirical studies, early large intervention effects are likely to dissipate and early statistically significant meta-analyses are likely to be false positives [4– 6,12,21].”
Silverman, M. H., Jedd, K., & Luciana, M. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.07.083 · Cited by 277
“While the field of fMRI research is moving toward larger samples, small sample sizes can impact meta-analytic findings due to the skewed nature of the sample of studies included (Pereira and Ioannidis, 2011).”
Imberger, G. L., Thorlund, K., Gluud, C. N., & Wetterslev, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011890 · Cited by 278
“This increased risk has been demonstrated by theoretical arguments,(8) 9 evidence from simulation studies,(2) 3 10–12 and evidence from empirical work.(13) Given that so many published Cochrane meta-analyses are underpowered and subject to continued updating, this Strengths and limitations of this study”
BackgroundInhaled Cannabis for Chronic Neuropathic Pain: A Meta-analysis of Individual Patient Data
Andreae, M. H., Carter, G. M., Shaparin, N., Suslov, K., Ellis, R. J., Ware, M. A., Abrams, D. I., Prasad, H., Wilsey, B. L., Indyk, D., et al. (2015) · Journal of Pain · doi:10.1016/j.jpain.2015.07.009 · Cited by 268
Empirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2012.13444 · Cited by 257
Self-citationAssociation Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses
Dechartres, A., Altman, D. G., Trinquart, L., Boutron, I., & Ravaud, P. (2014) · JAMA · doi:10.1001/jama.2014.8166 · Cited by 233
Meta‐research: The art of getting it wrong
Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.19 · Cited by 237
“…evaluation of medical meta-analyses that have been updated over a time window of 5 years shows indeed that the statistically significant treatment effects are modestly inflated, and the inflation is large when the significant summary effect is based on limited data (Pereira and Ioannidis, 2011).”
BackgroundSelf-citationLow statistical power in biomedical science: a review of three human research domains
Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F., & Munafò, M. R. (2017) · Royal Society Open Science · doi:10.1098/rsos.160254 · Cited by 208
“The validity of this approach is supported by evidence that pairs of meta-analyses published on the same topic at an interval of 5 years report very similar effect sizes [12].”
BackgroundResultHemkens, L. G., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2016) · BMJ · doi:10.1136/bmj.i493 · Cited by 220
Self-citationTrial sequential analysis: novel approach for meta-analysis
Kang, H. (2021) · Anesthesia and Pain Medicine · doi:10.17085/apm.21038 · Cited by 213
“Similarly, results from meta-analyses with too few studies and patients are assumed to have an increased likelihood of overestimation or underestimation due to lack of precision and power in the intervention effect [10,17].”
BackgroundResultNaci, H., & Ioannidis, J. P. A. (2015) · British Journal of Sports Medicine · doi:10.1136/bjsports-2015-f5577rep · Cited by 215
Self-citationSample size and power considerations in network meta-analysis
Thorlund, K., & Mills, E. J. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-41 · Cited by 197
“, precision) [14,15,18-24,26,27], and sound recommendations have been provided [16,22,28].”
BackgroundIoannidis, J. P. A. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.03.002 · Cited by 149
Self-citation(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 154
Clarke, M., Brice, A., & Chalmers, I. (2014) · PLoS ONE · doi:10.1371/journal.pone.0102670 · Cited by 120
“As a result, early studies and meta-analyses of them may tend to yield inflated estimates of effects and this needs to be taken into account in considering proposals for additional studies [66].”
BackgroundBooksmythe, I., Mautz, B. S., Davis, J., Nakagawa, S., & Jennions, M. D. (2015) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12220 · Cited by 104
Systematic Reviews of Anesthesiologic Interventions Reported as Statistically Significant
Imberger, G. L., Gluud, C. N., Boylan, J. F., & Wetterslev, J. (2015) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000000892 · Cited by 102
“77 No No (19) No No 5 5 Cunningham et al.”
BackgroundA Call for Examining Replication and Bias in Special Education Research
Cook, B. G. (2014) · Remedial and Special Education · doi:10.1177/0741932514528995 · Cited by 100
“Furthermore, Pereira and Ioannidis (2011) reported that among 80 meta-analyses with significant findings that were updated within a 5-year span, the original reviews reported significantly higher effects than subsequent meta-analyses that included additional studies.”
BackgroundPanagiotou, O. A., Contopoulos-Ioannidis, D. G., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f707 · Cited by 89
Self-citationThe Necessity for Clinical Reasoning in the Era of Evidence-Based Medicine
Sniderman, A. D., Lachapelle, K. J., Rachon, N. A., & Furberg, C. D. (2013) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2013.07.012 · Cited by 99
Mathur, M. B., & VanderWeele, T. J. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1464 · Cited by 92
Ukai, T., Shikata, S., Takeda, H., Dawes, L., Noguchi, Y., Nakayama, T., & Takemura, Y. C. (2016) · BMC Gastroenterology · doi:10.1186/s12876-016-0453-0 · Cited by 71
“Early trials tend to overestimate treatment effects for a variety of reasons, such as the under-reporting of disappointing results or the selection of favorable subgroups [22–24].”
BackgroundNaci, H., & Ioannidis, J. P. A. (2014) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-010814-124614 · Cited by 70
Self-citationO'Connor, E. J., Gaynes, B. N., Bu, B., Williams, C., & Ep, W. (2013) · Cited by 64
Henssler, J. F., Bschor, T., & Baethge, C. (2016) · The Canadian Journal of Psychiatry · doi:10.1177/0706743715620411 · Cited by 66
London, A. J., & Kimmelman, J. (2019) · The Hastings Center Report · doi:10.1002/hast.1034 · Cited by 62
Chen, Y., Chaudhary, S., & Li, C. R. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119764 · Cited by 62
Sivakumar, H., & Peyton, P. (2016) · British Journal of Anaesthesia · doi:10.1093/bja/aew170 · Cited by 57
Can systematic reviews with sparse data be trusted?
Afshari, A., Wetterslev, J., & Smith, A. F. (2016) · Anaesthesia · doi:10.1111/anae.13730 · Cited by 59
Lamberink, H. J., Otte, W. M., Sinke, M. R. T., Lakens, D., Glasziou, P., Tijdink, J. K., & Vinkers, C. H. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.06.014 · Cited by 55
Spielmans, G. I., Benish, S. G., Marin, C., Bowman, W. M., Menster, M., & Wheeler, A. J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.01.008 · Cited by 53
Mierop, A., Mikolajczak, M., Stahl, C., Béna, J., Luminet, O., Lane, A., & Corneille, O. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620921525 · Cited by 53
“In addition, as we discussed, if effect sizes estimated in previous meta-analyses reflected true effect sizes, these effect sizes are likely to be inflated because of various sources of bias and their reliance on underpowered studies (Pereira & Ioannidis, 2011; Walum et al., 2016).”
BackgroundHengartner, M. P. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00256 · Cited by 57
“Thus, what shall we tell our patients: is longterm psychoanalysis empirically supported or would they fare better (or at least as good) with a short-term therapy such as CBT? However, that may be, clinicians and researcher should be aware that the credibility of many meta-analyses is rather modest (Pereira and Ioannid…”
BackgroundFrost, S. A., Hou, Y. C., Lombardo, L., Metcalfe, L., Lynch, J. M., Hunt, L., Alexandrou, E., Brennan, K. J., Sanchez, D., Åneman, A., et al. (2018) · BMC Infectious Diseases · doi:10.1186/s12879-018-3521-y · Cited by 53
Martí-Carvajal, A. J., Ramón-Pardo, P., Javelle, É., Simon, F., Aldighieri, S., Horváth, H., Rodríguez-Abreu, J., & Revéiz, L. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179028 · Cited by 54
Panagiotou, O. A., & Ioannidis, J. P. A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.01.008 · Cited by 51
“Often some summary effect is nominally statistically significant, but the data are very heterogeneous across the included studies and/or there are hints for bias [44].”
BackgroundSelf-citationJanus kinase-1 and Janus kinase-2 inhibitors for treating myelofibrosis
Martí-Carvajal, A. J., Anand, V., & Solà, I. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010298.pub2 · Cited by 46
When may systematic reviews and meta-analyses be considered reliable?
Afshari, A., & Wetterslev, J. (2015) · European Journal of Anaesthesiology · doi:10.1097/eja.0000000000000186 · Cited by 45
Bai, X.-L., Li, Y., Feng, Z.-F., Cao, F., Wang, D.-D., Ma, J., Yang, D., Li, D.-R., Fang, Q., Wang, Y., et al. (2025) · British Journal of Sports Medicine · doi:10.1136/bjsports-2024-109392 · Cited by 45
Εvangelou, Ε., Siontis, K., Pfeiffer, T., & Ioannidis, J. P. A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.009 · Cited by 40
Self-citationBogg, T., & Lasecki, L. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01589 · Cited by 41
“As others have examined elsewhere, the early returns of meta-analytic efforts are often inflated or lack credibility (Pereira and Ioannidis, 2011).”
BackgroundClinical judgment in psychiatry. Requiem or reveille?
Fava, G. A. (2012) · Nordic Journal of Psychiatry · doi:10.3109/08039488.2012.701665 · Cited by 38
Are Meta-Analyses a Form of Medical Fake News?
Packer, M. (2017) · Circulation · doi:10.1161/circulationaha.117.030209 · Cited by 38
Albalawi, Z., McAlister, F. A., Thorlund, K., Wong, M. M., & Wetterslev, J. (2012) · International Journal of Cardiology · doi:10.1016/j.ijcard.2012.11.048 · Cited by 36
Testosterone, myocardial function, and mortality
Bianchi, V. E. (2018) · Heart Failure Reviews · doi:10.1007/s10741-018-9721-0 · Cited by 37
Meta-analyses triggered by previous (false-)significant findings: problems and solutions
Schuit, E., Roes, K. C. B., Mol, B. W., Kwee, A., Moons, K. G. M., & Groenwold, R. H. H. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0048-9 · Cited by 35
“Previously, Pereira and Ioannidis found that most meta-analyses with significant findings pertain to truly non-null-effects, but that exceptions are not uncommon (that is, more common than the expected 5%) [5].”
BackgroundPedroza, C., Han, W., Truong, V. T. T., Green, C., & Tyson, J. E. (2015) · Statistical Methods in Medical Research · doi:10.1177/0962280215620828 · Cited by 30
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 30
Dumas-Mallet, E., Button, K. S., Boraud, T., Munafo, M. R., & Gonon, F. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158064 · Cited by 30
“Indeed, a comparison between pairs of meta-analyses published at a five-years interval about the same association showed that the change in summary effect size was inversely related to the number of datasets [20].”
BackgroundResultXu, X., Liu, X., Li, J., Deng, X., Dai, T., Ji, Q., Xiong, D., & Xie, H. (2023) · Clinical Reviews in Allergy & Immunology · doi:10.1007/s12016-023-08964-2 · Cited by 29
“However, the thematic analysis of these meta-analyses is often single, and meta-analyses estimates of mean effect sizes may be imprecise, which may overestimate effect sizes, especially if the meta-analysis is based on only a few studies, and the size of observed effects, especially in meta-analyses with limited evide…”
BackgroundNeez, E., Hwang, T. J., Sahoo, S. A., & Naci, H. (2019) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1669 · Cited by 23
Limitations of Medical Research and Evidence at the Patient-Clinician Encounter Scale
Morris, A. H., & Ioannidis, J. P. A. (2013) · CHEST Journal · doi:10.1378/chest.12-1908 · Cited by 23
Self-citationAccumulation Bias in meta-analysis: the need to consider time in error control
ter Schure, J., & Grünwald, P. (2019) · F1000Research · doi:10.12688/f1000research.19375.1 · Cited by 27
“…that early studies are unreliable predictors of later replications in a study series (Roberts and Ker, 2015; Chalmers and Glasziou, 2016) (and see references 6-34 in Ioannidis (2008) and references 33-49 in Pereira and Ioannidis (2011)), which is also an indication of early study Accumulation Bias.”
Hodkinson, A., & Kontopantelis, E. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211022385 · Cited by 20
Prostaglandins for adult liver transplanted patients
Cavalcanti, A. B., de Vasconcelos, C. P., de Oliveira, M. P., Rother, E. T., & Ferraz, L. J. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006006.pub2 · Cited by 20
Bikdeli, B., McAndrew, T., Crowley, A., Chen, S., Mehdipoor, G., Redfors, B., Liu, Y., Zhang, Z., Liu, M., Zhang, Y., et al. (2019) · Thrombosis and Haemostasis · doi:10.1055/s-0039-1700872 · Cited by 19
Meta-Analyses and Conflict of Interest
Fava, G. A. (2011) · CNS Drugs · doi:10.2165/11587940-000000000-00000 · Cited by 19
“As often happens with innovations, the weight given to meta-analyses is likely to be reconsidered in due course, for example, statistically significant meta-analyses of clinical trials have been found to have modest credibility and inflated effects.([25]) Further, there is the need for a reassessment of journals’ poli…”
Anjum, R. L., Kerry, R., & Mumford, S. D. (2015) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12493 · Cited by 17
Burning issues in the meta-analysis of pharmaceutical trials for depression
Fountoulakis, K. Ν., Samara, M. T., & Siamouli, M. (2013) · Journal of Psychopharmacology · doi:10.1177/0269881113504014 · Cited by 17
Fuller, J. (2020) · Studies in History and Philosophy of Science Part C Studies in History and Philosophy of Biological and Biomedical Sciences · doi:10.1016/j.shpsc.2020.101270 · Cited by 17
The Unknown Prevalence of Postrandomization Bias in 15 Physical Therapy Journals: A Methods Review
Riley, S. P., Swanson, B. T., Shaffer, S. M., Sawyer, S. F., & Cleland, J. A. (2021) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2021.10491 · Cited by 16
Ling, X., Sun, X., Kong, H., Peng, S., Zheng, Y., Wen, J., & Yuan, B. (2021) · Frontiers in Pharmacology · doi:10.3389/fphar.2021.678631 · Cited by 16
“Some positive findings from meta-analysis might be caused by a random error (by chance) rather than the true effects of the intervention and these results might often increase the likelihood of overestimation (Type I errors) or underestimation (Type II errors) (Pereira and Ioannidis, 2011; Afshari et al., 2017), which…”
BackgroundExpanding our perspectives on research in musculoskeletal science and practice
Kerry, R. (2017) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2017.10.004 · Cited by 16
Yu, C., Carvalho, A. F., Thompson, T., Tsai, T., Tseng, P., Tu, Y., Yang, S., Yang, F., Chang, C., Hsu, C., et al. (2023) · Journal of Pineal Research · doi:10.1111/jpi.12857 · Cited by 15
“Importantly, the results of meta‐analyses with few trials or participants have poor credibility with possible type I errors (overestimated) and type II errors (underestimated).(12) Trial sequential analysis (TSA) is a statistical method to control for type I and II errors in systematic reviews and meta‐analyses and es…”
BackgroundFountoulakis, K. Ν., McIntyre, R. S., & Carvalho, A. F. (2015) · Current Neuropharmacology · doi:10.2174/1570159x13666150630174343 · Cited by 17
“, heterogeneity and discrepant methodologies across included trials) often lead to erroneous meta-analytic results and interpretations [4, 6, 7].”
Background“Transit makes you short”: On health impact assessment of transportation and the built environment
Ermagun, A., & Levinson, D. M. (2016) · Journal of Transport & Health · doi:10.1016/j.jth.2016.08.004 · Cited by 15
“This destructive trend in science has triggered the “ p -hacking” phenomena to the extent that Pereira and Ioannidis (2011) indicated that 16-37 percent of the statistically significant meta-analyses are false positive.”
InfluentialTriclosan-Containing Sutures for the Prevention of Surgical Site Infection
Jalalzadeh, H., Timmer, A. S., Buis, D. R., Dreissen, Y. E. M., Goosen, J. H. M., Graveland, H., Griekspoor, M., IJpma, F. F. A., van der Laan, M. J., Schaad, R. R., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.0306 · Cited by 14
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 15
Meta-Analyses Do Not Establish Improved Mortality With Ivermectin Use in COVID-19
Rothrock, S. G., Weber, K. D., Giordano, P. A., & Barneck, M. D. (2021) · American Journal of Therapeutics · doi:10.1097/mjt.0000000000001461 · Cited by 13
An umbrella review of effect size, bias, and power across meta‐analyses in emergency medicine
Parish, A. J., Yuan, D. M. K., Raggi, J. R., Omotoso, O. O., West, J. R., & Ioannidis, J. P. A. (2021) · Academic Emergency Medicine · doi:10.1111/acem.14312 · Cited by 15
Self-citationEpistemic Functions of Replicability in Experimental Sciences: Defending the Orthodox View
Sikorski, M., & Andreoletti, M. (2023) · Foundations of Science · doi:10.1007/s10699-023-09901-4 · Cited by 16
“Direct replication is especially useful if the original experiment is underpowered and therefore likely to deliver an inflated effect (see e.g., Ioannidis, 2005, 2008; Pereira & Ioannidis, 2011).”
BackgroundAwareness and use of systematic literature reviews and meta‐analyses by ministerial policy analysts
Bédard, P., & Ouimet, M. (2017) · Canadian Public Administration · doi:10.1111/capa.12215 · Cited by 17
Payne, T., Moran, B., Loadsman, J. A., Marschner, I. C., McCulloch, T. J., & Sanders, R. D. (2023) · British Journal of Anaesthesia · doi:10.1016/j.bja.2023.01.011 · Cited by 12
The quest for the holy volume therapy
ROBERTIS, E. D., Afshari, A., & Longrois, D. (2016) · European Journal of Anaesthesiology · doi:10.1097/eja.0000000000000441 · Cited by 12
Mathur, M. B., & VanderWeele, T. J. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/p3xyd · Cited by 13
Liu, Y., Lai, L., Wilhelm, S., Phillips, K. A., Guo, Y., Greenberg, J. L., & Ren, Z. (2024) · Psychological Medicine · doi:10.1017/s0033291724002733 · Cited by 13
“…offering a more comprehensive evaluation of therapeutic options for BDD. Meta-analyses with a limited number of trials are susceptible to type I errors (overestimation of effect sizes) or type II errors (underestimation of effect sizes) and possess low credibility (Pereira & Ioannidis, 2011).”
Warraich, N., Sá, M. P., Jacquemyn, X., Kuno, T., Serna-Gallegos, D. R., & Sultan, I. S. (2025) · Journal of the American Heart Association · doi:10.1161/jaha.124.038869 · Cited by 10
Jacquemyn, X., Sá, M. P., Rega, F., Verbrugghe, P., Meuris, B., Serna-Gallegos, D. R., Brown, J. A., Clavel, M.-A. A., Pîbarot, P., & Sultan, I. S. (2024) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2024.04.007 · Cited by 9
Saunders, B., & Inzlicht, M. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.01.018 · Cited by 9
“vide effect sizes that are unreliably inflated (Pereira and Ioannidis, 2011).”
BackgroundTowards a genetically informed approach in the social sciences: Strengths and an opportunity
Bates, T. C., & Lewis, G. J. (2012) · Personality and Individual Differences · doi:10.1016/j.paid.2012.03.002 · Cited by 9
“Pereira and Ioannidis (2011), for instance, examined the true-positive probability of conclusions based on a population of four hundred sixtyone meta-analyses of experimental clinical trials.”
BackgroundShah, S., Priyanka, P., & Sharma, S. (2022) · Indian Journal of Clinical Biochemistry · doi:10.1007/s12291-022-01091-3 · Cited by 8
“This ultimately leads to overestimation (type I error) or underestimation (type II error) of the intervention effect [ 24].”
The Rollercoaster of Paclitaxel in the Lower Limbs and Skeletons in the Closet: An Opinion Review
Katsanos, K. H., Kitrou, P. M., & Spiliοpoulos, S. (2021) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2021.03.537 · Cited by 7
Zanghelini, F., Zimmermann, I. R., Andrade, C. A. S., & Wu, O. (2018) · Journal of Palliative Care & Medicine · doi:10.4172/2165-7386.1000343 · Cited by 7
“This model is preferable when there is heterogeneity in outcomes and provide estimates that are more conservators [10].”
BackgroundMalik, I. S., Khan, Z. I., Ahmad, K., Nadeem, M., Akhtar, S., Ashfaq, A., Faisal, M., Alatar, A. A., Munir, M., & Ullah, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-13628-6 · Cited by 6
Zheng, S.-W., Lin, X., Xie, N.-S., Zhang, X., Deng, F., Zou, H.-Q., Zhan, X.-L., & Tang, G.-Y. (2025) · World Journal of Diabetes · doi:10.4239/wjd.v16.i11.111578 · Cited by 8
Systematic Reviews With Meta-analysis of Clinical Trials: Is There Enough Evidence?
Garcia-Alamino, J. M., & Cano, M. L. (2020) · Cirugía Española (English Edition) · doi:10.1016/j.cireng.2019.12.009 · Cited by 6
Metadata
- Authors
- Pereira, T. V., & Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Tufts University (US)
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2011-10
- Volume
- 64
- Issue
- 10
- Pages
- 1060-1069
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2004108172
- Cited by
- 160
- References
- 71
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 8 · 2023: 7 · 2022: 5 · 2021: 14 · 2020: 25 · 2019: 7 · 2018: 7 · 2017: 7 · 2016: 11 · 2015: 14 · 2014: 8 · 2013: 13 · 2012: 9
- FWCI
- 2.72
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministério da Educação
- Fundação de Amparo à Pesquisa do Estado de São Paulo
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- meta-analysis, statistical significance, effect size inflation, prior assumptions, binary outcomes, odds ratio, limited evidence, Cochrane Database of Systematic Reviews
- MeSH terms
- Clinical Trials as Topic, Health Services Research, Humans, Meta-Analysis as Topic, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
