The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Meta‐research: The art of getting it wrong
Abstract
Meta‐analysis has major strengths, but sometimes it can often lead to wrong and misleading answers. In this SRSM presidential address, I discuss some case studies that exemplify these problems, including examples from meta‐analyses of both clinical trials and observational associations. I also discuss issues of effect size estimation, bias (in particular significance‐chasing biases), and credibility in meta‐research. I examine the factors that affect the credibility of meta‐analyses, including magnitude of effects, multiplicity of analyses, scale of data, flexibility of analyses, reporting, and conflicts of interest. Under the current circumstances, a survey of expert meta‐analysts attending the SRSM meeting showed that most of them believe that the true effect is practically equally likely to lie within the 95% confidence interval of a meta‐analysis or outside of it. Finally, I address the placement of meta‐analysis in the wider current research agenda and make a plea for adoption of more prospective meta‐designs. In many/most/all fields, all primary original research may be designed, executed, and interpreted as a prospective meta‐analysis. Copyright © 2011 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 7×
“Many effects tend to diminish over time (Gehr et al., 2006; Ioannidis, 2005a, 2006; Jennions and Moeller, 2002; Krum and Tonkin, 2003; Leimu and Koricheva, 2004; Trikalinos et al., 2004; Zollner and Pritchard, 2007).”
BackgroundResultInfluentialSelf-citationOptimal Medical Therapy with or without PCI for Stable Coronary Disease
Boden, W. E., O'Rourke, R. A., Teo, K. K., Hartigan, P. M., Maron, D. J., Kostuk, W. J., Knudtson, M., Dada, M., Casperson, P., Harris, C. L., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa070829 · Cited by 4,699 · cited 1×
“Our meta-analysis in 2005 (Ioannidis and Katritsis, 2005) found no benefit from invasive management, and a large randomized trial published 2 years later also found no benefit (Boden et al., 2007).”
Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Nissen, S. E., & Wolski, K. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072761 · Cited by 4,675
Genome-wide association studies for complex traits: consensus, uncertainty and challenges
McCarthy, M. I., Abecasis, G. R., Cardon, L. R., Goldstein, D. B., Little, J., Ioannidis, J. P. A., & Hirschhorn, J. N. (2008) · Nature Reviews Genetics · doi:10.1038/nrg2344 · Cited by 2,891
Self-citationPublication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766
Self-citationSystematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
Self-citationWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Self-citationContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Self-citationThe appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052 · cited 4×
“2010, 1 169--184 1 7 5 A number of tests have been developed by meta-analysts to detect and exclude publication bias, but I am afraid that they are used beyond their real capacity (Ioannidis, 2008a; Ioannidis and Trikalinos, 2007b; Lau et al., 2006).”
BackgroundInfluentialSelf-citationEpidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891 · cited 1×
“The number of available trials and the total sample size is a bit larger than the average seen in current medical meta-analyses (Moher et al., 2007), but not by a lot.”
BackgroundPublication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818
Ioannidis, J. P. A. (2009) · Canadian Medical Association Journal · doi:10.1503/cmaj.081086 · Cited by 716
Self-citationInterpretation of tests of heterogeneity and bias in meta-analysis
Ioannidis, J. P. A. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.00986.x · Cited by 696 · cited 4×
“2010, 1 169--184 1 7 5 A number of tests have been developed by meta-analysts to detect and exclude publication bias, but I am afraid that they are used beyond their real capacity (Ioannidis, 2008a; Ioannidis and Trikalinos, 2007b; Lau et al., 2006).”
BackgroundInfluentialSelf-citationIoannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.4.281 · Cited by 632
Self-citationChan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613 · cited 3×
“…proportion of effects that are not reported in various fields of research based on some empirical data (as the study I just described and others (Chan and Altman, 2005; Chan et al., 2004a,b; Contopoulos-Ioannidis et al., 2006; Mathieu et al., 2009)), personal experience, and conversation with…”
BackgroundOutcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444
Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time
Trikalinos, T. A., Churchill, R., Ferri, M., Leucht, S., Tuunainen, A., Wahlbeck, K., & Ioannidis, J. P. A. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.018 · Cited by 177
Self-citationPereira, T. V., & Ioannidis, J. P. A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.12.012 · Cited by 160 · cited 1×
“…evaluation of medical meta-analyses that have been updated over a time window of 5 years shows indeed that the statistically significant treatment effects are modestly inflated, and the inflation is large when the significant summary effect is based on limited data (Pereira and Ioannidis, 2011).”
BackgroundSelf-citationLeimu, R., & Koricheva, J. (2004) · Proceedings of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rspb.2004.2828 · Cited by 156
Hedges, L. V., & Vevea, J. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch9 · Cited by 123
Availability of large-scale evidence on specific harms from systematic reviews of randomized trials.
Papanikolaou, P. N., & Ioannidis, J. P. A. (2004) · The American Journal of Medicine · doi:10.1016/j.amjmed.2004.04.026 · Cited by 79
Self-citation
Cited by
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Self-citationMeta-evaluation of meta-analysis: ten appraisal questions for biologists
Nakagawa, S., Noble, D. W. A., Senior, A. M., & Lagisz, M. (2017) · BMC Biology · doi:10.1186/s12915-017-0357-7 · Cited by 579
Homocysteine lowering interventions for preventing cardiovascular events
Martí-Carvajal, A. J., Solà, I., Lathyris, D., & Salanti, G. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006612.pub2 · Cited by 520
Szucs, D., & Ioannidis, J. P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117164 · Cited by 453
Self-citationAugusto, L., De Schrijver, A., Vesterdal, L., Smolander, A., Prescott, C., & Ranger, J. (2015) · Biological Reviews · doi:10.1111/brv.12119 · Cited by 411
Metadata
- Authors
- Ioannidis, J. P. A.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2010-07
- Volume
- 1
- Issue
- 3-4
- Pages
- 169-184
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W1972260676
- Cited by
- 227
- References
- 24
- FWCI
- 8.93
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.