Evaluation of networks of randomized trials
The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
Randomized trials may be designed and interpreted as single experiments or they may be seen in the context of other similar or relevant evidence. The amount and complexity of available randomized evidence vary for different topics. Systematic reviews may be useful in identifying gaps in the existing randomized evidence, pointing to discrepancies between trials, and planning future trials. A new, promising, but also very much debated extension of systematic reviews, mixed treatment comparison (MTC) meta-analysis, has become increasingly popular recently. MTC meta-analysis may have value in interpreting the available randomized evidence from networks of trials and can rank many different treatments, going beyond focusing on simple pairwise-comparisons. Nevertheless, the evaluation of networks also presents special challenges and caveats. In this article, we review the statistical methodology for MTC meta-analysis. We discuss the concept of inconsistency and methods that have been proposed to evaluate it as well as the methodological gaps that remain. We introduce the concepts of network geometry and asymmetry, and propose metrics for the evaluation of the asymmetry. Finally, we discuss the implications of inconsistency, network geometry and asymmetry in informing the planning of future trials.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Network meta-analysis results usually give a more precise estimate, compared with results from standard, pairwise analyses, [48, 49]”
From Efficacy of psychological therapies for irritable bowel syndrome: systematic review and network meta-analysis (2020) · cites this paper as [48]
In the citing paper
“network meta-analysis offers a methodological solution to provide more precise pooled head-to-head effect estimates than may otherwise be achieved 19, 20 .”
Citations
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Self-citationQuantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Self-citationProceedings of the National Academy of Sciences of the United States of America
Dowling, J. E., & Wald, G. (2009) · Nutrition Reviews · doi:10.1111/j.1753-4887.1981.tb06752.x · Cited by 17,893
Ecological Diversity and Its Measurement
Magurran, A. E. (1988) · Springer eBooks · doi:10.1007/978-94-015-7358-0 · Cited by 10,879
Keeley, E. C., Boura, J. A., & Grines, C. L. (2003) · The Lancet · doi:10.1016/s0140-6736(03)12113-7 · Cited by 4,361
Assessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256
The Nonconcept of Species Diversity: A Critique and Alternative Parameters
Hurlbert, S. H. (1971) · Ecology · doi:10.2307/1934145 · Cited by 4,183
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309
How should meta‐regression analyses be undertaken and interpreted?
Thompson, S. G., & Higgins, J. P. T. (2002) · Statistics in Medicine · doi:10.1002/sim.1187 · Cited by 3,026
Self-citationHammer, S. M., Squires, K. E., Hughes, M. D., Grimes, J. M., Demeter, L. M., Currier, J. S., Eron, J. J., Feinberg, J. E., Balfour, H. H., Deyton, L. R., et al. (1997) · New England Journal of Medicine · doi:10.1056/nejm199709113371101 · Cited by 2,640
Combination of direct and indirect evidence in mixed treatment comparisons
Lu, G., & Ades, A. E. (2004) · Statistics in Medicine · doi:10.1002/sim.1875 · Cited by 2,166
Self-citationPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147
(2000) · Queen Mary Research Online (Queen Mary University of London) · Cited by 1,795
Systematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Simultaneous comparison of multiple treatments: Combining direct and indirect evidence.
Caldwell, D. M., Ades, A. E., & Higgins, J. P. T. (2005) · BMJ · doi:10.1136/bmj.331.7521.897 · Cited by 1,600
Self-citationNetwork meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206
Cumulative Meta-Analysis of Therapeutic Trials for Myocardial Infarction
Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992) · New England Journal of Medicine · doi:10.1056/nejm199207233270406 · Cited by 1,140
Hammer, S. M., Katzenstein, D. A., Hughes, M. D., Gundacker, H., Schooley, R. T., Haubrich, R. H., Henry, W. K., Lederman, M. M., Phair, J. P., Niu, M., et al. (1996) · New England Journal of Medicine · doi:10.1056/nejm199610103351501 · Cited by 1,091
Psaty, B. M., Lumley, T., Furberg, C. D., Schellenbaum, G., Pahor, M., Alderman, M. H., & Weiss, N. S. (2003) · JAMA · doi:10.1001/jama.289.19.2534 · Cited by 1,088
The checkerboard score and species distributions
Stone, L., & Roberts, A. (1990) · Oecologia · doi:10.1007/bf00317345 · Cited by 1,051
Treatment for Adult HIV Infection: 2006 Recommendations of the International AIDS Society–USA Panel
Hammer, S. M., Saag, M. S., Schechter, M., Montaner, J. S. G., Schooley, R. T., Jacobsen, D. M., Thompson, M. A., Carpenter, C. C. J., Fischl, M. A., Gazzard, B. G., et al. (2006) · JAMA · doi:10.1001/jama.296.7.827 · Cited by 1,033
Song, F., Altman, D. G., Glenny, A.-M., & Deeks, J. J. (2003) · BMJ · doi:10.1136/bmj.326.7387.472 · Cited by 943
Users' Guides to the Medical Literature: IX. A Method for Grading Health Care Recommendations
Guyatt, G. H. (1995) · JAMA · doi:10.1001/jama.1995.03530220066035 · Cited by 846
Meta-analysis of the literature or of individual patient data: is there a difference?
Stewart, L. A., & Parmar, M. K. B. (1993) · The Lancet · doi:10.1016/0140-6736(93)93004-k · Cited by 769
Assessing Evidence Inconsistency in Mixed Treatment Comparisons
Lu, G., & Ades, A. E. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000001302 · Cited by 756
Self-citationHiggins, J., Thompson, S., Deeks, J., & Altman, D. (2002) · Journal of Health Services Research & Policy · doi:10.1258/1355819021927674 · Cited by 755
Self-citationIndirect comparisons of competing interventions
Glenny, A., Altman, D., Song, F., Sakarovitch, C., Deeks, J., D'Amico, R., Bradburn, M., & Eastwood, A. (2005) · Health Technology Assessment · doi:10.3310/hta9260 · Cited by 677
Summing up evidence: one answer is not always enough
Lau, J., Ioannidis, J. P., & Schmid, C. H. (1998) · The Lancet · doi:10.1016/s0140-6736(97)08468-7 · Cited by 655
Self-citationBias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636
BORROWING STRENGTH FROM EXTERNAL TRIALS IN A META-ANALYSIS
Higgins, J. P. T., & WHITEHEAD, A. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961230)15:24<2813::aid-sim366>3.0.co;2-u · Cited by 567
Self-citationBORROWING STRENGTH FROM EXTERNAL TRIALS IN A META-ANALYSIS
HIGGINS, J. P. T., & WHITEHEAD, A. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961230)15:24<2733::aid-sim562>3.0.co;2-0 · Cited by 557
Self-citationBalk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478
Epidemiology and reporting of randomised trials published in PubMed journals
Chan, A.-W., & Altman, D. G. (2005) · The Lancet · doi:10.1016/s0140-6736(05)71879-1 · Cited by 474
Heres, S., Davis, J., Maino, K., Jetzinger, E., Kissling, W., & Leucht, S. (2006) · American Journal of Psychiatry · doi:10.1176/appi.ajp.163.2.185 · Cited by 450
Improved confidence intervals for the difference between binomial proportions based on paired data
Newcombe, R. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19981130)17:22<2597::aid-sim940>3.0.co;2-g · Cited by 432
Houben, Beek, D. V. D., Hensen, Craen, A. J. M. D., Rauws, & Tytgat (1999) · Alimentary Pharmacology & Therapeutics · doi:10.1046/j.1365-2036.1999.00555.x · Cited by 333
Meta-analysis of multiple outcomes by regression with random effects
Berkey, C. S., Hoaglin, D. C., Antczak-Bouckoms, A., Mosteller, F., & Colditz, G. A. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19981130)17:22<2537::aid-sim953>3.0.co;2-c · Cited by 280
Synthesis of Environmental Evidence: Nitrogen Dioxide Epidemiology Studies
Hasselblad, V., Eddy, D. M., & Kotchmar, D. J. (1992) · Journal of the Air & Waste Management Association · doi:10.1080/10473289.1992.10467018 · Cited by 202
Kyrgiou, M., Salanti, G., Pavlidis, N., Paraskevaidis, E., & Ioannidis, J. P. A. (2006) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djj443 · Cited by 187
Self-citationMcAlister, F. A., Laupacis, A., Wells, G. A., Sackett, D. L., & for the Evidence-Based Medicine Working Group (1999) · JAMA · doi:10.1001/jama.282.14.1371 · Cited by 177
Clarke, M., & Chalmers, I. (1998) · JAMA · doi:10.1001/jama.280.3.280 · Cited by 169
Multiparameter Evidence Synthesis in Epidemiology and Medical Decision-Making: Current Approaches
Ades, A. E., & Sutton, A. J. (2006) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2005.00377.x · Cited by 159
Self-citationNixon, R. M., Bansback, N. J., & Brennan, A. (2006) · Statistics in Medicine · doi:10.1002/sim.2624 · Cited by 156
Cooper, N. J., Sutton, A. J., Lu, G., & Khunti, K. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.12.1269 · Cited by 152
Meta-analysis by the confidence profile method
Eddy, D. M. (1992) · Cited by 144
Evidence‐based sample size calculations based upon updated meta‐analysis
Sutton, A. J., Cooper, N. J., Jones, D. R., Lambert, P. C., Thompson, J. R., & Abrams, K. R. (2006) · Statistics in Medicine · doi:10.1002/sim.2704 · Cited by 144
Discussion Sections in Reports of Controlled Trials Published in General Medical Journals
Clarke, M. J. (2002) · JAMA · doi:10.1001/jama.287.21.2799 · Cited by 126
Pilote, L., Abrahamowicz, M., Rodrigues, E., Eisenberg, M. J., & Rahme, E. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-141-2-200407200-00008 · Cited by 126
Jansen, J. P. (2006) · Current Medical Research and Opinion · doi:10.1185/030079906x96308 · Cited by 125
Baker, S. G., & Kramer, B. S. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-13 · Cited by 116
Chou, R., Fu, R. F., Huffman, L. H., & Korthuis, P. T. (2006) · The Lancet · doi:10.1016/s0140-6736(06)69638-4 · Cited by 110
Systematic reviews: when is an update an update?
Moher, D., & Tsertsvadze, A. (2006) · The Lancet · doi:10.1016/s0140-6736(06)68358-x · Cited by 110
Lilford, R. J. (2003) · BMJ · doi:10.1136/bmj.326.7396.980 · Cited by 108
What are the implications of optimism bias in clinical research?
Chalmers, I., & Matthews, R. (2006) · The Lancet · doi:10.1016/s0140-6736(06)68153-1 · Cited by 98
Indirect comparisons: the mesh and mess of clinical trials
Ioannidis, J. P. A. (2006) · The Lancet · doi:10.1016/s0140-6736(06)69615-3 · Cited by 94
Self-citationAre all angiotensin-converting enzyme inhibitors interchangeable?
Furberg, C. D., & Pitt, B. (2001) · Journal of the American College of Cardiology · doi:10.1016/s0735-1097(01)01161-5 · Cited by 83
Playford, E. G., Webster, A. C., Sorrell, T. C., & Craig, J. C. (2006) · European Journal of Clinical Microbiology & Infectious Diseases · doi:10.1007/s10096-006-0182-3 · Cited by 71
Zhou, Z., Rahme, E., Abrahamowicz, M., Tu, J. V., Eisenberg, M. J., Humphries, K., Austin, P. C., & Pilote, L. (2005) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041403 · Cited by 56
Is the Jadad score the proper evaluation of trials?
Berger, V. W. (2006) · PubMed · Cited by 55
The quality of medical evidence in hematology-oncology
Djulbegović, B., Loughran, T. P., Hornung, C. A., Kloecker, G. H., Efthimiadis, E. N., Hadley, T. J., Englert, J., Hoskins, M., & Goldsmith, G. H. (1999) · The American Journal of Medicine · doi:10.1016/s0002-9343(98)00391-x · Cited by 51
Ioannidis, J. P. A., Cappelleri, J. C., Sacks, H. S., & Lau, J. (1997) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)00097-4 · Cited by 42
Self-citationChallenges in Systematic Reviews That Evaluate Drug Efficacy or Effectiveness
Santaguida, P. L., Helfand, M., & Raina, P. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-12_part_2-200506211-00006 · Cited by 37
Clinical Trials in Sub-Saharan Africa and Established Standards of Care
Kent, D. M., Mwamburi, D. M., Bennish, M. L., Kupelnick, B., & Ioannidis, J. P. A. (2004) · JAMA · doi:10.1001/jama.292.2.237 · Cited by 36
Self-citationSystematic Reviews in Health Care: Meta-Analysis in Context: Second Edition
Egger, M., Smith, G., & Altman, D. G. (2008) · Cited by 32
META-ANALYSIS: CURRENT ISSUES IN RESEARCH SYNTHESIS
Olkin, I. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960630)15:12<1263::aid-sim305>3.0.co;2-k · Cited by 29
Randomised trials comparing chemotherapy regimens for advanced non-small cell lung cancer
Ioannidis, J. P. A., Polycarpou, A., Ntais, C., & Pavlidis, N. A. (2003) · European Journal of Cancer · doi:10.1016/s0959-8049(03)00571-9 · Cited by 26
Self-citationPrinciples for evidence-based drug formulary policy
Simon, G. E., Psaty, B. M., Hrachovec, J. B., & Mora, M. (2005) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2005.0232.x · Cited by 25
The Cochrane Library and bronchiolitis: an umbrella review
Bialy, L. M., Smith, M. B. H., Bourke, T. W., & Becker, L. A. (2006) · Evidence-Based Child Health A Cochrane Review Journal · doi:10.1002/ebch.92 · Cited by 18
Griffin, S. C., Bojke, L., Main, C., & Palmer, S. (2006) · Value in Health · doi:10.1111/j.1524-4733.2006.00090.x · Cited by 18
Chung, A. E., Evans, C. S., White, P. J., & Lomotan, E. A. (2022) · Springer eBooks · doi:10.1007/978-3-030-93765-2_5 · Cited by 13
Research works printed but not linked: 4
- Piantadosi S., 1997, Clinical trials: A methodological approach
- Gray JAM., 2001, Evidence-based healthcare
- Berger VW, 2006, Journal of Rheumatology, 33, 1710
- 2005, EcoSim [computer program]. Version 7.72 Kelsey-Bear
Cited by
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
“Network meta-analyses, also known as multiple treatments meta-analyses, can be used to analyze data from comparisons of many different treatments [176,177].”
MethodSelf-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
“Network meta-analyses, also known as multiple treatments meta-analyses, can be used to analyze data from comparisons of many different treatments [176,177].”
MethodSelf-citationHutton, B., Salanti, G., Caldwell, D. M., Chaimani, A., Schmid, C. H., Cameron, C., Ioannidis, J. P. A., Straus, S., Thorlund, K., Jansen, J. P., et al. (2015) · Annals of Internal Medicine · doi:10.7326/m14-2385 · Cited by 8,269
Self-citation
Metadata
- Authors
- Salanti, G., Higgins, J. P., Ades, A., & Ioannidis, J. P.
- Affiliations
- University of Ioannina (GR)
- MRC Biostatistics Unit (GB)
- MRC Health Services Collaboration, Bristol, UK
- Tufts University (US)
- Corresponding author
- Ioannidis, John PA
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2007-10-09
- Volume
- 17
- Issue
- 3
- Pages
- 279-301
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2105669432
- Cited by
- 1,078
- References
- 69
- Citations by year
- 2026: 26 · 2025: 39 · 2024: 43 · 2023: 55 · 2022: 67 · 2021: 68 · 2020: 77 · 2019: 95 · 2018: 82 · 2017: 79 · 2016: 71 · 2015: 79 · 2014: 74 · 2013: 63 · 2012: 66
- FWCI
- 17.76
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- inconsistency, network geometry, network asymmetry, treatment ranking, trial planning, statistical methodology
- MeSH terms
- Network Meta-Analysis as Topic, Humans, Research Design, Meta-Analysis as Topic, Models, Statistical, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.