Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Network meta-analysis (NMA) has become a popular method to compare more than two treatments. This scoping review aimed to explore the characteristics and methodological quality of knowledge synthesis approaches underlying the NMA process. We also aimed to assess the statistical methods applied using the Analysis subdomain of the ISPOR checklist. METHODS: Comprehensive literature searches were conducted in MEDLINE, PubMed, EMBASE, and Cochrane Database of Systematic Reviews from inception until April 14, 2015. References of relevant reviews were scanned. Eligible studies compared at least four different interventions from randomised controlled trials with an appropriate NMA approach. Two reviewers independently performed study selection and data abstraction of included articles. All discrepancies between reviewers were resolved by a third reviewer. Data analysis involved quantitative (frequencies) and qualitative (content analysis) methods. Quality was evaluated using the AMSTAR tool for the conduct of knowledge synthesis and the ISPOR tool for statistical analysis. RESULTS: After screening 3538 citations and 877 full-text papers, 456 NMAs were included. These were published between 1997 and 2015, with 95% published after 2006. Most were conducted in Europe (51%) or North America (31%), and approximately one-third reported public sources of funding. Overall, 84% searched two or more electronic databases, 62% searched for grey literature, 58% performed duplicate study selection and data abstraction (independently), and 62% assessed risk of bias. Seventy-eight (17%) NMAs relied on previously conducted systematic reviews to obtain studies for inclusion in their NMA. Based on the AMSTAR tool, almost half of the NMAs incorporated quality appraisal results to formulate conclusions, 36% assessed publication bias, and 16% reported the source of funding. Based on the ISPOR tool, half of the NMAs did not report if an assessment for consistency was conducted or whether they accounted for inconsistency when present. Only 13% reported heterogeneity assumptions for the random-effects model. CONCLUSIONS: The knowledge synthesis methods and analytical process for NMAs are poorly reported and need improvement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hutton, B., Salanti, G., Caldwell, D. M., Chaimani, A., Schmid, C. H., Cameron, C., Ioannidis, J. P. A., Straus, S., Thorlund, K., Jansen, J. P., et al. (2015) · Annals of Internal Medicine · doi:10.7326/m14-2385 · Cited by 8,269 · cited 2×
“The recent publication of the PRISMA extension statement for NMAs [15] may lead to improvement in quality of reporting over time.”
BackgroundSelf-citationA basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793 · cited 1×
“a fixed-effect and a random-effects model to combine studies be based on a test of homogeneity, which is misguided [45, 46].”
BackgroundPRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement
McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.01.021 · Cited by 5,700
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755 · cited 2×
“designed and validated to assess the methodological quality of systematic reviews of RCTs [26], so it is appropriate for NMAs of RCTs.”
MethodSalanti, G. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1037 · Cited by 1,729 · cited 2×
“The use of NMA has increased rapidly since the mid2000s [4, 11, 12].”
BackgroundMethodSelf-citationSimultaneous comparison of multiple treatments: Combining direct and indirect evidence.
Caldwell, D. M., Ades, A. E., & Higgins, J. P. T. (2005) · BMJ · doi:10.1136/bmj.331.7521.897 · Cited by 1,600 · cited 2×
“on previously described NMA approaches (n = 76, 17%) [9], and a paper on a statistical approach to generate indirect evidence as an extension to pairwise meta-analysis (n = 71, 16%; Additional file 1: Appendix 5) [7].”
BackgroundMethodNetwork meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 1×
“keep abreast of the literature and inform decisions based on the totality of evidence [1, 2].”
BackgroundConceptual and technical challenges in network meta-analysis
Cipriani, A., Higgins, J. P. T., Geddes, J. R., & Salanti, G. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-159-2-201307160-00008 · Cited by 1,072
Self-citation
Research works printed but not linked: 27
- Patsopoulos NA, Analatos AA, Ioannidis JP. Relative citation impact of various study designs in the health sciences. JAMA. 2005;293(19):2362–6. · DOI · Paper page
- Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997;50(6):683–91. · DOI · Paper page
- Higgins JP, Whitehead A. Borrowing strength from external trials in a meta-analysis. Stat Med. 1996;15(24):2733–49. · DOI · Paper page
- Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med. 2004;23(20):3105–24. · DOI · Paper page
- Nikolakopoulou A, Chaimani A, Veroniki AA, Vasiliadis HS, Schmid CH, Salanti G. Characteristics of networks of interventions: a description of a database of 186 published networks. PLoS One. 2014;9(1):e86754. · DOI
- Lee AW. Review of mixed treatment comparisons in published systematic reviews shows marked increase since 2009. J Clin Epidemiol. 2014;67(2):138–43. · DOI
- Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health. 2011;14(4):429–37. · DOI · Paper page
- Jansen JP, Fleurence R, Devine B, Itzler R, Barrett A, Hawkins N, et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health. 2011;14(4):417–28. · DOI · Paper page
- Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32. · DOI · Paper page
- Petropoulou M, Nikolakopoulou A, Veroniki AA, Rios P, Vafaei A, Zarin W, et al. Bibliographic study showed improving statistical methodology of network meta-analyses published between 1999 and 2015. J Clin Epidemiol. 2016. · DOI
- Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ. 2009;338:b1147. · DOI · Paper page
- Greco T, Biondi-Zoccai G, Saleh O, Pasin L, Cabrini L, Zangrillo A, et al. The attractiveness of network meta-analysis: a comprehensive systematic and narrative review. Heart Lung Vessel. 2015;7(2):133–42.
- Sharif MO, Janjua-Sharif FN, Ali H, Ahmed F. Systematic reviews explained: AMSTAR-how to tell the good from the bad and the ugly. Oral Health Dent Manag. 2013;12(1):9–16.
- Thompson Reuters. Journal Citation Reports. 2015. http://wokinfo.com/products_tools/analytical/jcr/ . Accessed Aug 2016.
- Wickham H, Chang W, RStudio. Package ‘ggplot2’. An Implementation of the Grammar of Graphics. 2.1.0 2016. https://cran.r-project.org/web/packages/ggplot2/ggplot2.pdf . · DOI · Paper page
- Borenstein M, Hedges L, Higgins J, Rothstein H. Introduction to Meta-Analysis. Hoboken, NY: John Wiley & Sons, Ltd.; 2009. · DOI · Paper page
- Donahue KE, Jonas DE, Hansen RA, Roubey R, Jonas B, Lux LJ, et al. AHRQ Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2012.
- Shamliyan TA, Kane RL, Taylor FR. AHRQ Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
- Selph S, Carson S, Fu R, Thakurta S, Low A, McDonagh M. Drug Class Reviews. Portland, OR: Oregon Health & Science University; 2011.
- Smith B, Peterson K, Fu R, McDonagh M, Thakurta S. Drug Class Reviews. Portland, OR: Oregon Health & Science University; 2011.
- Brodszky V. Effectiveness of biological treatments based on ACR70 response in rheumatoid arthritis: indirect comparison and meta-regression using Bayes-model. Orv Hetil. 2011;152(23):919–28. · DOI
- Ni RH, Tang HL, Zhai SD, Li ZL. Multiple treatments for infantile rotavirus enteritis: a network meta-analysis. World J Gastroenterol. 2012;20(5):438–43.
- Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. · DOI · Paper page
- Clark HD, Wells GA, Huet C, McAlister FA, Salmi LR, Fergusson D, et al. Assessing the quality of randomized trials: reliability of the Jadad scale. Control Clin Trials. 1999;20(5):448–52. · DOI · Paper page
- Lexchin J. Sponsorship bias in clinical research. Int J Risk Saf Med. 2012;24(4):233–42. · DOI
- Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76. · DOI · Paper page
- Hoaglin DC. Misunderstandings about Q and 'Cochran's Q test' in meta-analysis. Stat Med. 2016;35(4):485–95. · DOI · Paper page
Cited by
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
Self-citationCINeMA: an approach for assessing confidence in the results of a network meta-analysis
Nikolakopoulou, A., Higgins, J. P. T., Papakonstantinou, T., Chaimani, A., Del Giovane, C., Egger, M., & Salanti, G. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003082 · Cited by 1,935
Self-citationMeta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709
“Developments in and applications of this powerful approach has increased dramatically in clinical medicine over the last 10 years 88 .”
Naci, H., Salcher-Konrad, M., Dias, S., Blum, M. R., Sahoo, S. A., Nunan, D., & Ioannidis, J. P. A. (2019) · British Journal of Sports Medicine · doi:10.1136/bjsports-2018-099921 · Cited by 326
BUGSnet: an R package to facilitate the conduct and reporting of Bayesian network Meta-analyses
Béliveau, A., Boyne, D. J., Slater, J., Brenner, D., & Arora, P. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0829-2 · Cited by 312
Metadata
- Authors
- Zarin, W., Veroniki, A. A., Nincic, V., Vafaei, A., Reynen, E., Motiwala, S. S., Antony, J., Sullivan, S. M., Rios, P., Daly, C., Ewusie, J., Petropoulou, M., Nikolakopoulou, A., Chaimani, A., Salanti, G., Straus, S. E., & Tricco, A. C.
- Venue
- BMC Medicine
- Published
- 2017-01-05
- Volume
- 15
- Issue
- 1
- Pages
- 3-3
- ISSN
- 1741-7015 (ISSN-L)
- OpenAlex
- W2562241822
- Cited by
- 97
- References
- 13
- FWCI
- 11.31
- Type
- journal article
- Language
- en
- Update notices
- correction (2017-03-14)
- Open access
- gold
- Funders
- Canadian Agency for Drugs and Technologies in Health
- Tier 2 Canada Research Chair in Knowledge Synthesis
- Tier 1 Canada Research Chair in Knowledge Translation
- Canadian Institutes of Health Research Banting Postdoctoral Fellowship Program
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. https://app.cochat.ai/papers/10.1186/s12916-016-0764-6
- The paper
- Zarin, W., Veroniki, A. A., Nincic, V., Vafaei, A., Reynen, E., Motiwala, S. S., Antony, J., Sullivan, S. M., Rios, P., Daly, C., Ewusie, J., Petropoulou, M., Nikolakopoulou, A., Chaimani, A., Salanti, G., Straus, S. E., & Tricco, A. C. (2017). Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Medicine. https://doi.org/10.1186/s12916-016-0764-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.