The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis
Open accessAbstract
Network meta-analysis (NMA), combining direct and indirect comparisons, is increasingly being used to examine the comparative effectiveness of medical interventions. Minimal guidance exists on how to rate the quality of evidence supporting treatment effect estimates obtained from NMA. We present a four-step approach to rate the quality of evidence in each of the direct, indirect, and NMA estimates based on methods developed by the GRADE working group. Using an example of a published NMA, we show that the quality of evidence supporting NMA estimates varies from high to very low across comparisons, and that quality ratings given to a whole network are uninformative and likely to mislead.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The overall certainty of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework [45].”
From The Effect of Different Psychological Interventions on Athletes' Competitive Anxiety: A Systematic Review and Bayesian Network Meta-Analysis. (2026) · cites this paper as [45]
In the citing paper
“Therefore, we downgraded the quality of evidence for the comparison between SGLT2i and finerenone, SGLT2i and GLP-1 RA.”
In the citing paper
“Network meta-analysis, rather than pairwise meta-analysis, provides useful information about the comparative effectiveness of treatments that have not been tested head to head.”
From Drug treatments for covid-19: living systematic review and network meta-analysis (2020) · cites this paper as [5]
In the citing paper
“Moreover, the incorporation of indirect evidence can strengthen evidence in comparisons that were tested head to head.5”
From Drug treatments for covid-19: living systematic review and network meta-analysis (2020) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Self-citationGRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology
Guyatt, G. H., Oxman, A. D., Schünemann, H. J., Tugwell, P., & Knottnerus, A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.011 · Cited by 3,094
Self-citationGRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Self-citationGRADE guidelines 6. Rating the quality of evidence—imprecision
Guyatt, G. H., Oxman, A. D., Kunz, R., Brozek, J., Alonso-Coello, P., Rind, D., Devereaux, P. J., Montori, V. M., Freyschuss, B., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.012 · Cited by 2,693
Self-citationChecking consistency in mixed treatment comparison meta-analysis
Dias, S., Welton, N. J., Caldwell, D. M., & Ades, A. E. (2010) · Statistics in Medicine · doi:10.1002/sim.3767 · Cited by 2,521
Graphical tools for network meta-analysis in STATA
Chaimani, A., Higgins, J. P. T., Mavridis, D., Spyridonos, P., & Salanti, G. (2013) · PLoS ONE · doi:10.1371/journal.pone.0076654 · Cited by 2,473
GRADE guidelines: 7. Rating the quality of evidence—inconsistency
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Glasziou, P., Jaeschke, R., Akl, E. A., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.03.017 · Cited by 2,320
Self-citationCombination of direct and indirect evidence in mixed treatment comparisons
Lu, G., & Ades, A. E. (2004) · Statistics in Medicine · doi:10.1002/sim.1875 · Cited by 2,172
Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies
Higgins, J. P. T., Jackson, D., Barrett, J. K., Lu, G., Ades, A. E., & White, I. R. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1044 · Cited by 2,098
GRADE guidelines: 8. Rating the quality of evidence—indirectness
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Falck-Ytter, Y., Jaeschke, R., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.014 · Cited by 1,843
Self-citationGRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819
Self-citationCipriani, A., Furukawa, T. A., Salanti, G., Geddes, J. R., Higgins, J. P., Churchill, R., Watanabe, N., Nakagawa, A., Omori, I. M., McGuire, H., et al. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60046-5 · Cited by 1,654
Outcomes associated with drug-eluting and bare-metal stents: a collaborative network meta-analysis
Stettler, C., Wandel, S., Allemann, S., Kastrati, A., Morice, M., Schömig, A., Pfisterer, M. E., Stone, G. W., Leon, M. B., de Lezo, J. S., et al. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61444-5 · Cited by 1,435
Evaluating the quality of evidence from a network meta-analysis
Salanti, G., Del Giovane, C., Chaimani, A., Caldwell, D. M., & Higgins, J. P. T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099682 · Cited by 1,402
Network meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206
Evaluation of networks of randomized trials
Salanti, G., Higgins, J. P., Ades, A., & Ioannidis, J. P. (2008) · Statistical Methods in Medical Research · doi:10.1177/0962280207080643 · Cited by 1,078
Conceptual and technical challenges in network meta-analysis
Cipriani, A., Higgins, J. P. T., Geddes, J. R., & Salanti, G. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-159-2-201307160-00008 · Cited by 1,072
Jansen, J. P., Fleurence, R., Devine, B., Itzler, R. F., Barrett, A., Hawkins, N., Lee, K., Boersma, C., Annemans, L., & Cappelleri, J. C. (2011) · Value in Health · doi:10.1016/j.jval.2011.04.002 · Cited by 1,029
Hoaglin, D. C., Hawkins, N., Jansen, J. P., Scott, D. A., Itzler, R. F., Cappelleri, J. C., Boersma, C., Thompson, D., Larholt, K. M., Díaz, M., et al. (2011) · Value in Health · doi:10.1016/j.jval.2011.01.011 · Cited by 769
Jansen, J. P., & Naci, H. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-159 · Cited by 656
Evaluation of inconsistency in networks of interventions
Veroniki, A. A., Vasiliadis, H. S., Higgins, J. P., & Salanti, G. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dys222 · Cited by 576
Vitamin D and vitamin D analogues for preventing fractures in post-menopausal women and older men
Avenell, A., Mak, J. C., & O'Connell, D. L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000227.pub4 · Cited by 482
Song, F., Loke, Y. K., Walsh, T., Glenny, A., Eastwood, A. J., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b1147 · Cited by 404
Network meta-analysis-highly attractive but more methodological research is needed
Li, T., Puhan, M. A., Vedula, S. S., Singh, S., Dickersin, K., & The Ad Hoc Network Meta-analysis Methods Meeting Working Group (2011) · BMC Medicine · doi:10.1186/1741-7015-9-79 · Cited by 372
Self-citationMustafa, R. A., Santesso, N., Brozek, J., Akl, E. A., Walter, S. D., Norman, G., Kulasegaram, M., Christensen, R., Guyatt, G. H., Falck-Ytter, Y., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.02.004 · Cited by 370
Self-citationWindecker, S., Stortecky, S., Stefanini, G. G., daCosta, B. R., Rutjes, A. W., Nisio, M. D., Siletta, M., Maione, A., Alfonso, F., Clemmensen, P. M., et al. (2014) · BMJ · doi:10.1136/bmj.g3859 · Cited by 365
How to Use an Article Reporting a Multiple Treatment Comparison Meta-analysis
Mills, E. J., Ioannidis, J. P. A., Thorlund, K., Schünemann, H. J., Puhan, M. A., & Guyatt, G. H. (2012) · JAMA · doi:10.1001/2012.jama.11228 · Cited by 365
Self-citationJansen, J. P., Trikalinos, T., Cappelleri, J. C., Daw, J., Andes, S., Eldessouki, R., & Salanti, G. (2014) · Value in Health · doi:10.1016/j.jval.2014.01.004 · Cited by 319
Murad, M. H., Drake, M. T., Mullan, R. J., Mauck, K. F., Stuart, L. M., Lane, M. A., Abu Elnour, N. O., Erwin, P. J., Hazem, A., Puhan, M. A., et al. (2012) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2011-3060 · Cited by 277
Self-citationPandor, A., Gomersall, T., Stevens, J., Wang, J., Al‐Mohammad, A., Bakhai, A., Cleland, J. G. F., Cowie, M., & Wong, R. (2013) · Heart · doi:10.1136/heartjnl-2013-303811 · Cited by 147
Puhan, M. A., Bachmann, L. M., Kleijnen, J., ter Riet, G., & Kessels, A. G. H. (2009) · BMC Medicine · doi:10.1186/1741-7015-7-2 · Cited by 106
Self-citationHutton, B., Salanti, G., Chaimani, A., Caldwell, D. M., Schmid, C., Thorlund, K., Mills, E., Catalá-López, F., Turner, L., Altman, D. G., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0092508 · Cited by 105
Evidence Synthesis for Decision Making 6
Dias, S., Sutton, A. J., Welton, N. J., & Ades, A. E. (2013) · Medical Decision Making · doi:10.1177/0272989x13487257 · Cited by 50
Palmerini, T., Biondi‐Zoccai, G., Riva, D. D., Mariani, A., Savini, C., Eusanio, M. D., Généreux, P., Frati, G., Marullo, A. G. M., Landoni, G., et al. (2013) · American Heart Journal · doi:10.1016/j.ahj.2013.03.011 · Cited by 42
Novel presentational approaches were developed for reporting network meta-analysis
Tan, S. H., Cooper, N. J., Bujkiewicz, S., Welton, N. J., Caldwell, D. M., & Sutton, A. J. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.11.006 · Cited by 40
How Far Do You Go? Efficient Searching for Indirect Evidence
Hawkins, N., Scott, D. A., & Woods, B. S. (2009) · Medical Decision Making · doi:10.1177/0272989x08330120 · Cited by 35
Evaluation of inconsistency in networks of interventions
Veroniki, A. A., Vasiliadis, H. S., Higgins, J. P. T., & Salanti, G. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dyt106 · Cited by 22
A simple regression model for network meta-analysis
Kessels, A. G. H., ter Riet, G., Puhan, M. A., Kleijnen, J., Bachmann, L., & Minder, C. E. (2013) · OA Epidemiology · doi:10.13172/2053-079x-1-1-690 · Cited by 12
Self-citation
Cited by
Sorted by citations
Hutton, B., Salanti, G., Caldwell, D. M., Chaimani, A., Schmid, C. H., Cameron, C., Ioannidis, J. P. A., Straus, S., Thorlund, K., Jansen, J. P., et al. (2015) · Annals of Internal Medicine · doi:10.7326/m14-2385 · Cited by 8,269
CINeMA: an approach for assessing confidence in the results of a network meta-analysis
Nikolakopoulou, A., Higgins, J. P. T., Papakonstantinou, T., Chaimani, A., Del Giovane, C., Egger, M., & Salanti, G. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003082 · Cited by 1,935
Schünemann, H. J., Cuello, C., Akl, E. A., Mustafa, R. A., Meerpohl, J. J., Thayer, K., Morgan, R. L., Gartlehner, G., Kunz, R., Katikireddi, S. V., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.012 · Cited by 911
Self-citationKhera, R., Murad, M. H., Chandar, A. K., Dulai, P. S., Wang, Z., Prokop, L. J., Loomba, R., Camilleri, M., & Singh, S. (2016) · JAMA · doi:10.1001/jama.2016.7602 · Cited by 865
Self-citationNetwork meta-analysis: an introduction for clinicians
Rouse, B., Chaimani, A., & Li, T. (2017) · Internal and Emergency Medicine · doi:10.1007/s11739-016-1583-7 · Cited by 875
“There are two proposed approaches for applying GRADE to NMA [33, 34].”
Self-citation
Metadata
- Authors
- Puhan, M. A., Schunemann, H. J., Murad, M. H., Li, T., Brignardello-Petersen, R., Singh, J. A., Kessels, A. G., Guyatt, G. H., & for the GRADE Working Group
- Affiliations
- University of Zurich (CH)
- Mayo Clinic (US)
- Johns Hopkins University (US)
- University of Alabama (US)
- Maastricht University (NL)
- McMaster University (CA)
- Corresponding author
- Puhan, M. A.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2014-09-24
- Volume
- 349
- Issue
- sep24 5
- Pages
- g5630-g5630
- ISSN
- 1756-1833 (ISSN-L)
- OpenAlex
- W2155798268
- Cited by
- 1,812
- References
- 39
- Citations by year
- 2026: 160 · 2025: 177 · 2024: 216 · 2023: 175 · 2022: 205 · 2021: 276 · 2020: 201 · 2019: 128 · 2018: 96 · 2017: 72 · 2016: 63 · 2015: 36 · 2014: 4
- FWCI
- 57.15
- Type
- journal article
- Language
- en
- Update notices
- correction (2015-06-17)
- Open access
- bronze
- doi.org
- Locations
- BMJ (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, other-oa)
- HighWire Press Open Archive (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Zurich Open Repository and Archive (University of Zurich)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- network meta-analysis, quality of evidence, GRADE, treatment effect estimates, comparative effectiveness
- MeSH terms
- Humans, Quality Assurance, Health Care, Meta-Analysis as Topic, Practice Guidelines as Topic, Evidence-Based Medicine, Guideline Adherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
