Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.

Evaluating the Quality of Evidence from a Network Meta-Analysis

Open access

Abstract

Systematic reviews that collate data about the relative effects of multiple interventions via network meta-analysis are highly informative for decision-making purposes. A network meta-analysis provides two types of findings for a specific outcome: the relative treatment effect for all pairwise comparisons, and a ranking of the treatments. It is important to consider the confidence with which these two types of results can enable clinicians, policy makers and patients to make informed decisions. We propose an approach to determining confidence in the output of a network meta-analysis. Our proposed approach is based on methodology developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group for pairwise meta-analyses. The suggested framework for evaluating a network meta-analysis acknowledges (i) the key role of indirect comparisons (ii) the contributions of each piece of direct evidence to the network meta-analysis estimates of effect size; (iii) the importance of the transitivity assumption to the validity of network meta-analysis; and (iv) the possibility of disagreement between direct evidence and indirect evidence. We apply our proposed strategy to a systematic review comparing topical antibiotics without steroids for chronically discharging ears with underlying eardrum perforations. The proposed framework can be used to determine confidence in the results from a network meta-analysis. Judgements about evidence from a network meta-analysis can be different from those made about evidence from pairwise meta-analyses.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0099682

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Network of topical antibiotics without steroids for chronically discharging ears. Edges are weighted according to the inverse of the variance of the direct summary ln(OR) (presented along the edges) and nodes are weighted according to the number of studies.
Figure 1 Network of topical antibiotics without steroids for chronically discharging ears. Edges are weighted according to the inverse of the variance of the direct summary ln(OR) (presented along the edges) and nodes are weighted according to the number of studies.CC BY · PubMed Central
Figure 2 Contributions matrix: percentage contribution of each direct estimate to the network meta-analysis estimates. Rows correspond to network meta-analysis ORs (separated for mixed and indirect evidence) and columns correspond to direct meta-analysis ORs. The contribution of each direct comparison to the total network evidence that provides the ranking of the treatments is presented separately (row named Entire network). The sizes of the boxes are proportional to the percentage contribution of each direct estimate to the network meta-analysis estimates (rows 1–6) and to the entire network (row 7). The last row shows the number of included direct comparisons. The names of the treatments are given in Figure 1.
Figure 2 Contributions matrix: percentage contribution of each direct estimate to the network meta-analysis estimates. Rows correspond to network meta-analysis ORs (separated for mixed and indirect evidence) and columns correspond to direct meta-analysis ORs. The contribution of each direct comparison to the total network evidence that provides the ranking of the treatments is presented separately (row named Entire network). The sizes of the boxes are proportional to the percentage contribution of each direct estimate to the network meta-analysis estimates (rows 1–6) and to the entire network (row 7). The last row shows the number of included direct comparisons. The names of the treatments are given in Figure 1.CC BY · PubMed Central
Figure 3 Study limitations for each network estimate for pairwise comparisons of topical antibiotics. Calculations are based on the contributions of direct evidence. The colours represent the risk of bias (green: low, yellow: moderate, red: high). The initial judgements about the risk of bias in the direct estimates are shown on the right side of the figure (there is no direct evidence for AC). The names of the treatments are given in Figure 1.
Figure 3 Study limitations for each network estimate for pairwise comparisons of topical antibiotics. Calculations are based on the contributions of direct evidence. The colours represent the risk of bias (green: low, yellow: moderate, red: high). The initial judgements about the risk of bias in the direct estimates are shown on the right side of the figure (there is no direct evidence for AC). The names of the treatments are given in Figure 1.CC BY · PubMed Central
Figure 4 Network estimates of mean ORs, their 95% confidence intervals and 95% predictive intervals (red extensions). The names of the treatments are given in Figure 1.
Figure 4 Network estimates of mean ORs, their 95% confidence intervals and 95% predictive intervals (red extensions). The names of the treatments are given in Figure 1.CC BY · PubMed Central
Figure 5 Study limitations weighted by contribution of direct estimates to the network of topical antibiotics. The colours represent the risk of bias (green: low, yellow: moderate, red: high). The names of the treatments are given in Figure 1.
Figure 5 Study limitations weighted by contribution of direct estimates to the network of topical antibiotics. The colours represent the risk of bias (green: low, yellow: moderate, red: high). The names of the treatments are given in Figure 1.CC BY · PubMed Central
Figure 6 Rankograms for topical antibiotics without steroids for chronically discharging ears. On the horizontal axes are the possible ranks and on the vertical axis the probability that each treatment achieves each rank.
Figure 6 Rankograms for topical antibiotics without steroids for chronically discharging ears. On the horizontal axes are the possible ranks and on the vertical axis the probability that each treatment achieves each rank.CC BY · PubMed Central
Figure 7 Comparison-adjusted funnel plot for the network of topical antibiotics without steroids for chronically discharging ears. Each observation is the difference between a study estimate and its direct meta-analysis mean effect. Studies on the right hand side ‘overestimate’ the effect of newer treatments.
Figure 7 Comparison-adjusted funnel plot for the network of topical antibiotics without steroids for chronically discharging ears. Each observation is the difference between a study estimate and its direct meta-analysis mean effect. Studies on the right hand side ‘overestimate’ the effect of newer treatments.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

9 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 1,402

Metadata

Authors
Salanti, G., Del Giovane, C., Chaimani, A., Caldwell, D. M., & Higgins, J. P. T.
Affiliations
  • University of Ioannina (GR)
  • University of Bristol (GB)
Corresponding author
Higgins, Julian P. T.
Venue
PLoS ONE
Published
2014-07-03
Volume
9
Issue
7
Pages
e99682
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2074147400
Cited by
1,402
References
32
Citations by year
2026: 116 · 2025: 181 · 2024: 150 · 2023: 144 · 2022: 151 · 2021: 170 · 2020: 152 · 2019: 115 · 2018: 96 · 2017: 56 · 2016: 46 · 2015: 17 · 2014: 8
FWCI
41.47
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • Bristol Research (University of Bristol) (submittedVersion)
Funders
  • Medical Research Council (MR/K025643/1, MR/K023233/1, G0902118)
  • FP7 Ideas: European Research Council (260559)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
network meta-analysis, quality of evidence, indirect comparison, transitivity assumption, direct and indirect evidence, treatment ranking, GRADE framework, topical antibiotics, chronic otitis media, tympanic membrane perforation
MeSH terms
Humans, Models, Theoretical, Meta-Analysis as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.