Assessing Confidence in the Results of Network Meta-Analysis (Cinema)
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Evaluation of the credibility of results from a meta-analysis has become an intrinsic part of the evidence synthesis process. We present a methodological framework to evaluate Confidence In the results from Network Meta-Analysis (CINeMA) when multiple interventions are compared. CINeMA considers six domains and we outline the methods used to form judgements about within-study bias, across-studies bias, indirectness, imprecision, heterogeneity and incoherence. Key to judgements about within-study bias and indirectness is the percentage contribution matrix, which shows how much information each study contributes to the results from network meta-analysis. The use of contribution matrix allows the semi-automation of the process, implemented in a freely available web application ( cinema.ispm.ch ). In evaluating imprecision, heterogeneity and inconsistency we consider the impact of these components of variability in forming clinical decisions. Via three examples, we show that CINeMA improves transparency and avoids the selective use of evidence when forming judgements, thus limiting subjectivity in the process. CINeMA is easy to apply even in large and complicated networks, like a network involving 18 different antidepressant drugs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Cipriani, A., Furukawa, T. A., Salanti, G., Chaimani, A., Atkinson, L. Z., Ogawa, Y., Leucht, S., Ruhe, H. G., Turner, E. H., Higgins, J. P. T., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(17)32802-7 · Cited by 3,541
Self-citationGRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Checking consistency in mixed treatment comparison meta-analysis
Dias, S., Welton, N. J., Caldwell, D. M., & Ades, A. E. (2010) · Statistics in Medicine · doi:10.1002/sim.3767 · Cited by 2,521 · cited 1×
“splitting” (39)) is an example of the first set of methods, which are often referred to as local 403 methods.”
MethodPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147 · cited 1×
“expressed in a prediction interval, which shows where the true effect of a new study similar 347 to the existing studies is expected to lie (28).”
BackgroundGRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819
Puhan, M. A., Schunemann, H. J., Murad, M. H., Li, T., Brignardello-Petersen, R., Singh, J. A., Kessels, A. G., Guyatt, G. H., & for the GRADE Working Group (2014) · BMJ · doi:10.1136/bmj.g5630 · Cited by 1,812 · cited 5×
“Direct comparisons (number of studies) CCTA vs Exercise ECG (1) CCTA vs SPECT-MPI (2) CCTA vs Standard care (7) CMR vs Standard care (2) Exercise ECG vs Standard care (1) Exercise ECG vs Stress Echo (4) SPECT-MPI vs Standard care (2) Standard care vs Stress Echo (1)”
BackgroundMethodInfluentialSimultaneous comparison of multiple treatments: Combining direct and indirect evidence.
Caldwell, D. M., Ades, A. E., & Higgins, J. P. T. (2005) · BMJ · doi:10.1136/bmj.331.7521.897 · Cited by 1,600 · cited 1×
“BACKGROUND AND DEFINITIONS 267 One of the key advantages of network meta-analysis compared to pairwise meta268 analysis is the ability to gain precision (36): adding indirect evidence on a particular 269 treatment comparison on top of direct evidence leads to narrower confidence intervals than 270 using the direct evi…”
BackgroundSelf-citationEvaluating the quality of evidence from a network meta-analysis
Salanti, G., Del Giovane, C., Chaimani, A., Caldwell, D. M., & Higgins, J. P. T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099682 · Cited by 1,402 · cited 3×
“CINeMA is based on 53 the GRADE framework, with several conceptual and semantic differences (5).”
BackgroundMethodSelf-citationTurner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837
Self-citationChaimani, A., & Salanti, G. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.57 · Cited by 521 · cited 1×
“trial registries do not indicate important discrepancies with published reports 209 - Empirical examination of patterns of results between small and large studies, using the 210 comparison-adjusted funnel plot (31,32), regression models (33) or selection models 211 (34) do not indicate that results from small studies…”
ResultSelf-citationJansen, J. P., Trikalinos, T., Cappelleri, J. C., Daw, J., Andes, S., Eldessouki, R., & Salanti, G. (2014) · Value in Health · doi:10.1016/j.jval.2014.01.004 · Cited by 319
Self-citationNaci, H., Brugts, J., & Ades, T. (2013) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.111.000071 · Cited by 292 · cited 1×
“We illustrate the methods using three examples: a network of trials that compare 74 outcomes of various diagnostic strategies in patients with suspected acute coronary 75 syndrome (11), a network of trials comparing the effectiveness of 18 antidepressants for 76 major depression (12), and a network comparing adverse e…”
MethodPage, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281
Furukawa, T. A., Salanti, G., Atkinson, L. Z., Leucht, S., Ruhe, H. G., Turner, E. H., Chaimani, A., Ogawa, Y., Takeshima, N., Hayasaka, Y., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010919 · Cited by 214
Self-citationUse of network meta-analysis in clinical guidelines
Kanters, S., Ford, N., Druyts, E., Thorlund, K., Mills, E. J., & Bansback, N. (2016) · Bulletin of the World Health Organization · doi:10.2471/blt.16.174326 · Cited by 121
Research works printed but not linked: 39
- 10.1186/s12916-016-0764-6 · DOI · Paper page
- 10.1016/j.jclinepi.2010.04.026 · DOI · Paper page
- 2011, Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2, ValueHealth, 14, 429
- 2013, Evidence synthesis for decision making 7: a reviewer’s checklist, MedDecisMaking, 33, 679
- 2013, Evidence synthesis for decision making 2: a generalized linear modeling framework for pairwise and network meta-analysis of randomized controlled trials, Med Decis Mak Int J Soc Med Decis Mak, 33, 607
- 10.1136/bmj.k504 · DOI · Paper page
- 10.1136/bmj.d5928 · DOI · Paper page
- 10.1016/j.jclinepi.2017.10.005 · DOI · Paper page
- 2018, Estimating the contribution of studies in network meta-analysis: paths, flows and streams, F1000Research, 7, 610, 10.12688/f1000research.14770.1 · DOI · Paper page
- 10.1371/journal.pone.0003081 · DOI · Paper page
- 10.1371/journal.pone.0066844 · DOI · Paper page
- Scherer RW , Langenberg P , von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000005. · DOI · Paper page
- 10.1136/bmj.f5248 · DOI · Paper page
- 10.1136/bmj.315.7109.640 · DOI · Paper page
- 10.1258/jrsm.2011.11k042 · DOI
- 10.1136/bmj.326.7400.1171 · DOI · Paper page
- 10.1056/NEJMsa065779 · DOI · Paper page
- 2015, Visualizing assumptions and results in network meta-analysis: The network graphs package, 15, 905
- Mavridis D , Efthimiou O , Leucht S , Salanti G. Publication bias and small-study effects magnified effectiveness of antipsychotics but their relative ranking remained invariant. J Clin Epidemiol. 2015 Jun 5; · DOI
- 10.1002/sim.5494 · DOI · Paper page
- 10.1016/j.jclinepi.2011.04.014 · DOI · Paper page
- 10.1161/CIRCOUTCOMES.111.000071 · DOI · Paper page
- 10.1016/j.jclinepi.2011.03.017 · DOI · Paper page
- 2013, Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials, Med Decis Mak Int J Soc Med Decis Mak, 33, 641
- 10.1002/jrsm.1045 · DOI · Paper page
- 10.1002/jrsm.1044 · DOI · Paper page
- 10.1016/j.jclinepi.2014.08.012 · DOI · Paper page
- 10.1198/016214505000001302 · DOI · Paper page
- 2009, Modeling between-trial variance structure in mixed treatment comparisons, Biostat Oxf Engl, 10, 792
- 10.1016/S0895-4356(97)00049-8 · DOI · Paper page
- Dias S , Welton NJ , Sutton AJ , Caldwell DM , Lu G , Ades AE. Inconsistency in Networks of Evidence Based on Randomised Controlled Trials [Internet]. London: National Institute for Health and Care Excellence (NICE); 2014. (NICE Decision Support Unit Technical Support Documents). Available from: http://www.ncbi.nlm.nih.gov/books/NBK310372/
- 10.1186/1471-2288-14-106 · DOI
- 10.1186/1471-2288-12-138 · DOI · Paper page
- 10.1371/journal.pone.0099682 · DOI · Paper page
- 10.1016/j.jclinepi.2017.10.005 · DOI · Paper page
- 10.1136/bmj.g5630 · DOI · Paper page
- 2016, A threshold analysis assessed the credibility of conclusions from network meta-analysis, J Clin Epidemiol, 80, 68, 10.1016/j.jclinepi.2016.07.003 · DOI
- 10.1016/j.jclinepi.2016.11.002 · DOI
- 10.1186/s12916-016-0764-6 · DOI · Paper page
Cited by
Huhn, M., Nikolakopoulou, A., Schneider-Thoma, J., Krause, M., Samara, M., Peter, N., Arndt, T., Bäckers, L., Rothe, P., Cipriani, A., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(19)31135-3 · Cited by 1,753
Self-citationPillinger, T., McCutcheon, R. A., Vano, L., Mizuno, Y., Arumuham, A., Hindley, G., Beck, K., Natesan, S., Efthimiou, O., Cipriani, A., et al. (2020) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30416-x · Cited by 1,090
Papakonstantinou, T., Nikolakopoulou, A., Higgins, J. P. T., Egger, M., & Salanti, G. (2020) · Campbell Systematic Reviews · doi:10.1002/cl2.1080 · Cited by 596
Self-citationYildiz, A., Siafis, S., Mavridis, D., Vieta, E., & Leucht, S. (2023) · The Lancet Psychiatry · doi:10.1016/s2215-0366(23)00199-2 · Cited by 104
Boutron, I., Chaimani, A., Devane, D., Meerpohl, J. J., Rada, G., Hróbjartsson, A., Tovey, D., Grasselli, G., & Ravaud, P. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013769 · Cited by 41
Self-citation
Metadata
- Authors
- Nikolakopoulou, A., Higgins, J. P., Papakonstantinou, T., Chaimani, A., Del Giovane, C., Egger, M., & Salanti, G.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-04-05
- OpenAlex
- W2926978124
- Cited by
- 49
- References
- 16
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.