Common pitfalls and mistakes in the set-up, analysis and interpretation of results in network meta-analysis: what clinicians should look for in a published article
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Objective: Several tools have been developed to evaluate the extent to which the findings from a network meta-analysis would be valid; however, applying these tools is a time-consuming task and often requires specific expertise. Clinicians have little time for critical appraisal, and they need to understand the key elements that help them select network meta-analyses that deserve further attention, optimising time and resources. This paper is aimed at providing a practical framework to assess the methodological robustness and reliability of results from network meta-analysis. Methods: As a working example, we selected a network meta-analysis about drug treatments for generalised anxiety disorder, which was published in 2011 in the British Medical Journal. The same network meta-analysis was previously used to illustrate the potential of this methodology in a methodological paper published in JAMA. Results: We reanalysed the 27 studies included in this network following the methods reported in the original article and compared our findings with the published results. We showed how different methodological approaches and the presentation of results can affect conclusions from network meta-analysis. We divided our results into three sections, according to the specific issues that should always be addressed in network meta-analysis: (1) understanding the evidence base, (2) checking the statistical analysis and (3) checking the reporting of findings. Conclusions: The validity of the results from network meta-analysis depends on the plausibility of the transitivity assumption. The risk of bias introduced by limitations of individual studies must be considered first and judgement should be used to infer about the plausibility of transitivity. Inconsistency exists when treatment effects from direct and indirect evidence are in disagreement. Unlike transitivity, inconsistency can be always evaluated statistically, and it should be specifically investigated and reported in the published paper. Network meta-analysis allows researchers to list treatments in preferential order; however, in this paper we demonstrated that rankings could be misleading if based on the probability of being the best. Clinicians should always be interested in the effect sizes rather than the naive rankings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Salanti, G., Ades, A. E., & Ioannidis, J. P. A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.016 · Cited by 3,766
Self-citationInterpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884
Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies
Higgins, J. P. T., Jackson, D., Barrett, J. K., Lu, G., Ades, A. E., & White, I. R. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1044 · Cited by 2,098
Bucher, H. C., Guyatt, G. H., Griffith, L. E., & Walter, S. D. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00049-8 · Cited by 2,052
Cipriani, A., Furukawa, T. A., Salanti, G., Geddes, J. R., Higgins, J. P., Churchill, R., Watanabe, N., Nakagawa, A., Omori, I. M., McGuire, H., et al. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60046-5 · Cited by 1,654
Self-citationEvaluating the quality of evidence from a network meta-analysis
Salanti, G., Del Giovane, C., Chaimani, A., Caldwell, D. M., & Higgins, J. P. T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099682 · Cited by 1,402
Self-citationDias, S., Sutton, A. J., Ades, A. E., & Welton, N. J. (2013) · Medical Decision Making · doi:10.1177/0272989x12458724 · Cited by 1,207
Network meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206
Conceptual and technical challenges in network meta-analysis
Cipriani, A., Higgins, J. P. T., Geddes, J. R., & Salanti, G. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-159-2-201307160-00008 · Cited by 1,072 · cited 1×
“We divided our results into three sections, according to the specific issues that should always be addressed in network meta-analysis: (1) understanding the evidence base, (2) checking the statistical analysis and (3) checking the reporting of findings.”
Self-citationDemystifying trial networks and network meta-analysis
Mills, E. J., Thorlund, K., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f2914 · Cited by 742 · cited 1×
“We divided our results into three sections, according to the specific issues that should always be addressed in network meta-analysis: (1) understanding the evidence base, (2) checking the statistical analysis and (3) checking the reporting of findings.”
Jansen, J. P., & Naci, H. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-159 · Cited by 656
Association Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses
Dechartres, A., Altman, D. G., Trinquart, L., Boutron, I., & Ravaud, P. (2014) · JAMA · doi:10.1001/jama.2014.8166 · Cited by 233
A primer on network meta-analysis with emphasis on mental health
Mavridis, D., Giannatsi, M., Cipriani, A., & Salanti, G. (2015) · Evidence Based Mental Health · doi:10.1136/eb-2015-102088 · Cited by 198
Self-citationHutton, B., Salanti, G., Chaimani, A., Caldwell, D. M., Schmid, C., Thorlund, K., Mills, E., Catalá-López, F., Turner, L., Altman, D. G., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0092508 · Cited by 105
Self-citation
Cited by
Network meta-analysis: application and practice using Stata
Shim, S., Yoon, B.-H., Shin, I.-S., & Bae, J.-M. (2017) · Epidemiology and Health · doi:10.4178/epih.e2017047 · Cited by 656
Cuijpers, P., Noma, H., Karyotaki, E., Cipriani, A., & Furukawa, T. A. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.0268 · Cited by 539
Self-citationSystemic pharmacological treatments for chronic plaque psoriasis: a network meta-analysis
Sbidian, E., Chaimani, A., Garcia-Doval, I., Do, G., Hua, C., Mazaud, C., Droitcourt, C., Hughes, C., Ingram, J. R., Naldi, L., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011535.pub2 · Cited by 433
Self-citationCuijpers, P., Quero, S., Noma, H., Ciharova, M., Miguel, C., Karyotaki, E., Cipriani, A., Cristea, I. A., & Furukawa, T. A. (2021) · World Psychiatry · doi:10.1002/wps.20860 · Cited by 388
Self-citationLack of evidence to favor specific preventive interventions in psychosis: a network meta‐analysis
Davies, C., Cipriani, A., Ioannidis, J. P. A., Radua, J., Stahl, D., Provenzani, U., McGuire, P., & Fusar‐Poli, P. (2018) · World Psychiatry · doi:10.1002/wps.20526 · Cited by 229
Self-citation
Metadata
- Authors
- Chaimani, A., Salanti, G., Leucht, S., Geddes, J. R., & Cipriani, A.
- Venue
- Evidence Based Mental Health
- Publisher
- BMJ
- Published
- 2017-07-24
- Volume
- 20
- Issue
- 3
- Pages
- 88-94
- ISSN
- 1362-0347 (ISSN-L)
- 1468-960X
- OpenAlex
- W2737074968
- Cited by
- 106
- References
- 14
- FWCI
- 8.43
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.