Prevalence of evidence of inconsistency and its association with network structural characteristics in 201 published networks of interventions
Open accessAbstract
BACKGROUND: Network meta-analysis (NMA) has attracted growing interest in evidence-based medicine. Consistency between different sources of evidence is fundamental to the reliability of the NMA results. The purpose of the present study was to estimate the prevalence of evidence of inconsistency and describe its association with different NMA characteristics. METHODS: We updated our collection of NMAs with articles published up to July 2018. We included networks with randomised clinical trials, at least four treatment nodes, at least one closed loop, a dichotomous primary outcome, and available arm-level data. We assessed consistency using the design-by-treatment interaction (DBT) model and testing all the inconsistency parameters globally through the Wald-type chi-squared test statistic. We estimated the prevalence of evidence of inconsistency and its association with different network characteristics (e.g., number of studies, interventions, intervention comparisons, loops). We evaluated the influence of the network characteristics on the DBT p-value via a multivariable regression analysis and the estimated Pearson correlation coefficients. We also evaluated heterogeneity in NMA (consistency) and DBT (inconsistency) random-effects models. RESULTS: We included 201 published NMAs. The p-value of the design-by-treatment interaction (DBT) model was lower than 0.05 in 14% of the networks and lower than 0.10 in 20% of the networks. Networks including many studies and comparing few interventions were more likely to have small DBT p-values (less than 0.10), which is probably because they yielded more precise estimates and power to detect differences between designs was higher. In the presence of inconsistency (DBT p-value lower than 0.10), the consistency model displayed higher heterogeneity than the DBT model. CONCLUSIONS: Our findings show that inconsistency was more frequent than what would be expected by chance, suggesting that researchers should devote more resources to exploring how to mitigate inconsistency. The results of this study highlight the need to develop strategies to detect inconsistency (because of the relatively high prevalence of evidence of inconsistency in published networks), and particularly in cases where the existing tests have low power.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Hutton, B., Salanti, G., Caldwell, D. M., Chaimani, A., Schmid, C. H., Cameron, C., Ioannidis, J. P. A., Straus, S., Thorlund, K., Jansen, J. P., et al. (2015) · Annals of Internal Medicine · doi:10.7326/m14-2385 · Cited by 8,269 · see citing works
Self-citationChecking consistency in mixed treatment comparison meta-analysis
Dias, S., Welton, N. J., Caldwell, D. M., & Ades, A. E. (2010) · Statistics in Medicine · doi:10.1002/sim.3767 · Cited by 2,521 · see citing works
Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies
Higgins, J. P. T., Jackson, D., Barrett, J. K., Lu, G., Ades, A. E., & White, I. R. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1044 · Cited by 2,098 · see citing works
Self-citationBucher, H. C., Guyatt, G. H., Griffith, L. E., & Walter, S. D. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00049-8 · Cited by 2,052 · see citing works
CINeMA: an approach for assessing confidence in the results of a network meta-analysis
Nikolakopoulou, A., Higgins, J. P. T., Papakonstantinou, T., Chaimani, A., Del Giovane, C., Egger, M., & Salanti, G. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003082 · Cited by 1,935 · see citing works
Self-citationPuhan, M. A., Schunemann, H. J., Murad, M. H., Li, T., Brignardello-Petersen, R., Singh, J. A., Kessels, A. G., Guyatt, G. H., & for the GRADE Working Group (2014) · BMJ · doi:10.1136/bmj.g5630 · Cited by 1,812 · cited 5× · see citing works
“Puhan MA, Schünemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.”
BackgroundMethodInfluentialSalanti, G. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1037 · Cited by 1,729 · see citing works
Self-citationMethods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685 · see citing works
Self-citationSimultaneous comparison of multiple treatments: Combining direct and indirect evidence.
Caldwell, D. M., Ades, A. E., & Higgins, J. P. T. (2005) · BMJ · doi:10.1136/bmj.331.7521.897 · Cited by 1,600 · see citing works
Self-citationWhite, I. R., Barrett, J. K., Jackson, D., & Higgins, J. P. T. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1045 · Cited by 1,114 · see citing works
Self-citationA comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses
Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1316 · Cited by 1,074 · see citing works
Self-citationConceptual and technical challenges in network meta-analysis
Cipriani, A., Higgins, J. P. T., Geddes, J. R., & Salanti, G. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-159-2-201307160-00008 · Cited by 1,072 · see citing works
Self-citationWhite, I. R. (2015) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x1501500403 · Cited by 778 · see citing works
Self-citationAssessing Evidence Inconsistency in Mixed Treatment Comparisons
Lu, G., & Ades, A. E. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000001302 · Cited by 756 · see citing works
Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis
Brignardello-Petersen, R., Bonner, A., Alexander, P. E., Siemieniuk, R. A., Furukawa, T. A., Rochwerg, B., Hazlewood, G. S., Alhazzani, W., Mustafa, R. A., Murad, M. H., et al. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.005 · Cited by 753 · see citing works
Evidence Synthesis for Decision Making 4
Dias, S., Welton, N. J., Sutton, A. J., Caldwell, D. M., Lu, G., & Ades, A. E. (2013) · Medical Decision Making · doi:10.1177/0272989x12455847 · Cited by 734 · see citing works
Jansen, J. P., & Naci, H. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-159 · Cited by 656 · see citing works
Evaluation of inconsistency in networks of interventions
Veroniki, A. A., Vasiliadis, H. S., Higgins, J. P., & Salanti, G. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dys222 · Cited by 576 · see citing works
Self-citationTurner, R. M., Jackson, D., Wei, Y., Thompson, S. G., & Higgins, J. P. T. (2015) · Statistics in Medicine · doi:10.1002/sim.6381 · Cited by 410 · see citing works
Self-citationWelton, N. J., Caldwell, D. M., Adamopoulos, E., & Vedhara, K. (2009) · American Journal of Epidemiology · doi:10.1093/aje/kwp014 · Cited by 362 · see citing works
Nikolakopoulou, A., White, I. R., & Salanti, G. (2020) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315119403-10 · Cited by 307 · see citing works
Self-citationSong, F., Xiong, T., Bhurke, S., Loke, Y. K., Sutton, A. J., Eastwood, A. J., Holland, R., Chen, Y., Glenny, A., Deeks, J. J., et al. (2011) · BMJ · doi:10.1136/bmj.d4909 · Cited by 199 · see citing works
Modeling between-trial variance structure in mixed treatment comparisons
Lu, G., & Ades, A. E. (2009) · Biostatistics · doi:10.1093/biostatistics/kxp032 · Cited by 196 · see citing works
Song, F., Harvey, I. M., & Lilford, R. J. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.06.006 · Cited by 195 · see citing works
Indirect Comparisons: A Review of Reporting and Methodological Quality
Donegan, S., Williamson, P. R., Gamble, C. L., & Smith, C. T. (2010) · PLoS ONE · doi:10.1371/journal.pone.0011054 · Cited by 146 · see citing works
Caldwell, D. M., Welton, N. J., & Ades, A. E. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.08.025 · Cited by 135 · see citing works
Πετροπούλου, Μ., Nikolakopoulou, A., Veroniki, A. A., Rios, P. D. L., Vafaei, A., Zarin, W., Gianatsi, M., Sullivan, S. M., Tricco, A. C., Chaimani, A., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.11.002 · Cited by 135 · see citing works
Self-citationBafeta, A., Trinquart, L., Séror, R., & Ravaud, P. (2013) · BMJ · doi:10.1136/bmj.f3675 · Cited by 106 · see citing works
Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review
Zarin, W., Veroniki, A. A., Nincic, V., Vafaei, A., Reynen, E., Motiwala, S. S., Antony, J., Sullivan, S. M., Rios, P. D. L., Daly, C., et al. (2017) · BMC Medicine · doi:10.1186/s12916-016-0764-6 · Cited by 94 · see citing works
Self-citationSong, F., Clark, A., Bachmann, M. O., & Maas, J. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-138 · Cited by 92 · see citing works
Veroniki, A. A., Mavridis, D., Higgins, J. P. T., & Salanti, G. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-106 · Cited by 79 · see citing works
Self-citationVeroniki, A. A., Tsokani, S., Zevgiti, S., Pagkalidou, I., Kontouli, K.-M., Ambarcioglu, P., Pandis, N., Lunny, C., Nikolakopoulou, A., Papakonstantinou, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01780-9 · Cited by 53 · see citing works
Self-citationNetwork Meta-Analysis using Frequentist Methods [R package netmeta version 1.3-0]
Rücker, G., Krahn, U., König, J., Efthimiou, O., Davies, A. L., Papakonstantinou, T., & Schwarzer, G. (2021) · Cited by 40 · see citing works
Self-citationnmadb: Network Meta-Analysis Database API
Papakonstantinou, T. (2019) · doi:10.32614/cran.package.nmadb · Cited by 13 · see citing works
Announcement of the Stata Journal Editors’ Prize 2015
Newton, H. J., & Cox, N. J. (2015) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x1501500101 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Twenty years of network meta‐analysis: Continuing controversies and recent developments
Ades, A. E., Welton, N. J., Dias, S., Phillippo, D., & Caldwell, D. M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1700 · Cited by 103 · see citing works
Papola, D., Ostuzzi, G., Tedeschi, F., Gastaldon, C., Purgato, M., Giovane, C. D., Pompoli, A., Pauley, D., Karyotaki, E., Sijbrandij, M., et al. (2022) · Psychological Medicine · doi:10.1017/s0033291722003683 · Cited by 83 · see citing works
Network meta-analysis: a powerful tool for clinicians, decision-makers, and methodologists
Flórez, I. D., la Cruz-Mena, J. E. D., & Veroniki, A.-A. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111537 · Cited by 68 · see citing works
Self-citationWewege, M. A., Bagg, M. K., Jones, M. D., Ferraro, M. C., Cashin, A. G., Rizzo, R. R. N., Leake, H. B., Hagstrom, A. D., Sharma, S., McLachlan, A. J., et al. (2023) · BMJ · doi:10.1136/bmj-2022-072962 · Cited by 66 · see citing works
Wang, D., Schneider-Thoma, J., Siafis, S., Qin, M., Wu, H., Zhu, Y., Davis, J. M., Priller, J., & Leucht, S. (2024) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2024.03.003 · Cited by 44 · see citing works
“We also applied the design-by-treatment interaction test, again considering p-values less than 0.1 as important(Veroniki et al., 2021).”
MethodInfluentialGraphical tools for visualizing the results of network meta‐analysis of multicomponent interventions
Seitidis, G., Tsokani, S., Christogiannis, C., Kontouli, K., Fyraridis, A., Nikolakopoulos, S., Veroniki, A. A., & Mavridis, D. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1617 · Cited by 15 · see citing works
Self-citationDoleman, B., Jakobsen, J. C., Mathiesen, O., Cooper, N. J., Sutton, A. J., & Hardman, J. G. (2025) · British Journal of Anaesthesia · doi:10.1016/j.bja.2024.12.039 · Cited by 14 · see citing works
Local inconsistency detection using the Kullback–Leibler divergence measure
Spineli, L. M. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02680-4 · Cited by 13 · see citing works
Inconsistency identification in network meta‐analysis via stochastic search variable selection
Seitidis, G., Nikolakopoulos, S., Ntzoufras, I., & Mavridis, D. (2023) · Statistics in Medicine · doi:10.1002/sim.9891 · Cited by 7 · see citing works
Self-citationSpineli, L. M. (2025) · Systematic Reviews · doi:10.1186/s13643-025-02984-z · Cited by 4 · see citing works
Shaikh, M. Y. D., Shaikh, M. D., Hirani, S., Nanote, A., Prasad, R., & Wanjari, M. B. (2023) · Cureus · doi:10.7759/cureus.43104 · Cited by 5 · see citing works
Yoon, I. M., Kim, K. J., Lee, K.-H., Yoo, D.-W., & Oh, H. (2025) · Current Therapeutic Research · doi:10.1016/j.curtheres.2025.100776 · Cited by 4 · see citing works
Two decades of network meta‐analysis: Roadmap to their applications and challenges
Veroniki, A. A., Flórez, I. D., Hutton, B., Straus, S. E., & Tricco, A. C. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1744 · Cited by 4 · see citing works
Self-citationBigdeli, N., Yalfani, A., Doosti‐Irani, A., & Qodrati, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-22574-2 · Cited by 3 · see citing works
Between Rigor and Relevance: Why the EU HTA Guidelines on Indirect Comparisons Miss the Mark
Aballéa, S., Toumi, M., Wojciechowski, P., Clay, E., Falissard, B., Simoens, S., Auquier, P., Capri, S., Ruof, J., Fricke, F.-U., et al. (2026) · Journal of Market Access & Health Policy · doi:10.3390/jmahp14020030 · Cited by 3 · see citing works
“Crying in a cupboard”: Five minutes with . . . Sarah Abrams
Mahase, E. (2023) · BMJ · doi:10.1136/bmj.p708 · Cited by 1 · see citing works
Korsavva, S., Valkova, F. B., & Perez, I. C. (2025) · Geriatric Care · doi:10.4081/gc.2025.13659
Spineli, L. M., Sierra, A. M. G., & Yepes-Núñez, J. J. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02634-x
- Authors
- Veroniki, A. A., Tsokani, S., White, I. R., Schwarzer, G., Rücker, G., Mavridis, D., Higgins, J. P. T., & Salanti, G.
- Affiliations
- St. Michael's Hospital (CA)
- University of Ioannina (GR)
- MRC Clinical Trials Unit at UCL (GB)
- University of Freiburg (DE)
- University of Bristol (GB)
- University of Bern (CH)
- Corresponding author
- Veroniki, Areti Angeliki
- ORCID
- Venue
- BMC Medical Research Methodology
- Published
- 2021-10-25
- Volume
- 21
- Issue
- 1
- Pages
- 224-224
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC8543923
- OpenAlex
- W3153887738
- Cited by
- 18
- References
- 36
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 5 · 2023: 4 · 2022: 2
- FWCI
- 1.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- BORIS (University Library Bern) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- National Science Foundation
- National Institute for Health and Care Research (NIHR200268, NF-SI-0617-10145)
- European Commission (754936, HORIZON2020)
- Deutsche Forschungsgemeinschaft
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (179158, 320030)
- Canadian Institutes of Health Research (MC_UU_12023/21)
- Medical Research Council (MR/K025643/1, MC_UU_00004/06, MC_UU_12023/21)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- network meta-analysis, inconsistency, network structure, network heterogeneity, consistency testing, closed loops, statistical power
- MeSH terms
- Network Meta-Analysis, Network Meta-Analysis As Topic, Network Meta-Analysis as Topic, Humans, Regression Analysis, Reproducibility of Results, Prevalence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
