ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Background: Selective outcome reporting and publication bias threaten the validity of systematic reviews and meta-analyses and can affect clinical decision-making. A rigorous method to evaluate the impact of this bias on the results of network meta-analyses of interventions is lacking. We present a tool to assess the Risk Of Bias due to Missing Evidence in Network meta-analysis (ROB-MEN). Methods: ROB-MEN first evaluates the risk of bias due to missing evidence for each of the possible pairwise comparison that can be made between the interventions in the network. This step considers possible bias due to the presence of studies with unavailable results ( within-study assessment of bias ) and the potential for unpublished studies ( across-study assessment of bias ). The second step combines the judgements about the risk of bias due to missing evidence in pairwise comparisons with (i) the contribution of direct comparisons to the network meta-analysis estimates, (ii) possible small-study effects evaluated by network meta-regression, and (iii) any bias from unobserved comparisons. Then, a level of “low risk”, “some concerns”, or “high risk” for the bias due to missing evidence is assigned to each estimate, which is our tool’s final output. Results: We describe the methodology of ROB-MEN step-by-step using an illustrative example from a published NMA of non-diagnostic modalities for the detection of coronary artery disease in patients with low risk acute coronary syndrome. We also report a full application of the tool on a larger and more complex published network of 18 drugs from head-to-head studies for the acute treatment of adults with major depressive disorder. Conclusions: ROB-MEN is the first tool for evaluating the risk of bias due to missing evidence in network meta-analysis and applies to networks of all sizes and geometry. The use of ROB-MEN is facilitated by an R Shiny web application that produces the Pairwise Comparisons and ROB-MEN Table and is incorporated in the reporting bias domain of the CINeMA framework and software.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
Self-citationROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationSterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · cited 1×
“tion bias [2, 5, 8] in pairwise meta-analysis, as summarised in the Cochrane Handbook for Systematic”
MethodSelf-citationFunnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Self-citationCipriani, A., Furukawa, T. A., Salanti, G., Chaimani, A., Atkinson, L. Z., Ogawa, Y., Leucht, S., Ruhe, H. G., Turner, E. H., Higgins, J. P. T., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(17)32802-7 · Cited by 3,541
Self-citationGraphical tools for network meta-analysis in STATA
Chaimani, A., Higgins, J. P. T., Mavridis, D., Spyridonos, P., & Salanti, G. (2013) · PLoS ONE · doi:10.1371/journal.pone.0076654 · Cited by 2,473
Self-citationSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267
Self-citationPeters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · cited 1×
“For comparisons with at least 10 studies (in the first column in Table 2), judgements can additionally consider statistical techniques such as contour-enhanced funnel plots [4], meta-regression models and statistical tests for small-study effects [2, 6, 7, 27–29], or selection models for pairwise meta-analysis (e.”
BackgroundCINeMA: an approach for assessing confidence in the results of a network meta-analysis
Nikolakopoulou, A., Higgins, J. P. T., Papakonstantinou, T., Chaimani, A., Del Giovane, C., Egger, M., & Salanti, G. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003082 · Cited by 1,935
Self-citation
Cited by
Page, M. J., Sterne, J. A. C., Boutron, I., Hróbjartsson, A., Kirkham, J. J., Li, T., Lundh, A., Mayo-Wilson, E., McKenzie, J. E., Stewart, L. A., et al. (2023) · BMJ · doi:10.1136/bmj-2023-076754 · Cited by 134
“14 The ROB-ME tool is not intended to examine a related source of bias, which we describe as the risk of bias in selection of the reported result.”
MethodSelf-citationVeroniki, A. A., Hutton, B., Stevens, A., McKenzie, J. E., Page, M. J., Moher, D., McGowan, J., Straus, S. E., Li, T., Munn, Z., et al. (2025) · JBI Evidence Synthesis · doi:10.11124/jbies-24-00308 · Cited by 110
Self-citationDong, S., Schneider-Thoma, J., Bighelli, I., Siafis, S., Wang, D., Burschinski, A., Schestag, K., Samara, M., & Leucht, S. (2024) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-023-01654-2 · Cited by 68
Efficacy and Safety of Drugs for Gastroparesis: Systematic Review and Network Meta-analysis
Ingrosso, M. R., Camilleri, M., Tack, J., Ianiro, G., Black, C. J., & Ford, A. C. (2023) · Gastroenterology · doi:10.1053/j.gastro.2022.12.014 · Cited by 53
Liao, C.-D., Huang, S.-W., Chen, H.-C., Huang, M.-H., Liou, T.-H., & Lin, C.-L. (2024) · Nutrients · doi:10.3390/nu16070941 · Cited by 45
“The downgrading level for the study limitation was based on the amounts of contribution of each direct estimate and corresponding risk of bias to the NMA estimates [86,87].”
Background
Metadata
- Authors
- Chiocchia, V., Nikolakopoulou, A., Higgins, J. P. T., Page, M. J., Papakonstantinou, T., Cipriani, A., Furukawa, T. A., Siontis, G. C. M., Egger, M., & Salanti, G.
- Venue
- BMC Medicine
- Published
- 2021-11-23
- Volume
- 19
- Issue
- 1
- Pages
- 304-304
- ISSN
- 1741-7015 (ISSN-L)
- OpenAlex
- W3215732646
- Cited by
- 129
- References
- 18
- FWCI
- 8.76
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis. https://app.cochat.ai/papers/10.1186/s12916-021-02166-3
- The paper
- Chiocchia, V., Nikolakopoulou, A., Higgins, J. P. T., Page, M. J., Papakonstantinou, T., Cipriani, A., Furukawa, T. A., Siontis, G. C. M., Egger, M., & Salanti, G. (2021). ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis. BMC Medicine. https://doi.org/10.1186/s12916-021-02166-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.