The DOI matches its Crossref record, with a small difference in title or year; 33 of the 33 references it cites were found in a registry or an index.
ROB-ME: a tool for assessing risk of bias due to missing evidence in systematic reviews with meta-analysis
Open accessAbstract
Various methods are available to help users assess whether selective non-publication of studies or selective non-reporting of study results has occurred, but not its impact on a meta-analysis. This limitation of existing methods leaves users to decide their own approach for judging the risk of bias in a meta-analysis result. In this paper, Page and colleagues describe the ROB-ME (risk of bias due to missing evidence) tool, a structured approach for assessing the risk of bias that arises when entire studies, or particular results within studies, are missing from a meta-analysis because of the P value, magnitude, or direction of the study results. The tool is anticipated to help authors and users of systematic reviews identify meta-analyses at high risk of bias and interpret results appropriately.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We were not able to assess publication bias as none of the outcomes studied included more than 10 studies, which is a pre-requisite according to Cochrane's recommendations.83”
From Interventions to improve adherence to lipid-lowering drugs: a systematic review and meta-analysis (2025) · cites this paper as [83]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
Self-citationThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationSterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Self-citationSynthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline
Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., Thomas, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6890 · Cited by 4,628 · cited 1×
“32 33 We also plan to develop an interactive online version of the tool to facilitate use and prepare a bank of worked examples for educational purposes.”
MethodSelf-citationPeters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053
Self-citationThe GRADE Working Group clarifies the construct of certainty of evidence
Hultcrantz, M., Rind, D., Akl, E. A., Treweek, S., Mustafa, R. A., Iorio, A., Alper, B. S., Meerpohl, J. J., Murad, M. H., Ansari, M. T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.006 · Cited by 853
Self-citationCompleting ‘Summary of findings’ tables and grading the certainty of the evidence
Schünemann, H. J., Higgins, J. P., Vist, G. E., Glasziou, P., Akl, E. A., Skoetz, N., Guyatt, G. H., & on behalf of the Cochrane GRADEing Methods Group (formerly Applicability and Recommendations Methods Group) and the Cochrane Statistical Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch14 · Cited by 669
Self-citationSynthesizing and presenting findings using other methods
McKenzie, J. E., & Brennan, S. E. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch12 · Cited by 423
Self-citationAssessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402
Self-citationDefining the criteria for including studies and how they will be grouped for the synthesis
McKenzie, J. E., Brennan, S., Ryan, R. E., Thomson, H., Johnston, R. V., & Thomas, J. (2019) · Wiley eBooks · doi:10.1002/9781119536604.ch3 · Cited by 391
Self-citationPage, M. J., Sterne, J. A. C., Higgins, J. P. T., & Egger, M. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1468 · Cited by 294
Self-citationPage, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019703 · Cited by 278
Self-citationConsidering bias and conflicts of interest among the included studies
Boutron, I., Page, M. J., Higgins, J. P., Altman, D. G., Lundh, A., Hróbjartsson, A., & on behalf of the Cochrane Bias Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch7 · Cited by 258
Self-citationMoreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., & Ades, A. E. (2009) · BMJ · doi:10.1136/bmj.b2981 · Cited by 184
Self-citationA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Kirkham, J. J., Altman, D. G., Chan, A.-W., Gamble, C., Dwan, K. M., & Williamson, P. R. (2018) · BMJ · doi:10.1136/bmj.k3802 · Cited by 144 · cited 2×
“…(eg, due to a substantial amount of missing data); the outcome was Box 1: Possible scenarios in which study results are missing (adapted from Kirkham et al 6 ) Example scenarios where it is reasonable to suspect, given a lack of explanation from the study investigators, that a result is…”
BackgroundSelf-citationMayo-Wilson, E., Fusco, N., Li, T., Hong, H., Canner, J. K., & Dickersin, K. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.007 · Cited by 131
Self-citationROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis
Chiocchia, V., Nikolakopoulou, A., Higgins, J. P. T., Page, M. J., Papakonstantinou, T., Cipriani, A., Furukawa, T. A., Siontis, G. C. M., Egger, M., & Salanti, G. (2021) · BMC Medicine · doi:10.1186/s12916-021-02166-3 · Cited by 129
Self-citationA proposed framework for developing quality assessment tools
Whiting, P., Wolff, R., Mallett, S., Simera, I., & Savović, J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0604-6 · Cited by 121
Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy
Mayo-Wilson, E., Li, T., Fusco, N., Bertizzolo, L., Canner, J. K., Cowley, T., Doshi, P., Ehmsen, J., Gresham, G., Guo, N., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.07.014 · Cited by 108
Self-citationVEVEA, J. L., COBURN, K., & SUTTON, A. (2019) · The Handbook of Research Synthesis and Meta-Analysis · doi:10.7758/9781610448864.21 · Cited by 96
Self-citationSearching clinical trials registers: guide for systematic reviewers
Hunter, K. E., Webster, A. C., Page, M. J., Willson, M. L., McDonald, S. J., Berber, S., Skeers, P., Tan, A. G., Parkhill, A., & Seidler, A. L. (2022) · BMJ · doi:10.1136/bmj-2021-068791 · Cited by 94
Self-citationMoreno, S. G., Sutton, A. J., Thompson, J. R., Ades, A. E., Abrams, K. R., & Cooper, N. J. (2012) · Statistics in Medicine · doi:10.1002/sim.4488 · Cited by 90
Self-citationGravier, F., Smondack, P., Prieur, G., Médrinal, C., Combret, Y., Muir, J., Baste, J., Cuvelier, A., Boujibar, F., & Bonnevie, T. (2021) · Thorax · doi:10.1136/thoraxjnl-2021-217242 · Cited by 88
Speich, B., Gryaznov, D., Busse, J. W., Gloy, V. L., Lohner, S., Klatte, K., Taji Heravi, A., Ghosh, N., Lee, H., Mansouri, A., et al. (2022) · PLOS Medicine · doi:10.1371/journal.pmed.1003980 · Cited by 69
Self-citationTurner, E. H., Cipriani, A., Furukawa, T. A., Salanti, G., & de Vries, Y. A. (2022) · PLoS Medicine · doi:10.1371/journal.pmed.1003886 · Cited by 68
Self-citationRethinking the assessment of risk of bias due to selective reporting: a cross-sectional study
Page, M. J., & Higgins, J. P. T. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0289-2 · Cited by 60
Self-citationMaier, M., VanderWeele, T. J., & Mathur, M. B. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1256 · Cited by 60
Mueller, K. F., Meerpohl, J. J., Briel, M., Antes, G., von Elm, E., Lang, B., Motschall, E., Schwarzer, G., & Bassler, D. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.015 · Cited by 48
Self-citationThe significance of the trial outcome was associated with publication rate and time to publication
Song, S. Y., Koo, D.-H., Jung, S.-Y., Kang, W., & Kim, E. Y. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.009 · Cited by 36
What to do with a clinical trial with conflicts of interest
Lundh, A., Boutron, I., Stewart, L. A., & Hróbjartsson, A. (2019) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111230 · Cited by 22
Self-citationPollock, A., Campbell, P., Synnot, A., Nunn, J., Hill, S. R., Morris, J. B., Goodare, H., Watts, C., & Morley, R. (2018) · Discovery Research Portal (University of Dundee) · doi:10.1002/14651858.cd201801
Cited by
Sorted by citations
Systematic review and meta-analysis of preclinical studies
Ineichen, B. V., Held, U., Salanti, G., Macleod, M. R., & Wever, K. E. (2024) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00347-x · Cited by 28
Siafis, S., Chiocchia, V., Macleod, M. R., Austin, C., Homiar, A., Tinsdeall, F., Friedrich, C., Ramage, F., Kennett, J., Nomura, N., et al. (2024) · Wellcome Open Research · doi:10.12688/wellcomeopenres.21302.1 · Cited by 24
Triggers, Types, and Treatments for Kounis Syndrome: A Systematic Review
PÉREZ, E., Santaularia-Tomas, M., Martin-Dorantes, M. A., Villarreal-Jimenez, E., Olivera-Mar, A., Sanchez-Felix, E., Pérez-Navarrete, A., Millet-Herrera, J. L., Huchim-Mendez, O., Alejos-Briceño, R., et al. (2025) · Clinics and Practice · doi:10.3390/clinpract15030059 · Cited by 21
Milojevic, M., Freemantle, N., Hayanga, J. W. A., Kelly, R. F., Myers, P. O., Petersen, R. H., Opitz, I., Sabik, J. F., & Bakaeen, F. G. (2024) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2024.07.048 · Cited by 18
Lai, N. M., Lim, Y. S., Chaiyakunapruk, N., Lee, S. W. H., Liabsuetrakul, T., Sabbir, M., Mahmud, S. M. H., Siddiqi, U. R., Ahmed, T., Hasan, F. U. L., et al. (2025) · Child and Adolescent Mental Health · doi:10.1111/camh.70022 · Cited by 30
Metadata
- Authors
- Page, M. J., Sterne, J. A. C., Boutron, I., Hróbjartsson, A., Kirkham, J. J., Li, T., Lundh, A., Mayo-Wilson, E., McKenzie, J. E., Stewart, L. A., Sutton, A. J., Bero, L., Dunn, A. G., Dwan, K., Elbers, R. G., Kanukula, R., Meerpohl, J. J., Turner, E. H., & Higgins, J. P. T.
- Affiliations
- Monash University (AU)
- National Institute for Health and Care Research (GB)
- Inserm (FR)
- University of Southern Denmark (DK)
- Manchester Academic Health Science Centre (GB)
- University of Colorado Anschutz (US)
- Department of Epidemiology, UNC Gillings School of Global Public Health, Chapel Hill, NC, USA
- University of York (GB)
- University of Leicester (GB)
- Center for Bioethics and Humanities, University of Colorado, Aurora, CO, USA
- The University of Sydney (AU)
- Liverpool School of Tropical Medicine (GB)
- Erasmus MC (NL)
- University of Freiburg (DE)
- Oregon Health & Science University (US)
- Corresponding author
- Page, Matthew J
- ORCID
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2023-11-20
- Volume
- 383
- Pages
- e076754
- ISSN
- 1756-1833 (ISSN-L)
- OpenAlex
- W4388824619
- Cited by
- 134
- References
- 28
- Citations by year
- 2026: 63 · 2025: 50 · 2024: 17 · 2023: 4
- FWCI
- 9.50
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.bmj.com/company/legal-information/terms-conditions/legal-information/tdm-licencepolicy (tdm)
- Locations
- BMJ (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by-nc-nd)
- Bristol Research (University of Bristol) (publishedVersion, cc-by)
- University of Southern Denmark Research Portal (University of Southern Denmark) (publishedVersion)
- LSTM Online Archive (Liverpool School of Tropical Medicine) (acceptedVersion)
- Research Explorer (The University of Manchester) (submittedVersion, other-oa)
- HAL (submittedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- HighWire Press Open Archive (submittedVersion)
- Explore Bristol Research (submittedVersion, cc-by)
- Funders
- University Hospitals Bristol NHS Foundation Trust
- NIHR Bristol Biomedical Research Centre
- Australian Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- risk of bias, absence of evidence, publication bias, meta-analysis, systematic reviews, bias assessment tools
- MeSH terms
- Systematic Reviews as Topic, Humans, Research Design, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
