The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2
Open accessAbstract
BACKGROUND: AMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another assessment tool which was designed to evaluate the level of bias present within a systematic review. Our objective was to compare, contrast and establish both inter-rater reliability and usability of both tools as part of two overviews of systematic reviews. Strictly speaking, one tool assesses methodological quality (AMSTAR-2) and the other assesses risk of bias (ROBIS), but there is considerable overlap between the tools in terms of the signalling questions. METHODS: for weighted analysis). RESULTS: Thirty-one systematic reviews were included. For AMSTAR-2, the median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher (> 0.61). For ROBIS, the median agreement for all questions was also 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher. CONCLUSION: ROBIS is an effective tool for assessing risk of bias in systematic reviews and AMSTAR-2 is an effective tool at assessing quality. The median agreement between raters for both tools was identical (0.61). Reviews that included a meta-analysis were easier to rate with ROBIS; however, further developmental work could improve its use in reviews without a formal synthesis. AMSTAR-2 was more straightforward to use; however, more response options would be beneficial.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As AMSTAR 2 and the ROBIS tool 58 overlap in many sig naling questions, 37 the ROBIS tool was not used in this overview.”
From Acceptance and Commitment Therapy for Chronic Pain: An Overview of Systematic Reviews with Meta-Analysis of Randomized Clinical Trials (2024) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
ROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
Self-citationShea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755
Murad, M. H., Asi, N., Alsawas, M., & Alahdab, F. (2016) · Evidence Based Medicine · doi:10.1136/ebmed-2016-110401 · Cited by 1,388
Gwet, K. L. (2014) · Medical Entomology and Zoology · Cited by 1,095
Acupuncture for treating fibromyalgia
Deare, J. C., Zheng, Z., Xue, C. C., Liu, J., Shang, J., Scott, S., & Littlejohn, G. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007070.pub2 · Cited by 271
Lactobacillus reuteri to Treat Infant Colic: A Meta-analysis
Sung, V., D'Amico, F. J., Cabana, M. D., Chau, K., Koren, G., Savino, F., Szajewska, H., Deshpande, G. C., Dupont, C. M., Indrio, F., et al. (2017) · PEDIATRICS · doi:10.1542/peds.2017-1811 · Cited by 235
Research works printed but not linked: 1
- Terhorst L, Schneider M. Complementary and alternative medicine in the treatment of pain in fibromyalgia: a systematic review of randomized controlled trials. Portland: From International Research Congress on Integrative Medicine and Health; 2012. 15-18 May 2012
Cited by
Sorted by citations
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 215
Schleider, J. L., Zapata, J. P., Rapoport, A., Wescott, A., Ghosh, A., Kaveladze, B., Szkody, E. M., & Ahuvia, I. L. (2025) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-081423-025033 · Cited by 130
Martinez-Calderon, J., García-Muñoz, C., Rufo-Barbero, C., Matias-Soto, J., & Cano-García, F. J. (2024) · The Journal of Pain · doi:10.1016/j.jpain.2023.09.013 · Cited by 79
Umbrella reviews: a methodological guide
Fernandez, R. S., Sharifnia, A. M., & Khalil, H. (2025) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvaf012 · Cited by 88
Benefits of Participation in Clinical Trials: An Umbrella Review
Bouzalmate-Hajjaj, A., Massó Guijarro, P., Khan, K. S., Bueno‐Cavanillas, A., & Ibáñez, N. C. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192215368 · Cited by 71
Metadata
- Authors
- Perry, R., Whitmarsh, A., Leach, V., & Davies, P.
- Affiliations
- National Institute for Health and Care Research (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Corresponding author
- Perry, R.
- Venue
- Systematic Reviews
- Published
- 2021-10-25
- Volume
- 10
- Issue
- 1
- Pages
- 273-273
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC8543959
- OpenAlex
- W3173323133
- Cited by
- 130
- References
- 49
- Citations by year
- 2026: 22 · 2025: 24 · 2024: 33 · 2023: 37 · 2022: 13
- FWCI
- 0.86
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- University Hospitals Bristol NHS Foundation Trust
- National Institute for Health and Care Research
- National Institutes of Health
- National Institute for Health Research Applied Research Collaboration West
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- AMSTAR 2, ROBIS, overviews of systematic reviews, risk of bias, methodological quality assessment, inter-rater reliability, reporting quality, meta-analysis
- MeSH terms
- Systematic Reviews as Topic, Humans, Reproducibility of Results, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
