The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2
Open accessAbstract
BACKGROUND: AMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another assessment tool which was designed to evaluate the level of bias present within a systematic review. Our objective was to compare, contrast and establish both inter-rater reliability and usability of both tools as part of two overviews of systematic reviews. Strictly speaking, one tool assesses methodological quality (AMSTAR-2) and the other assesses risk of bias (ROBIS), but there is considerable overlap between the tools in terms of the signalling questions. METHODS: for weighted analysis). RESULTS: Thirty-one systematic reviews were included. For AMSTAR-2, the median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher (> 0.61). For ROBIS, the median agreement for all questions was also 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher. CONCLUSION: ROBIS is an effective tool for assessing risk of bias in systematic reviews and AMSTAR-2 is an effective tool at assessing quality. The median agreement between raters for both tools was identical (0.61). Reviews that included a meta-analysis were easier to rate with ROBIS; however, further developmental work could improve its use in reviews without a formal synthesis. AMSTAR-2 was more straightforward to use; however, more response options would be beneficial.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As AMSTAR 2 and the ROBIS tool 58 overlap in many sig naling questions, 37 the ROBIS tool was not used in this overview.”
From Acceptance and Commitment Therapy for Chronic Pain: An Overview of Systematic Reviews with Meta-Analysis of Randomized Clinical Trials (2024) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
ROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
Self-citationShea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755
Murad, M. H., Asi, N., Alsawas, M., & Alahdab, F. (2016) · Evidence Based Medicine · doi:10.1136/ebmed-2016-110401 · Cited by 1,388
Gwet, K. L. (2014) · Medical Entomology and Zoology · Cited by 1,095
Acupuncture for treating fibromyalgia
Deare, J. C., Zheng, Z., Xue, C. C., Liu, J., Shang, J., Scott, S., & Littlejohn, G. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007070.pub2 · Cited by 271
Lactobacillus reuteri to Treat Infant Colic: A Meta-analysis
Sung, V., D'Amico, F. J., Cabana, M. D., Chau, K., Koren, G., Savino, F., Szajewska, H., Deshpande, G. C., Dupont, C. M., Indrio, F., et al. (2017) · PEDIATRICS · doi:10.1542/peds.2017-1811 · Cited by 235
A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool
Lorenz, R. C., Matthias, K., Pieper, D., Wegewitz, U. E., Morche, J., Nocon, M., Rissling, O., Schirm, J., & Jacobs, A. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.05.028 · Cited by 212
Macfarlane, G. J., El‐Metwally, A., Silva, V. D., Ernst, E. K., Dowds, G. L., & Moots, R. J. (2011) · Lara D. Veeken · doi:10.1093/rheumatology/ker119 · Cited by 145
Acupuncture for fibromyalgia--a systematic review of randomized clinical trials
Mayhew, E., & Ernst, E. (2006) · Lara D. Veeken · doi:10.1093/rheumatology/kel406 · Cited by 127
Complementary and alternative medicine in fibromyalgia and related syndromes
Holdcraft, L. C., Assefi, N. P., & Buchwald, D. (2003) · Best Practice & Research Clinical Rheumatology · doi:10.1016/s1521-6942(03)00037-8 · Cited by 125
Urbańska, M., & Szajewska, H. (2014) · European Journal of Pediatrics · doi:10.1007/s00431-014-2328-0 · Cited by 125
Probiotics to Prevent or Treat Excessive Infant Crying
Sung, V., Collett, S., de Gooyer, T., Hiscock, H., Tang, M. L. K., & Wake, M. A. (2013) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2013.2572 · Cited by 119
Pieper, D., Puljak, L., González‐Lorenzo, M., & Minozzi, S. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.12.004 · Cited by 118
Manipulative therapies for infantile colic
Dobson, D., Lucassen, P. L., Miller, J., Vlieger, A. M., Prescott, P., & Lewith, G. T. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004796.pub2 · Cited by 114
Probiotics for infantile colic: a systematic review
Anabrees, J., Indrio, F., Paes, B. A., & AlFaleh, K. M. (2013) · BMC Pediatrics · doi:10.1186/1471-2431-13-186 · Cited by 102
Burda, B. U., Holmer, H. K., & Norris, S. L. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0237-1 · Cited by 93
Infant Colic—What works: A Systematic Review of Interventions for Breast‐fed Infants
Harb, T., Matsuyama, M., David, M., & Hill, R. J. (2016) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/mpg.0000000000001075 · Cited by 92
Baranowsky, J., Klose, P., Musial, F., Häuser, W., Dobos, G., & Langhorst, J. (2009) · Rheumatology International · doi:10.1007/s00296-009-0977-5 · Cited by 88
Nutritional Supplements and Other Complementary Medicines for Infantile Colic: A Systematic Review
Perry, R. E., Hunt, K. J., & Ernst, E. (2011) · PEDIATRICS · doi:10.1542/peds.2010-2098 · Cited by 87
Self-citationXu, M., Wang, J., Wang, N., Sun, F., Wang, L. Q., & Liu, X. (2015) · PLoS ONE · doi:10.1371/journal.pone.0141445 · Cited by 85
Probiotics for the Treatment of Infantile Colic: A Systematic Review
Bird, A. S., Gregory, P. J., Jalloh, M. A., Cochrane, Z. R., & Hein, D. J. (2016) · Journal of Pharmacy Practice · doi:10.1177/0897190016634516 · Cited by 85
Herbal Medicines for Gastrointestinal Disorders in Children and Adolescents: A Systematic Review
Anheyer, D., Frawley, J., Koch, A. K., Lauche, R., Langhorst, J., Dobos, G., & Cramer, H. (2017) · PEDIATRICS · doi:10.1542/peds.2017-0062 · Cited by 81
Efficacy of Lactobacillus reuteri DSM 17938 for infantile colic
Gutiérrez‐Castrellón, P., Indrio, F., Galvis, A. B., Jiménez-Gutiérrez, C., Jiménez–Escobar, I., & López‐Velázquez, G. (2017) · Medicine · doi:10.1097/md.0000000000009375 · Cited by 67
for the ANRS 1215 study group, Boussari, O., Subtil, F., Genolini, C., Bastard, M., Iwaz, J., Fonton, N., Étard, J.-F., & Écochard, R. (2015) · BMC Medical Research Methodology · doi:10.1186/1471-2288-15-10 · Cited by 66
Perry, R. E., Leach, V. A., Davies, P. A., Penfold, C. M., Ness, A., & Churchill, R. C. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0487-6 · Cited by 64
Self-citationKomplementäre und alternative Verfahren beim Fibromyalgiesyndrom
Langhorst, J., Häuser, W., Bernardy, K., Lucius, H., Settan, M., Winkelmann, A., & Musial, F. (2012) · Der Schmerz · doi:10.1007/s00482-012-1178-9 · Cited by 63
Silva, V. D., El‐Metwally, A., Ernst, E., Lewith, G. T., & Macfarlane, G. J. (2010) · Lara D. Veeken · doi:10.1093/rheumatology/keq025 · Cited by 63
Terhorst, L. A., Schneider, M., Kim, K. H., Goozdich, L. M., & Stilley, C. S. (2011) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2011.05.006 · Cited by 61
Martín-Sánchez, E., Torralba, E., Díaz-Domínguez, E., Barriga-Martín, A., & Martín, J. L. R. (2009) · The Open Rheumatology Journal · doi:10.2174/1874312900903010025 · Cited by 59
Efficacy of acupuncture on fibromyalgia syndrome: a Meta-analysis
Bai, Y., Guo, Y., Wang, H., Chen, B., Wang, Z., Liu, Y., Zhao, X., & Li, Y. (2014) · 中医杂志：英文版 · doi:10.1016/s0254-6272(15)30037-6 · Cited by 58
Probiotics for management of infantile colic: a systematic review of randomized controlled trials
Dryl, R., & Szajewska, H. (2017) · Archives of Medical Science · doi:10.5114/aoms.2017.66055 · Cited by 58
Chiropractic treatment for fibromyalgia: a systematic review
Ernst, E. (2009) · Clinical Rheumatology · doi:10.1007/s10067-009-1217-9 · Cited by 52
Carnes, D., Plunkett, A., Ellwood, J., & Miles, C. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019040 · Cited by 50
Gleberzon, B. J., Arts, J., Mei, A., & McManus, E. L. (2012) · Europe PMC (PubMed Central) · Cited by 48
A systematic review of homoeopathy for the treatment of fibromyalgia
Perry, R. E., Terry, R. H., & Ernst, E. (2010) · Clinical Rheumatology · doi:10.1007/s10067-009-1361-2 · Cited by 37
Self-citationNascimento, S. D. S., DeSantana, J. M., Nampo, F. K., Ribeiro, Ê. A. N., da Silva, D. L., de Araújo‐Júnior, J. X., Almeida, J. R. G. D. S., Bonjardim, L. R., Araújo, A. A. D. S., & Quintans‐Júnior, L. J. (2013) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2013/149468 · Cited by 36
An overview of systematic reviews of complementary and alternative therapies for infantile colic
Perry, R. E., Leach, V. A., Penfold, C. M., & Davies, P. A. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1191-5 · Cited by 36
Self-citationHomeopathy in the treatment of fibromyalgia—A comprehensive literature-review and meta-analysis
Boehm, K., Raak, C., Cramer, H., Lauche, R., & Ostermann, T. (2014) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2014.06.005 · Cited by 35
Acupoint Stimulation for Fibromyalgia: A Systematic Review of Randomized Controlled Trials
Cao, H., Li, X., Han, M., & Liu, J. (2013) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2013/362831 · Cited by 30
Efficacy of acupuncture on fibromyalgia syndrome：a Meta-analysis
Yang, Bai, ̆, Y., -, G., Hong, H., Wang, Bo, B., Chen, Zhan-kui, Yangyang, et al. (2014) · 中医杂志：英文版 · Cited by 21
Skjeie, H., Skonnord, T., Brekke, M., Klovning, A., Fetveit, A., Landgren, K., Hallström, I., & Brurberg, K. G. (2018) · Scandinavian Journal of Primary Health Care · doi:10.1080/02813432.2018.1426146 · Cited by 17
Quels traitements pour les coliques du nourrisson ?
Bruyas-Bertholon, V., Lachaux, A., Dubois, J.-P., Fourneret, P., & Letrilliart, L. (2012) · La Presse Médicale · doi:10.1016/j.lpm.2012.01.015 · Cited by 12
The efficacy of acupuncture in the treatment of fibromyalgia syndrome
Daya, S. (2007) · Cited by 5
Langhorst, J., Häuser, W., Bernardy, K., Lucius, H., Settan, M., Winkelmann, A., & Musial, F. (2012) · Der Schmerz · Cited by 3
Research works printed but not linked: 1
- Terhorst L, Schneider M. Complementary and alternative medicine in the treatment of pain in fibromyalgia: a systematic review of randomized controlled trials. Portland: From International Research Congress on Integrative Medicine and Health; 2012. 15-18 May 2012
Cited by
Sorted by citations
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 215
Schleider, J. L., Zapata, J. P., Rapoport, A., Wescott, A., Ghosh, A., Kaveladze, B., Szkody, E. M., & Ahuvia, I. L. (2025) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-081423-025033 · Cited by 130
Martinez-Calderon, J., García-Muñoz, C., Rufo-Barbero, C., Matias-Soto, J., & Cano-García, F. J. (2024) · The Journal of Pain · doi:10.1016/j.jpain.2023.09.013 · Cited by 79
Umbrella reviews: a methodological guide
Fernandez, R. S., Sharifnia, A. M., & Khalil, H. (2025) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvaf012 · Cited by 88
Benefits of Participation in Clinical Trials: An Umbrella Review
Bouzalmate-Hajjaj, A., Massó Guijarro, P., Khan, K. S., Bueno‐Cavanillas, A., & Ibáñez, N. C. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192215368 · Cited by 71
Metadata
- Authors
- Perry, R., Whitmarsh, A., Leach, V., & Davies, P.
- Affiliations
- National Institute for Health and Care Research (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Corresponding author
- Perry, R.
- Venue
- Systematic Reviews
- Published
- 2021-10-25
- Volume
- 10
- Issue
- 1
- Pages
- 273-273
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC8543959
- OpenAlex
- W3173323133
- Cited by
- 130
- References
- 49
- Citations by year
- 2026: 22 · 2025: 24 · 2024: 33 · 2023: 37 · 2022: 13
- FWCI
- 0.86
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- University Hospitals Bristol NHS Foundation Trust
- National Institute for Health and Care Research
- National Institutes of Health
- National Institute for Health Research Applied Research Collaboration West
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- AMSTAR 2, ROBIS, overviews of systematic reviews, risk of bias, methodological quality assessment, inter-rater reliability, reporting quality, meta-analysis
- MeSH terms
- Systematic Reviews as Topic, Humans, Reproducibility of Results, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
