Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systematic reviews are pivotal components in the development of evidence-based clinical guidelines. To evaluate the methodological quality of these systematic reviews, several tools have been proposed. Among them, the assessment of multiple systematic reviews (AMSTAR) checklist is probably used most frequently. This tool comprises 11 items related to the steps taken when conducting a systematic review, and it is claimed to have good face and content validity. The objective of this debate paper was twofold: (a) to critically evaluate the ability of all AMSTAR checklist items to adequately determine the methodological quality of a systematic review; and (b) to describe difficulties regarding interpretation of the checklist, and provide potential solutions for these difficulties. DISCUSSION: Some items of the AMSTAR checklist seem to assess the quality of reporting of a systematic review more than its methodological quality. For example, item 7 may not "capture" the true methodological quality of primary studies included in the systematic review. Item 10 does not likely result in the collection of in-depth information on the presence of publication bias in the systematic review. Furthermore, some items may be difficult to interpret, hindering accurate assessment. For example, item 5 does not explicitly indicate whether a list of documents excluded in each phase of selection (i.e., after evaluation of titles and abstracts, and after full-text assessment) should be reported. The present debate paper evaluated and discussed some methodological limitations of the AMSTAR checklist, as well as challenges involved in evaluation of the checklist's items. Several suggestions are also made to optimize the use of this checklist. The information in this paper may stimulate further discussion among systematic reviewers, methodologists and clinicians.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Going from evidence to recommendations
Guyatt, G. H., Oxman, A. D., Kunz, R., Falck-Ytter, Y., Vist, G. E., Liberati, A., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39493.646875.ae · Cited by 1,810
Sterne, J. A., Egger, M., & Moher, D. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch10 · Cited by 651
Kung, J. (2010) · The Open Dentistry Journal · doi:10.2174/1874210601004020084 · Cited by 329 · cited 1×
“Another group of researchers developed a revision of the AMSTAR checklist (R-AMSTAR) with quantitative item scoring [4].”
BackgroundHalladay, C. W., Trikalinos, T. A., Schmid, I. T., Schmid, C. H., & Dahabreh, I. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.017 · Cited by 166 · cited 1×
“Because lists of publications do not match perfectly among databases [7], a search of at least 2 electronic databases seems to be important to increase comprehensiveness (interestingly, new evidence suggests that the benefits for searching sources beyond one major database such as MEDLINE may be modest [8]).”
BackgroundHo, R. S., Wu, X., Yuan, J., Liu, S., Lai, X., Wong, S. Y., & Chung, V. C. (2015) · npj Primary Care Respiratory Medicine · doi:10.1038/npjpcrm.2014.102 · Cited by 50
Research works printed but not linked: 8
- Farrington DP. Methodological quality standards for evaluation research. Ann Am Acad Polit Soc Sci. 2003;1:49–68. · DOI · Paper page
- Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of Revised Assessment of Multiple Systematic Reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J. 2010;4:84–91.
- Pieper D, Buechter RB, Li L, Prediger B, Eikermann M. Systematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties. J Clin Epidemiol. 2015;68:574–83. · DOI · Paper page
- Higgins JPT, Altman DG, Sterne JAC (editors). Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S (editors). Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org .
- Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org .
- Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1–12. · DOI · Paper page
- Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. · DOI · Paper page
- Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S (editors). Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org .
Cited by
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448
Recommendations for assessing the risk of bias in systematic reviews of health-care interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.004 · Cited by 221
Selecting and implementing overview methods: implications from five exemplar overviews
Pollock, A., Campbell, P., Brunton, G., Hunt, H., & Estcourt, L. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0534-3 · Cited by 134
(2017) · Europe PMC (PubMed Central) · Cited by 61
Metadata
- Authors
- Faggion, C. M.
- Venue
- BMC Medical Research Methodology
- Published
- 2015-08-13
- Volume
- 15
- Issue
- 1
- Pages
- 63-63
- ISSN
- 1471-2288 (ISSN-L)
- OpenAlex
- W1889300381
- Cited by
- 109
- References
- 7
- FWCI
- 15.55
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0/ (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. https://app.cochat.ai/papers/10.1186/s12874-015-0062-6
- The paper
- Faggion, C. M. (2015). Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-015-0062-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.