CoChat

Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor

Open access
Faggion, Clovis Mariano · 2015 · BMC Medical Research Methodology
Cited by 109

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

BACKGROUND: Systematic reviews are pivotal components in the development of evidence-based clinical guidelines. To evaluate the methodological quality of these systematic reviews, several tools have been proposed. Among them, the assessment of multiple systematic reviews (AMSTAR) checklist is probably used most frequently. This tool comprises 11 items related to the steps taken when conducting a systematic review, and it is claimed to have good face and content validity. The objective of this debate paper was twofold: (a) to critically evaluate the ability of all AMSTAR checklist items to adequately determine the methodological quality of a systematic review; and (b) to describe difficulties regarding interpretation of the checklist, and provide potential solutions for these difficulties. DISCUSSION: Some items of the AMSTAR checklist seem to assess the quality of reporting of a systematic review more than its methodological quality. For example, item 7 may not "capture" the true methodological quality of primary studies included in the systematic review. Item 10 does not likely result in the collection of in-depth information on the presence of publication bias in the systematic review. Furthermore, some items may be difficult to interpret, hindering accurate assessment. For example, item 5 does not explicitly indicate whether a list of documents excluded in each phase of selection (i.e., after evaluation of titles and abstracts, and after full-text assessment) should be reported. The present debate paper evaluated and discussed some methodological limitations of the AMSTAR checklist, as well as challenges involved in evaluation of the checklist's items. Several suggestions are also made to optimize the use of this checklist. The information in this paper may stimulate further discussion among systematic reviewers, methodologists and clinicians.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-015-0062-6

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Research works printed but not linked: 8

  1. Farrington DP. Methodological quality standards for evaluation research. Ann Am Acad Polit Soc Sci. 2003;1:49–68. · DOI · Paper page
  2. Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of Revised Assessment of Multiple Systematic Reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J. 2010;4:84–91.
  3. Pieper D, Buechter RB, Li L, Prediger B, Eikermann M. Systematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties. J Clin Epidemiol. 2015;68:574–83. · DOI · Paper page
  4. Higgins JPT, Altman DG, Sterne JAC (editors). Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S (editors). Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org .
  5. Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org .
  6. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1–12. · DOI · Paper page
  7. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. · DOI · Paper page
  8. Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S (editors). Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org .

Cited by

See all 109

Metadata

Authors
Faggion, C. M.
Venue
BMC Medical Research Methodology
Published
2015-08-13
Volume
15
Issue
1
Pages
63-63
ISSN
1471-2288 (ISSN-L)
OpenAlex
W1889300381
Cited by
109
References
7
FWCI
15.55
Type
journal article
Language
en
Open access
gold
Licence
creativecommons.org/licenses/by/4.0/ (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. https://app.cochat.ai/papers/10.1186/s12874-015-0062-6
The paper
Faggion, C. M. (2015). Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-015-0062-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.