CoChat

GRADE Methods for Guideline Development: Time to Evolve?

Norris, Susan L., Bero, Lisa · 2016 · Annals of Internal Medicine
Cited by 45

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Abstract

Ideas and Opinions6 December 2016GRADE Methods for Guideline Development: Time to Evolve?FREESusan L. Norris, MD, MPH, MSc and Lisa Bero, PhDSusan L. Norris, MD, MPH, MScFrom the World Health Organization, Geneva, Switzerland, and University of Sydney, New South Wales, Australia. and Lisa Bero, PhDFrom the World Health Organization, Geneva, Switzerland, and University of Sydney, New South Wales, Australia.Author, Article, and Disclosure Informationhttps://doi.org/10.7326/M16-1254 SectionsAboutVisual AbstractPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail The GRADE (Grading of Recommendations Assessment, Development and Evaluation) working group was established in 2000 to develop a "common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations" (1). The GRADE approach consists of 2 basic determinations. Systematic reviewers or guideline developers make an assessment of the certainty in effect estimates for each outcome that is important or critical for decision making. Guideline developers then translate evidence into recommendations using a framework that encompasses the relative values of the outcomes according to persons affected by the recommendations, preferences about interventions, acceptability, feasibility, effect on equity across subpopulations, resource considerations, and other relevant explicit factors.The GRADE approach has been adopted (sometimes with modifications) by more than 80 entities, and more than 100 publications about GRADE methods are indexed in PubMed. These accomplishments have been achieved through strong leadership and thoughtful contributions from many clinical and public health scientists. There are, however, several challenges and limitations with GRADE and its implementation.Challenges and Limitations of the GRADE ApproachAlthough 1 of the key strengths of GRADE is that it makes judgments transparent, implementation of the approach can be challenging. Various groups disagree on how to assess a body of evidence and the meaning and suitability of GRADE terminology (2). This assessment comes down to a single determination: high-, moderate-, low-, or very-low-quality evidence. Guideline panels can view this assessment in an oversimplified or overly precise manner, or they can ignore it altogether and base recommendations on other considerations.The challenges of implementing GRADE affect its reliability. Although persons with extensive experience using GRADE have "substantial" interrater reliability when assessing the quality of a body of evidence (3), experienced evaluators (using a modified GRADE approach) have low interrater reliability when assessing complex bodies of evidence consisting of different study designs (4). An expert panel achieved only fair interrater agreement on the strength of recommendations based on a body of evidence (5).The predictive validity of GRADE assessments of the certainty in effect estimates (the degree to which the assessment of high-, moderate-, low- or very-low-quality evidence predicts whether the effect estimate will change with further research) is reported as limited (6). When the quality of evidence was low, the data predicted that future studies had a smaller effect on the estimate of effect than anticipated by GRADE experts, whereas high-quality evidence was influenced more by new data than anticipated.Although GRADE methods are evolving (7), they are not currently applicable to many questions that guideline developers face, including those about assessing risk and causality, establishing risk thresholds, or assessing animal studies. Further, GRADE does not provide explicit guidance for complex interventions or when the evidence is linked across a causal pathway, and conceptual frameworks are generally absent. There is only limited GRADE guidance on how to assess the quality of a body of evidence addressing resource use.The GRADE approach is also challenging to apply to different types of data because it was developed for quantitative data with a pooled estimate and CI for each outcome. When data are qualitative or the outcomes cannot be pooled due to heterogeneity, GRADE must be adapted, although the same framework and elements can still be applied. No GRADE guidance is available on how to assess the quality of data from mathematical models or how to incorporate the results of modeling into the development of recommendations.The Way ForwardSeveral steps can be taken to strengthen the GRADE framework and approach to advance the science supporting guideline development. Further evaluation by the GRADE working group and independent teams that focuses on reliability and validity of the GRADE criteria for assessing the quality of a body of evidence is a priority. Because GRADE is widely applied, there is now a large cohort of systematic reviews that could be used to compare the quality of a body of evidence and its evoluti

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.7326/m16-1254

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 45

Metadata

Authors
Norris, S. L., & Bero, L.
Venue
Annals of Internal Medicine
Published
2016-12-06
Volume
165
Issue
11
Pages
810-811
ISSN
  • 0003-4819 (ISSN-L)
  • 1539-3704
OpenAlex
W2522240987
Cited by
45
References
1
FWCI
5.14
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.