GRADE Methods for Guideline Development: Time to Evolve?
The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Ideas and Opinions6 December 2016GRADE Methods for Guideline Development: Time to Evolve?FREESusan L. Norris, MD, MPH, MSc and Lisa Bero, PhDSusan L. Norris, MD, MPH, MScFrom the World Health Organization, Geneva, Switzerland, and University of Sydney, New South Wales, Australia. and Lisa Bero, PhDFrom the World Health Organization, Geneva, Switzerland, and University of Sydney, New South Wales, Australia.Author, Article, and Disclosure Informationhttps://doi.org/10.7326/M16-1254 SectionsAboutVisual AbstractPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail The GRADE (Grading of Recommendations Assessment, Development and Evaluation) working group was established in 2000 to develop a "common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations" (1). The GRADE approach consists of 2 basic determinations. Systematic reviewers or guideline developers make an assessment of the certainty in effect estimates for each outcome that is important or critical for decision making. Guideline developers then translate evidence into recommendations using a framework that encompasses the relative values of the outcomes according to persons affected by the recommendations, preferences about interventions, acceptability, feasibility, effect on equity across subpopulations, resource considerations, and other relevant explicit factors.The GRADE approach has been adopted (sometimes with modifications) by more than 80 entities, and more than 100 publications about GRADE methods are indexed in PubMed. These accomplishments have been achieved through strong leadership and thoughtful contributions from many clinical and public health scientists. There are, however, several challenges and limitations with GRADE and its implementation.Challenges and Limitations of the GRADE ApproachAlthough 1 of the key strengths of GRADE is that it makes judgments transparent, implementation of the approach can be challenging. Various groups disagree on how to assess a body of evidence and the meaning and suitability of GRADE terminology (2). This assessment comes down to a single determination: high-, moderate-, low-, or very-low-quality evidence. Guideline panels can view this assessment in an oversimplified or overly precise manner, or they can ignore it altogether and base recommendations on other considerations.The challenges of implementing GRADE affect its reliability. Although persons with extensive experience using GRADE have "substantial" interrater reliability when assessing the quality of a body of evidence (3), experienced evaluators (using a modified GRADE approach) have low interrater reliability when assessing complex bodies of evidence consisting of different study designs (4). An expert panel achieved only fair interrater agreement on the strength of recommendations based on a body of evidence (5).The predictive validity of GRADE assessments of the certainty in effect estimates (the degree to which the assessment of high-, moderate-, low- or very-low-quality evidence predicts whether the effect estimate will change with further research) is reported as limited (6). When the quality of evidence was low, the data predicted that future studies had a smaller effect on the estimate of effect than anticipated by GRADE experts, whereas high-quality evidence was influenced more by new data than anticipated.Although GRADE methods are evolving (7), they are not currently applicable to many questions that guideline developers face, including those about assessing risk and causality, establishing risk thresholds, or assessing animal studies. Further, GRADE does not provide explicit guidance for complex interventions or when the evidence is linked across a causal pathway, and conceptual frameworks are generally absent. There is only limited GRADE guidance on how to assess the quality of a body of evidence addressing resource use.The GRADE approach is also challenging to apply to different types of data because it was developed for quantitative data with a pooled estimate and CI for each outcome. When data are qualitative or the outcomes cannot be pooled due to heterogeneity, GRADE must be adapted, although the same framework and elements can still be applied. No GRADE guidance is available on how to assess the quality of data from mathematical models or how to incorporate the results of modeling into the development of recommendations.The Way ForwardSeveral steps can be taken to strengthen the GRADE framework and approach to advance the science supporting guideline development. Further evaluation by the GRADE working group and independent teams that focuses on reliability and validity of the GRADE criteria for assessing the quality of a body of evidence is a priority. Because GRADE is widely applied, there is now a large cohort of systematic reviews that could be used to compare the quality of a body of evidence and its evoluti
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mustafa, R. A., Santesso, N., Brozek, J., Akl, E. A., Walter, S. D., Norman, G., Kulasegaram, M., Christensen, R., Guyatt, G. H., Falck-Ytter, Y., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.02.004 · Cited by 370
Cited by
Aidara-Kane, A., Angulo, F. J., Conly, J. M., Minato, Y., Silbergeld, E. K., McEwen, S. A., Collignon, P. J., for the WHO Guideline Development Group, Balkhy, H., Collignon, P., et al. (2018) · Antimicrobial Resistance & Infection Control · doi:10.1186/s13756-017-0294-9 · Cited by 354
Using systematic reviews in guideline development: The GRADE approach
Zhang, Y., Akl, E. A., & Schünemann, H. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1313 · Cited by 260
Steenland, K., Schubauer-Berigan, M. K., Vermeulen, R., Lunn, R. M., Straif, K., Zahm, S., Stewart, P., Arroyave, W. D., Mehta, S. S., & Pearce, N. (2020) · Environmental Health Perspectives · doi:10.1289/ehp6980 · Cited by 69
Mechanick, J. I., Pessah-Pollack, R., Camacho, P., Correa, R., Figaro, M. K., Garber, J. R., Jasim, S., Pantalone, K. M., Trence, D., & Upala, S. (2017) · Endocrine Practice · doi:10.4158/ep171866.gl · Cited by 25
Metadata
- Authors
- Norris, S. L., & Bero, L.
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2016-12-06
- Volume
- 165
- Issue
- 11
- Pages
- 810-811
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2522240987
- Cited by
- 45
- References
- 1
- FWCI
- 5.14
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.