Developing clinical practice guidelines: types of evidence and outcomes; values and economics, synthesis, grading, and presentation and deriving recommendations
Open accessThe DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Abstract
Clinical practice guidelines are one of the foundations of efforts to improve healthcare. In 1999, we authored a paper about methods to develop guidelines. Since it was published, the methods of guideline development have progressed both in terms of methods and necessary procedures and the context for guideline development has changed with the emergence of guideline clearinghouses and large scale guideline production organisations (such as the UK National Institute for Health and Clinical Excellence). It therefore seems timely to, in a series of three articles, update and extend our earlier paper. In this second paper, we discuss issues of identifying and synthesizing evidence: deciding what type of evidence and outcomes to include in guidelines; integrating values into a guideline; incorporating economic considerations; synthesis, grading, and presentation of evidence; and moving from evidence to recommendations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Self-citationGrading quality of evidence and strength of recommendations
(2004) · BMJ · doi:10.1136/bmj.328.7454.1490 · Cited by 8,477 · cited 2×
“A number of publications describe the GRADE approach and its development [44,49-57].”
BackgroundSelf-citationAGREE II: advancing guideline development, reporting and evaluation in health care
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Fervers, B., Graham, I. D., Grimshaw, J., Hanna, S. E., et al. (2010) · Canadian Medical Association Journal · doi:10.1503/cmaj.090449 · Cited by 3,790 · cited 1×
“gov) that identify and systematically characterize guidelines on a number of domains and the development of robust guideline appraisal instruments such as the AGREE tool [3,4].”
BackgroundSelf-citationWhat is ‘quality of evidence’ and why is it important to clinicians?
Guyatt, G. H., Oxman, A. D., Kunz, R., Vist, G. E., Falck-Ytter, Y., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39490.551019.be · Cited by 3,250 · cited 1×
“A number of publications describe the GRADE approach and its development [44,49-57].”
BackgroundSelf-citationClinical Practice Guidelines We Can Trust
Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services, & Institute of Medicine (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,081
Going from evidence to recommendations
Guyatt, G. H., Oxman, A. D., Kunz, R., Falck-Ytter, Y., Vist, G. E., Liberati, A., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39493.646875.ae · Cited by 1,810 · cited 1×
“Those making recommendations may have higher or lower confidence that following their recommendation will do more good than harm across the range of patients for whom the recommendation is intended [62].”
BackgroundSelf-citation(2005) · Integrative Oncology · doi:10.1201/9780203088319-19 · Cited by 1,579
(2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.18 · Cited by 1,446
Fischhoff, B. (1980) · Operations Research · doi:10.1287/opre.28.1.28 · Cited by 1,416
Clinical practice guidelines : directions for a new program
Field, M. J., & Lohr, K. N. (1990) · Cited by 1,415
Screening for Breast Cancer: An Update for the U.S. Preventive Services Task Force
Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-10-200911170-00009 · Cited by 1,248
Clinical guidelines: Developing guidelines
Shekelle, P. G., Woolf, S. H., Eccles, M., & Grimshaw, J. (1999) · BMJ · doi:10.1136/bmj.318.7183.593 · Cited by 1,222 · cited 1×
“In 1999, we authored a paper about methods to develop guidelines [2].”
MethodSelf-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 929
Brożek, J. L., Akl, E. A., Alonso‐Coello, P., Lang, D., Jaeschke, R., Williams, J. W., Phillips, B., Lelgemann, M., Lethaby, A., Bousquet, J., et al. (2009) · Allergy · doi:10.1111/j.1398-9995.2009.01973.x · Cited by 894 · cited 1×
“A number of publications describe the GRADE approach and its development [44,49-57].”
BackgroundSelf-citationGrol, R., Dalhuijsen, J., Thomas, S., Veld, C. I. ‘., Rutten, G., & Mokkink, H. (1998) · BMJ · doi:10.1136/bmj.317.7162.858 · Cited by 877 · cited 1×
“Dutch general practitioners followed guideline recommendations that were vague or nonspecific 35% of the time, while ‘clear’ recommendations were followed 67% of the time [36].”
BackgroundEddy, D. M. (1990) · Annals of Internal Medicine · doi:10.7326/0003-4819-113-3-214 · Cited by 837
Clinical Practice Guidelines: Directions for a New Program
Committee to Advise the Public Health Service on Clinical Practice Guidelines, & Institute of Medicine (1990) · National Academies Press eBooks · doi:10.17226/1626 · Cited by 803
Shared decision making in clinical medicine: past research and future directions
Frosch, D. L., & Kaplan, R. M. (1999) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00097-5 · Cited by 721 · cited 1×
“The second option for dealing with close calls is for the group to eschew a blanket recommendation but to instead encourage shared or informed decision-making, in which the patient is encouraged to review the tradeoffs with their clinician and make an individual decision based on personal preferences [28,29].”
BackgroundThe periodic health examination. Canadian Task Force on the Periodic Health Examination.
(1979) · PubMed Central · Cited by 685
Readings on the Principles and Applications of Decision Analysis
Howard, R. A., & Matheson, J. E. (1989) · Cited by 685
Patient-Centered Medicine: A Professional Evolution
Laine, C. (1996) · JAMA · doi:10.1001/jama.1996.03530260066035 · Cited by 632 · cited 1×
“The disadvantage of this approach is its inherent paternalism and the risk of misjudgments by the group [27].”
BackgroundRosen, D. H., & Hoang, U. (2017) · Oxford Medicine Online · doi:10.1093/med/9780190628871.001.0001 · Cited by 617
Schünemann, H. J., Jaeschke, R., Cook, D. J., Bria, W. F., El-Solh, A. A., Ernst, A., Fahy, B. F., Gould, M. K., Horan, K. L., Krishnan, J. A., et al. (2006) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.200602-197st · Cited by 577 · cited 1×
“A number of publications describe the GRADE approach and its development [44,49-57].”
BackgroundSelf-citationUse of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive
Jaeschke, R., Guyatt, G. H., Dellinger, P., Schunemann, H., Levy, M. M., Kunz, R., Norris, S., Bion, J., & for the GRADE working group (2008) · BMJ · doi:10.1136/bmj.a744 · Cited by 577 · cited 1×
“A number of publications describe the GRADE approach and its development [44,49-57].”
BackgroundAGREE II: Advancing guideline development, reporting and evaluation in health care
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Fervers, B., Graham, I. D., Grimshaw, J., Hanna, S. E., et al. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.001 · Cited by 563
Self-citationPuhan, M. A., Garcia-Aymerich, J., Frey, M., ter Riet, G., Antó, J. M., Agustí, A. G., Gómez, F. P., Rodríguez-Roisín, R., Moons, K. G., Kessels, A. G., et al. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61301-5 · Cited by 521 · cited 1×
“If the guideline panel judges that the balance between desirable and undesirable effects varies by baseline risk, it can issue separate recommendations for groups with different baselines risks when tools for risk stratification are available for the guideline users [60,61].”
BackgroundThe Periodic Health Examination
(1934) · New England Journal of Medicine · doi:10.1056/nejm193401252100413 · Cited by 483
Causal Models in the Social Sciences.
Henry, N. W., & Blalock, H. M. (1976) · Contemporary Sociology · doi:10.2307/2063121 · Cited by 472
Shared decision making about screening and chemoprevention
Sheridan, S. L., Harris, R. P., & Woolf, S. H. (2004) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2003.09.011 · Cited by 441
Self-citationScreening for Osteoporosis: U.S. Preventive Services Task Force Recommendation Statement
U.S. Preventive Services Task Force* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-154-5-201103010-00307 · Cited by 436
Incorporating Patients' Preferences into Medical Decisions
Kassirer, J. P. (1994) · New England Journal of Medicine · doi:10.1056/nejm199406303302611 · Cited by 415 · cited 1×
“ferent values, or utilities, they assign to the outcomes [25,26].”
BackgroundHow to formulate research recommendations
Brown, P., Brunnhuber, K., Chalkidou, K., Chalmers, I., Clarke, M., Fenton, M., Forbes, C., Glanville, J., Hicks, N. J., Moody, J., et al. (2006) · BMJ · doi:10.1136/bmj.38987.492014.94 · Cited by 381 · cited 1×
“The research recommendations should be detailed regarding the specific research questions that should be addressed, particularly which patient-important outcomes should be measured, and other relevant aspects of what research is needed [65].”
BackgroundChanging clinical behaviour by making guidelines specific
Michie, S., & Johnston, M. (2004) · BMJ · doi:10.1136/bmj.328.7435.343 · Cited by 309 · cited 1×
“Michie and Johnston, using evidence from psychological research, went so far as to conclude that the ‘most cost effective intervention to increase the implementation of guidelines is rewriting guidelines in behaviorally specific terms’ [38].”
BackgroundLetters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
Schünemann, H. J., Best, D., Vist, G. E., & Oxman, A. D. (2003) · PubMed Central · Cited by 289
Self-citationBrożek, J. L., Akl, E. A., Jaeschke, R., Lang, D. M., Bossuyt, P., Glasziou, P., Helfand, M., Ueffing, E., Alonso‐Coello, P., Meerpohl, J., et al. (2009) · Allergy · doi:10.1111/j.1398-9995.2009.02083.x · Cited by 270 · cited 1×
“A number of publications describe the GRADE approach and its development [44,49-57].”
BackgroundSelf-citationThe Next Step in Guideline Development: Incorporating Patient Preferences
(2008) · JAMA · doi:10.1001/jama.300.4.436 · Cited by 265 · cited 1×
“org) [49] emphasizes the link between the quality of a body of evidence and the recommendation, but recognizes that other factors beyond the quality of evidence contribute to the strength of a recommendation, such as patient values and preferences [58,59].”
BackgroundImproving the Use of Research Evidence in Guideline Development
Oxman, A. D., Schünemann, H. J., Fretheim, A., Boyd, E. A., Bero, L. A., & Edejer, T. T.-T. (2007) · BIBSYS Brage (BIBSYS (Norway)) · Cited by 261
Self-citationA manual for assessing health practices & designing practice policies : the explicit approach
Eddy, D. M. (1992) · Medical Entomology and Zoology · Cited by 218
Screening for Osteoporosis: An Update for the U.S. Preventive Services Task Force
Nelson, H. D., Haney, E. M., Dana, T., Bougatsos, C., & Chou, R. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-2-201007200-00262 · Cited by 216 · cited 1×
“Services Task Force when considering a guideline about screening for osteoporosis [14].”
BackgroundEngaging Health Care Consumers to Improve the Quality of Care
Hibbard, J. H. (2003) · Medical Care · doi:10.1097/00005650-200301001-00007 · Cited by 211 · cited 1×
“The advantage of this approach is its respect for autonomy and individual choice, in which guidelines become a tool for patient empowerment, engagement, and activation [31,32].”
BackgroundHow to develop cost-conscious guidelines
Eccles, M., & Mason, J. (2001) · Health Technology Assessment · doi:10.3310/hta5160 · Cited by 211 · cited 1×
“The methods of incorporating economic considerations are shaped by the methods of guideline development [34].”
MethodSelf-citationKassirer, J. P., & Pauker, S. G. (1981) · New England Journal of Medicine · doi:10.1056/nejm198112103052409 · Cited by 184
The Autonomous Patient: Ending Paternalism in Medical Care
Coulter, A. (2002) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 174
Schünemann, H. J., Fretheim, A., & Oxman, A. D. (2006) · Health Research Policy and Systems · doi:10.1186/1478-4505-4-22 · Cited by 166
Self-citationShekelle, P., Kravitz, R. L., Beart, J., Marger, M., Wang, M., & Lee, M. L. (2000) · PubMed · Cited by 155
Self-citationUsing Existing Systematic Reviews in Complex Systematic Reviews
Whitlock, E. P., Lin, J. S., Chou, R., Shekelle, P., & Robinson, K. A. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-10-200805200-00010 · Cited by 144 · cited 1×
“line development [17].”
Self-citationTransparent Development of the WHO Rapid Advice Guidelines
Schünemann, H. J., Hill, S. R., Kakad, M., Vist, G. E., Bellamy, R., Stockman, L., Wisløff, T. F., Mar, C. D., Hayden, F., Uyeki, T. M., et al. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040119 · Cited by 137 · cited 1×
“A number of publications describe the GRADE approach and its development [44,49-57].”
BackgroundSelf-citationImproving the use of research evidence in guideline development: introduction
SURE, Oxman, A. D., Fretheim, A., & Schünemann, H. J. (2006) · Health Research Policy and Systems · doi:10.1186/1478-4505-4-12 · Cited by 135
Self-citationSchünemann, H. J., Fretheim, A., & Oxman, A. D. (2006) · Health Research Policy and Systems · doi:10.1186/1478-4505-4-21 · Cited by 134
Self-citationGuidelines You Can Follow and Can Trust: An Ideal and an Example
McDonald, C. J. (1994) · JAMA · doi:10.1001/jama.1994.03510350082042 · Cited by 114 · cited 1×
“McDonald [35] and others have lamented the existence of recommendations that are”
BackgroundBalancing benefits and harms in health care
Cuervo, L. G., & Clarke, M. (2003) · BMJ · doi:10.1136/bmj.327.7406.65 · Cited by 104
"What lies beneath it all?" – an interview study of GPs' attitudes to the use of guidelines
Carlsen, B., & Norheim, O. F. (2008) · BMC Health Services Research · doi:10.1186/1472-6963-8-218 · Cited by 97 · cited 1×
“Whatever choice is made, it is best for guideline developers to be transparent about value judgments [23].”
BackgroundClaxton, K., Cohen, J. T., & Neumann, P. J. (2005) · Health Affairs · doi:10.1377/hlthaff.24.1.93 · Cited by 97 · cited 1×
“Nonetheless, the criteria in Table 4 must be met for a recommendation for use of interventions in the context of research to be sensible [63,64].”
BackgroundKent, E. (2003) · Nursing Standard · doi:10.7748/ns.17.20.29.s52 · Cited by 91
Systems to Rate the Strength of Scientific Evidence: Summary
West, S. L., King, V., Carey, T., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · Cited by 86
The 2009 Breast Cancer Screening Recommendations of the US Preventive Services Task Force
Woolf, S. H. (2010) · JAMA · doi:10.1001/jama.2009.1989 · Cited by 75 · cited 1×
“However, for two decades, experts with different opinions about the methods used in the trials have reached different conclusions about the quality of the evidence and the likely mortality reduction from mammography at different ages [19].”
BackgroundSelf-citationWoolf, S. H., & George, J. N. (2000) · Hematology/Oncology Clinics of North America · doi:10.1016/s0889-8588(05)70310-5 · Cited by 69 · cited 1×
“These judgments inevitably color how groups characterize the evidence and frame recommendations in the face of uncertainty [20].”
BackgroundSelf-citationAn Annotated Algorithm Approach to Clinical Guideline Development
Hadorn, D. C. (1992) · JAMA · doi:10.1001/jama.1992.03480240073037 · Cited by 68 · cited 1×
“define the types of evidence to be reviewed and the outcomes represent measures of effectiveness, whereas the ‘arrows’ and outcomes in algorithms depict clinical choices, test results, or pathophysiologic events in the workup and treatment of patients [15,16].”
BackgroundImproving the use of research evidence in guideline development: 14. Reporting guidelines
Oxman, A. D., Schünemann, H. J., & Fretheim, A. (2006) · Health Research Policy and Systems · doi:10.1186/1478-4505-4-26 · Cited by 55
Self-citationHussain, T., Michel, G., & Shiffman, R. N. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.11.001 · Cited by 49 · cited 1×
“, should chromones be used in treatment of allergic rhinitis, given all other available treatment options?) [40].”
BackgroundAkl, E. A., Maroun, N., Guyatt, G. H., Oxman, A. D., Alonso‐Coello, P., Vist, G. E., Devereaux, P. J., Montori, V. M., & Schünemann, H. J. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.011 · Cited by 47 · cited 1×
“The little evidence that does exist suggests that consumers of healthcare recommendations prefer knowing about the underlying quality of evidence, and that symbols to indicate the strength of recommendations are more informative than numbers [44,45].”
BackgroundSelf-citationUsing Value of Information Analysis to Inform Publicly Funded Research Priorities
Ginnelly, L., Claxton, K., Sculpher, M., & Golder, S. (2005) · Applied Health Economics and Health Policy · doi:10.2165/00148365-200504010-00006 · Cited by 41 · cited 1×
“Nonetheless, the criteria in Table 4 must be met for a recommendation for use of interventions in the context of research to be sensible [63,64].”
BackgroundMaking recommendations on preventive practices: methodological issues.
Rn, B., & Sw, F. (1988) · PubMed · Cited by 40
Incorporating Patient Preferences in Evidence-Based Medicine—Reply
Krahn, M. D. (2008) · JAMA · doi:10.1001/jama.2008.731 · Cited by 39
Grading evidence and recommendations
Oxman, A. D., Briss, P. A., Eccles, M., Grimshaw, J., Guyatt, G. H., Phillips, B., Schünemann, H. J., Vist, G. E., & Zaza, S. (2001) · BMC News and views · doi:10.1186/2048-4623-1-s3-pc148 · Cited by 36
Self-citationInformed Decision Making in Outpatient Practice: Time to Get Back to Basics
Braddock III, C. H., Edwards, K. A., Hasenberg, N. M., Laidley, T. L., & Levinson, W. (1999) · JAMA · doi:10.1001/jama.282.24.2313 · Cited by 19 · cited 1×
“The second option for dealing with close calls is for the group to eschew a blanket recommendation but to instead encourage shared or informed decision-making, in which the patient is encouraged to review the tradeoffs with their clinician and make an individual decision based on personal preferences [28,29].”
BackgroundFrom BODE to ADO to outcomes in multimorbid COPD patients
Schünemann, H. J. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61539-7 · Cited by 12
Self-citation
Cited by
Dent, E., Morley, J. E., Cruz-Jentoft, A. J., Arai, H., Kritchevsky, S. B., Guralnik, J., Bauer, J. M., Pahor, M., Clark, B. C., Cesari, M., et al. (2018) · The Journal of nutrition, health and aging · doi:10.1007/s12603-018-1139-9 · Cited by 1,130
Physical Frailty: ICFSR International Clinical Practice Guidelines for Identification and Management
Dent, E., Morley, J. E., Cruz-Jentoft, A. J., Woodhouse, L., Rodríguez-Mañas, L., Fried, L. P., Woo, J., Aprahamian, I., Sanford, A., Lundy, J., et al. (2019) · The Journal of nutrition, health and aging · doi:10.1007/s12603-019-1273-z · Cited by 1,049
Munn, Z., Aromataris, E., Tufanaru, C., Stern, C., Porritt, K., Farrow, J., Lockwood, C., Stephenson, M., Moola, S., Lizarondo, L., et al. (2019) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000152 · Cited by 977
Laine, L., Kaltenbach, T., Barkun, A., McQuaid, K. R., Subramanian, V., Soetikno, R., East, J. E., Farraye, F. A., Feagan, B., Ioannidis, J., et al. (2015) · Gastroenterology · doi:10.1053/j.gastro.2015.01.031 · Cited by 776
Schünemann, H. J., Wiercioch, W., Etxeandia, I., Falavigna, M., Santesso, N., Mustafa, R., Ventresca, M., Brignardello-Petersen, R., Laisaar, K.-T., Kowalski, S., et al. (2014) · Canadian Medical Association Journal · doi:10.1503/cmaj.131237 · Cited by 686
Self-citation
Metadata
- Authors
- Woolf, S., Schünemann, H. J., Eccles, M. P., Grimshaw, J. M., & Shekelle, P.
- Venue
- Implementation Science
- Published
- 2012-07-04
- Volume
- 7
- Issue
- 1
- Pages
- 61-61
- ISSN
- 1748-5908 (ISSN-L)
- OpenAlex
- W2132864892
- Cited by
- 232
- References
- 67
- FWCI
- 9.26
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.