Improving the use of research evidence in guideline development: 16. Evaluation
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the last of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on evaluating guidelines and recommendations, including their quality, whether they are likely to be up-to-date, and their implementation. We also considered the role of guideline developers in undertaking evaluations that are needed to inform recommendations. METHODS: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: Our answers to these questions were informed by a review of instruments for evaluating guidelines, several studies of the need for updating guidelines, discussions of the pros and cons of different research designs for evaluating the implementation of guidelines, and consideration of the use of uncertainties identified in systematic reviews to set research priorities. How should the quality of guidelines or recommendations be appraised? WHO should put into place processes to ensure that both internal and external review of guidelines is undertaken routinely. A checklist, such as the AGREE instrument, should be used. The checklist should be adapted and tested to ensure that it is suitable to the broad range of recommendations that WHO produces, including public health and health policy recommendations, and that it includes questions about equity and other items that are particularly important for WHO guidelines. When should guidelines or recommendations be updated? Processes should be put into place to ensure that guidelines are monitored routinely to determine if they are in need of updating. People who are familiar with the topic, such as Cochrane review groups, should do focused, routine searches for new research that would require revision of the guideline. Periodic review of guidelines by experts not involved in developing the guidelines should also be considered. Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with members serving fixed periods with a rotating membership. How should the impact of guidelines or recommendations be evaluated? WHO headquarters and regional offices should support member states and those responsible for policy decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, practical support and coordination of efforts. Before-after evaluations should be used cautiously and when there are important uncertainties regarding the effects of a policy or its implementation, randomised evaluations should be used when possible. What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking? Guideline panels should routinely identify important uncertainties and research priorities. This source of potential priorities for research should be used systematically to inform priority-setting processes for global research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J., Thomas, R., MacLennan, G., Fraser, C., Ramsay, C., Vale, L., Whitty, P., Eccles, M., Matowe, L., Shirran, L., et al. (2004) · Health Technology Assessment · doi:10.3310/hta8060 · Cited by 3,044 · cited 2×
“Strategies ranging from passive dissemination to intensive, complex interventions have been used to implement guidelines and a range of study designs has been used to evaluate the impact of these strategies using a range of outcome measures [18,19].”
BackgroundMethod(2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.18 · Cited by 1,446
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J. M., Thomas, R. E., MacLennan, G., Fraser, C., Ramsay, C. R., Vale, L., Whitty, P., Eccles, M. P., Matowe, L., Shirran, L., et al. (2005) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462305290190 · Cited by 864
Research designs for studies evaluating the effectiveness of change and improvement strategies
Eccles, M. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.47 · Cited by 586
Cited by
Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines: 2010 Revision
Brożek, J. L., Bousquet, J., Baena-Cagnani, C. E., Bonini, S., Canonica, G. W., Casale, T. B., van Wijk, R. G., Ohta, K., Zuberbier, T., & Schünemann, H. J. (2010) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2010.06.047 · Cited by 1,813
Self-citationFrancke, A. L., Smit, M. C., de Veer, A. J., & Mistiaen, P. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-38 · Cited by 1,134
Schünemann, H. J., Wiercioch, W., Etxeandia, I., Falavigna, M., Santesso, N., Mustafa, R., Ventresca, M., Brignardello-Petersen, R., Laisaar, K.-T., Kowalski, S., et al. (2014) · Canadian Medical Association Journal · doi:10.1503/cmaj.131237 · Cited by 686
Self-citationA Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement
Chen, Y., Yang, K., Marušić, A., Qaseem, A., Meerpohl, J. J., Flottorp, S., Akl, E. A., Schünemann, H. J., Chan, E. S. Y., Falck-Ytter, Y., et al. (2017) · Annals of Internal Medicine · doi:10.7326/m16-1565 · Cited by 665
Self-citationAlonso-Coello, P., Irfan, A., Sola, I., Gich, I., Delgado-Noguera, M., Rigau, D., Tort, S., Bonfill, X., Burgers, J., & Schunemann, H. (2010) · BMJ Quality & Safety · doi:10.1136/qshc.2010.042077 · Cited by 472
Metadata
- Authors
- Oxman, A. D., Schünemann, H. J., & Fretheim, A.
- Venue
- Health Research Policy and Systems
- Published
- 2006-12
- Volume
- 4
- Issue
- 1
- Pages
- 28-28
- ISSN
- 1478-4505 (ISSN-L)
- OpenAlex
- W2145308289
- Cited by
- 70
- References
- 4
- FWCI
- 4.97
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Improving the use of research evidence in guideline development: 16. Evaluation. https://app.cochat.ai/papers/10.1186/1478-4505-4-28
- The paper
- Oxman, A. D., Schünemann, H. J., & Fretheim, A. (2006). Improving the use of research evidence in guideline development: 16. Evaluation. Health Research Policy and Systems. https://doi.org/10.1186/1478-4505-4-28
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.