Guideline appraisal with AGREE II: systematic review of the current evidence on how users handle the 2 overall assessments
Open accessAbstract
INTRODUCTION: The Appraisal of Guidelines for Research & Evaluation (AGREE) II instrument is the most commonly used guideline appraisal tool. It includes 23 appraisal criteria (items) organized within 6 domains and 2 overall assessments (1. overall guideline quality; 2. recommendation for use). The aim of this systematic review was twofold. Firstly, to investigate how often AGREE II users conduct the 2 overall assessments. Secondly, to investigate the influence of the 6 domain scores on each of the 2 overall assessments. MATERIALS AND METHODS: A systematic bibliographic search was conducted for publications reporting guideline appraisals with AGREE II. The impact of the 6 domain scores on the overall assessment of guideline quality was examined using a multiple linear regression model. Their impact on the recommendation for use (possible answers: "yes", "yes, with modifications", "no") was examined using a multinomial regression model. RESULTS: 118 relevant publications including 1453 guidelines were identified. 77.1% of the publications reported results for at least one overall assessment, but only 32.2% reported results for both overall assessments. The results of the regression analyses showed a statistically significant influence of all domains on overall guideline quality, with Domain 3 (rigour of development) having the strongest influence. For the recommendation for use, the results showed a significant influence of Domains 3 to 5 ("yes" vs. "no") and Domains 3 and 5 ("yes, with modifications" vs. "no"). CONCLUSIONS: The 2 overall assessments of AGREE II are underreported by guideline assessors. Domains 3 and 5 have the strongest influence on the results of the 2 overall assessments, while the other domains have a varying influence. Within a normative approach, our findings could be used as guidance for weighting individual domains in AGREE II to make the overall assessments more objective. Alternatively, a stronger content analysis of the individual domains could clarify their importance in terms of guideline quality. Moreover, AGREE II should require users to transparently present how they conducted the assessments.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study adapted relevant portions of the AGREE II instrument to extract data on methodologic and reporting quality in the reviewed clinical practice guidelines.
“We adapted relevant portions of the Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) [28] to extract data on methodologic and reporting quality in the reviewed CPGs.”
Methodologic and reporting quality was not assumed to be high among the guidelines, so key features of quality were assessed using criteria adapted from AGREE II.
“methodologic and reporting quality was not assumed to be high among the guidelines, so key features of quality deemed relevant by the investigators were assessed using criteria adapted from the AGREE II [28] appraisal tool for guidelines”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Audit and feedback: effects on professional practice and healthcare outcomes
Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O'Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000259.pub3 · Cited by 5,485 · see citing works
Clinical Practice Guidelines We Can Trust
Board on Health Care Services (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,102 · see citing works
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J., Thomas, R., MacLennan, G., Fraser, C., Ramsay, C., Vale, L., Whitty, P., Eccles, M., Matowe, L., Shirran, L., et al. (2004) · Health Technology Assessment · doi:10.3310/hta8060 · Cited by 3,044 · see citing works
Local opinion leaders: effects on professional practice and health care outcomes
Flodgren, G., Parmelli, E., Doumit, G., Gattellari, M., O'Brien, M. A., Grimshaw, J., & Eccles, M. P. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000125.pub4 · Cited by 1,070 · see citing works
Are Guidelines Following Guidelines?
Shaneyfelt, T. M., Mayo‐Smith, M. F., & Rothwangl, J. (1999) · JAMA · doi:10.1001/jama.281.20.1900 · Cited by 846 · see citing works
Rotter, T., Kinsman, L., James, E. L., Machotta, A., Gothe, H., Willis, J., Snow, P. C., & Kügler, J. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006632.pub2 · Cited by 824 · see citing works
Tailored interventions to address determinants of practice
Baker, R., Camosso-Stefinovic, J., Gillies, C., Shaw, E. J., Cheater, F., Flottorp, S., Robertson, N., Wensing, M., Fiander, M., Eccles, M. P., et al. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005470.pub3 · Cited by 711 · see citing works
Practice guidelines developed by specialty societies: the need for a critical appraisal
Grilli, R. G., Magrini, N., Penna, A., Mura, G., & Liberati, A. (2000) · The Lancet · doi:10.1016/s0140-6736(99)02171-6 · Cited by 663 · see citing works
Systematic Review and Meta-Analysis of Practice Facilitation Within Primary Care Settings
Baskerville, N. B., Liddy, C., & Hogg, W. (2012) · The Annals of Family Medicine · doi:10.1370/afm.1312 · Cited by 571 · see citing works
Schneider, A., Hommel, G., & Blettner, M. (2010) · Deutsches Ärzteblatt international · doi:10.3238/arztebl.2010.0776 · Cited by 526 · see citing works
Alonso‐Coello, P., Irfan, A., Solà, I., Gich, I. J., Delgado‐Noguera, M., Comas, D. R., Tort, S., Bonfill, X., Burgers, J. S., & Schünemann, H. J. (2010) · BMJ Quality & Safety · doi:10.1136/qshc.2010.042077 · Cited by 473 · see citing works
Failure of Clinical Practice Guidelines to Meet Institute of Medicine Standards
Kung, J., Miller, R. R., & Mackowiak, P. A. (2012) · Archives of Internal Medicine · doi:10.1001/2013.jamainternmed.56 · Cited by 293 · see citing works
Ordinal Logistic Regression in Medical Research
Bender, R., & Grouven, U. (1997) · Journal of the Royal College of Physicians of London · doi:10.1016/s0035-8819(25)00123-0 · Cited by 289 · see citing works
Ensuring Quality Cancer Care by the Use of Clinical Practice Guidelines and Critical Pathways
Smith, T. J., & Hillner, B. E. (2001) · Journal of Clinical Oncology · doi:10.1200/jco.2001.19.11.2886 · Cited by 239 · see citing works
Guideline implementation in allied health professions: a systematic review of the literature
Hakkennes, S., & Dodd, K. (2008) · Quality and Safety in Health Care · doi:10.1136/qshc.2007.023804 · Cited by 218 · see citing works
Local opinion leaders: effects on professional practice and health care outcomes
Flodgren, G., Parmelli, E., Doumit, G. D., Gattellari, M., O’Brien, M. A., Grimshaw, J., & Eccles, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000125.pub3 · Cited by 191 · see citing works
Zhang, A. H., & Liu, X. H. (2011) · International Journal of Evidence-Based Healthcare · doi:10.1111/j.1744-1609.2011.00223.x · Cited by 190 · see citing works
Gagliardi, A. R., & Brouwers, M. C. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-007047 · Cited by 176 · see citing works
Pediatric Clinical Practice Guidelines for Acute Procedural Pain: A Systematic Review
Lee, G. Y., Yamada, J., Kyololo, O. M., Shorkey, A., & Stevens, B. (2014) · PEDIATRICS · doi:10.1542/peds.2013-2744 · Cited by 163 · see citing works
Tunnicliffe, D. J., Singh‐Grewal, D., Kim, S., Craig, J. C., & Tong, A. (2015) · Arthritis Care & Research · doi:10.1002/acr.22591 · Cited by 154 · see citing works
Medves, J., Godfrey, C. M., Turner, C., Paterson, M., Harrison, M., MacKenzie, L. D., & Durando, P. (2010) · International Journal of Evidence-Based Healthcare · doi:10.1111/j.1744-1609.2010.00166.x · Cited by 150 · see citing works
Clinical guidelines for postpartum women and infants in primary care–a systematic review
Haran, C., van Driel, M., Mitchell, B. L., & Brodribb, W. E. (2014) · BMC Pregnancy and Childbirth · doi:10.1186/1471-2393-14-51 · Cited by 142 · see citing works
Guidelines for Therapeutic Drug Monitoring of Vancomycin: A Systematic Review
Zhikang, Y., Li, C., & Zhai, S.-D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099044 · Cited by 106 · see citing works
Brosseau, L., Rahman, P., Toupin‐April, K., Poitras, S., KING, J. A. C., Angelis, G. D., Loew, L., Casimiro, L., Paterson, G., & McEwan, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0082986 · Cited by 106 · see citing works
Impact of a Clinical Guidelines Program for Breast and Colon Cancer in a French Cancer Center
Ray-Coquard, I. L. (1997) · JAMA · doi:10.1001/jama.1997.03550190055044 · Cited by 103 · see citing works
Ray-Coquard, I., Philip, T. O., de Laroche, G., Froger, X., Jp, S., Voloch, A., Mathieu‐Daudé, H., Fervers, B., Farsi, F., Browman, G. P., et al. (2002) · British Journal of Cancer · doi:10.1038/sj.bjc.6600057 · Cited by 97 · see citing works
Predictors of high quality clinical practice guidelines: examples in oncology
Fervers, B., Burgers, J. S., Haugh, M. C., Brouwers, M. C., Browman, G. P., Cluzeau, F., & Philip, T. O. (2005) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzi011 · Cited by 97 · see citing works
Quality of Clinical Practice Guidelines for Glycemic Control in Type 2 Diabetes Mellitus
Holmer, H., Ogden, L., Burda, B., & Norris, S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0058625 · Cited by 86 · see citing works
Kryworuchko, J., Stacey, D., Bai, N., & Graham, I. D. (2009) · Implementation Science · doi:10.1186/1748-5908-4-49 · Cited by 83 · see citing works
Sabharwal, S., Patel, N. K., Gauher, S., Holloway, I., & Athansiou, T. (2014) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-014-3530-0 · Cited by 76 · see citing works
Altman, R. D., Schemitsch, E. H., & Bedi, A. (2015) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2015.04.013 · Cited by 74 · see citing works
Brusamento, S., Legido‐Quigley, H., Пантели, Д., Turk, E., Knai, C., Saliba, V., Car, J., McKee, M., & Busse, R. (2012) · Health Policy · doi:10.1016/j.healthpol.2012.08.005 · Cited by 73 · see citing works
Brosseau, L., Rahman, P., Poitras, S., Toupin‐April, K., Paterson, G., Smith, C. A. M., KING, J. A. C., Casimiro, L., Angelis, G. D., Loew, L., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0095369 · Cited by 66 · see citing works
Lopez‐Vargas, P. A., Tong, A., Sureshkumar, P., Johnson, D. W., & Craig, J. C. (2013) · Nephrology · doi:10.1111/nep.12119 · Cited by 63 · see citing works
Koh, C., Zhao, X., Samala, N. R., et al. (2013) · Hepatology · doi:10.1002/hep.26578 · Cited by 60 · see citing works
Appraisal of WHO Guidelines in Maternal Health Using the AGREE II Assessment Tool
Polus, S., Lerberg, P. M., Vogel, J. P., Watananirun, K., Souza, J. P., Mathai, M., & Gülmezoglu, A. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0038891 · Cited by 53 · see citing works
Appraisal of guidelines developed by the World Health Organization
Burda, B. U., Chambers, A., & Johnson, J. (2014) · Public Health · doi:10.1016/j.puhe.2014.01.002 · Cited by 49 · see citing works
Diagnosis of pheochromocytoma: a clinical practice guideline appraisal using AGREE II instrument
Yan, J., Min, J., & Zhou, B. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2012.01873.x · Cited by 45 · see citing works
Shen, J., Sun, M., Zhou, B., & Yan, J. (2014) · European Journal of Endocrinology · doi:10.1530/eje-14-0345 · Cited by 44 · see citing works
van den Berg, J., & Berger, M. Y. (2011) · BMC Family Practice · doi:10.1186/1471-2296-12-134 · Cited by 41 · see citing works
Occupational Asthma guidelines: a systematic quality appraisal using the AGREE II instrument
Lytras, T., Bonovas, S., Chronis, C., Konstantinidis, A. K., Kopsachilis, F., Papamichail, D. P., & Dounias, G. (2013) · Occupational and Environmental Medicine · doi:10.1136/oemed-2013-101656 · Cited by 40 · see citing works
To what extent are current guidelines for cutaneous melanoma follow up based on scientific evidence?
Marciano, N. J., Merlin, T., Bessen, T., & Street, J. (2014) · International Journal of Clinical Practice · doi:10.1111/ijcp.12393 · Cited by 37 · see citing works
Quality of Guidelines for Cognitive Rehabilitation Following Traumatic Brain Injury
Bragge, P., Pattuwage, L., Marshall, S., et al. (2014) · Journal of Head Trauma Rehabilitation · doi:10.1097/htr.0000000000000066 · Cited by 33 · see citing works
Critical evaluation of the scientific content in clinical practice guidelines
Abdelsattar, Z. M., Reames, B. N., Regenbogen, S. E., et al. (2014) · Cancer · doi:10.1002/cncr.29124 · Cited by 28 · see citing works
Chua, M. E., Mendoza, J., See, M., et al. (2015) · Canadian Urological Association Journal · doi:10.5489/cuaj.2424 · Cited by 28 · see citing works
Burnett, H. F., Tanoshima, R., Chandranipapongse, W., et al. (2014) · The Pharmacogenomics Journal · doi:10.1038/tpj.2014.47 · Cited by 28 · see citing works
Creating Clinical Practice Guidelines
Guyatt, G. H., & Vandvik, P. O. (2013) · CHEST Journal · doi:10.1378/chest.13-0463 · Cited by 28 · see citing works
Yaman, M. E., Güdeloğlu, A., Şentürk, S., et al. (2015) · The Spine Journal · doi:10.1016/j.spinee.2015.01.012 · Cited by 24 · see citing works
Wang, Y., Luo, Q., Li, Y., Wang, H., Deng, S., Wei, S., & Li, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0103939 · Cited by 20 · see citing works
Damiani, G., Silvestrini, G., Trozzi, L., Maci, D., Iodice, L., & Ricciardi, W. (2014) · Clinical Interventions in Aging · doi:10.2147/cia.s65046 · Cited by 19 · see citing works
Ríos, E., Pamela Serón, S., Lanas, F. T., Bonfill, X., Quigley, E. M. M., & Alonso‐Coello, P. (2014) · European Journal of Gastroenterology & Hepatology · doi:10.1097/meg.0000000000000033 · Cited by 18 · see citing works
Haddadi, M., Muhammadnejad, S., Sadeghi-Fazel, F., Zandieh, Z., Rahimi, G., Sadighi, S., Akbari, P., Mohagheghi, M. A., Mosavi-Jarrahi, A., & Amanpour, S. (2015) · Asian Pacific Journal of Cancer Prevention · doi:10.7314/apjcp.2015.16.3.1057 · Cited by 14 · see citing works
Interventions to Modify Health Care Provider Adherence to Asthma Guidelines
Okelo, S. O., Butz, A. M., Sharma, R., et al. (2013) · Europe PMC (PubMed Central) · Cited by 13 · see citing works
Tudor, K. I., Kozina, P. N., & Marušić, A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0069877 · Cited by 13 · see citing works
Santos, F., Solà, I., Rigau, D., et al. (2012) · Current Clinical Pharmacology · doi:10.2174/157488412799218842 · Cited by 12 · see citing works
White, P. E., Shee, A. W., & Finch, C. F. (2013) · British Journal of Sports Medicine · doi:10.1136/bjsports-2013-092720 · Cited by 11 · see citing works
Schoenmaker, N. J., Tromp, W. F., van der Lee, J. H., et al. (2013) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gft303 · Cited by 10 · see citing works
Quality assessment of guidelines on thromboprophylaxis in orthopaedic surgery
Sabharwal, S., Gauher, S., Kyriacou, S., et al. (2014) · The Bone & Joint Journal · doi:10.1302/0301-620x.96b1.32943 · Cited by 10 · see citing works
Gutarra-Vilchez, R. B., Barajas-Nava, L. A., Alemán, A., Solà, I., Gich, I. J., Bonfill, X., & Alonso‐Coello, P. (2014) · Human Fertility · doi:10.3109/14647273.2013.872301 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [25]Siering U, Hoffmann-Eßer W, Neugebauer EA, Eikermann M. Is there a cut-off for high-quality guidelines? A systematic analysis of current guideline appraisals using the AGREE-II instrument [poster]. G-I-N Conference; 7th - 10th October 2015; Amsterdam, The Netherlands.
Cited by
Sorted by citations
Cara, K. C., Goldman, D. M., Kollman, B. K., Amato, S. S., Tull, M. D., & Karlsen, M. C. (2023) · Advances in Nutrition · doi:10.1016/j.advnut.2023.03.007 · Cited by 44 · see citing works
- Authors
- Hoffmann-Eßer, W., Siering, U., Neugebauer, E. A. M., Brockhaus, A. C., Lampert, U., & Eikermann, M.
- Affiliations
- Witten/Herdecke University (DE)
- Institute for Quality and Efficiency in Health Care (DE)
- Corresponding author
- Hoffmann-Eßer, Wiebke
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-03-30
- Volume
- 12
- Issue
- 3
- Pages
- e0174831
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5373625
- OpenAlex
- W2598568459
- Cited by
- 171
- References
- 60
- Citations by year
- 2026: 17 · 2025: 31 · 2024: 22 · 2023: 18 · 2022: 17 · 2021: 19 · 2020: 17 · 2019: 19 · 2018: 11
- FWCI
- 2.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- AGREE II, guideline evaluation
- MeSH terms
- Literature, Regression Analysis, Software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
