Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project
Open accessAbstract
BACKGROUND: International interest in clinical practice guidelines has never been greater but many published guidelines do not meet the basic quality requirements. There have been renewed calls for validated criteria to assess the quality of guidelines. OBJECTIVE: To develop and validate an international instrument for assessing the quality of the process and reporting of clinical practice guideline development. METHODS: The instrument was developed through a multi-staged process of item generation, selection and scaling, field testing, and refinement procedures. 100 guidelines selected from 11 participating countries were evaluated independently by 194 appraisers with the instrument. Following refinement the instrument was further field tested on three guidelines per country by a new set of 70 appraisers. RESULTS: The final version of the instrument contained 23 items grouped into six quality domains with a 4 point Likert scale to score each item (scope and purpose, stakeholder involvement, rigour of development, clarity and presentation, applicability, editorial independence). 95% of appraisers found the instrument useful for assessing guidelines. Reliability was acceptable for most domains (Cronbach's alpha 0.64-0.88). Guidelines produced as part of an established guideline programme had significantly higher scores on editorial independence and, after the publication of a national policy, had significantly higher quality scores on rigour of development (p<0.005). Guidelines with technical documentation had higher scores on that domain (p<0.0001). CONCLUSIONS: This is the first time an appraisal instrument for clinical practice guidelines has been developed and tested internationally. The instrument is sensitive to differences in important aspects of guidelines and can be used consistently and easily by a wide range of professionals from different backgrounds. The adoption of common standards should improve the consistency and quality of the reporting of guideline development worldwide and provide a framework to encourage international comparison of clinical practice guidelines.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Hyperinsulinemia is believed to worsen hyperandrogenism and the reproductive and metabolic manifestations of polycystic ovary syndrome in adolescents.
“Hyperinsulinemia is believed to exacerbate the hyperandrogenism and the reproductive and metabolic manifestations of PCOS in adolescents.”
From The Diagnosis of Polycystic Ovary Syndrome during Adolescence (2015) · cites this paper as [18]
With one published exception, current definitions of polycystic ovary syndrome in adolescent or adult women do not include insulin resistance or hyperinsulinemia as a diagnostic criterion.
“With the exception of one published recommendation, none of the current definitions of PCOS in adolescent or adult women include insulin resistance/hyperinsulinemia as a diagnostic criterion [1, 10-12, 18, 19, 112]”
From The Diagnosis of Polycystic Ovary Syndrome during Adolescence (2015) · cites this paper as [18]
Features of polycystic ovary syndrome overlap with normal pubertal development.
“Features of PCOS overlap normal pubertal development.”
From The Diagnosis of Polycystic Ovary Syndrome during Adolescence (2015) · cites this paper as [18]
Insulin resistance and hyperinsulinemia are intrinsic to polycystic ovary syndrome in adults and adolescents, whether lean or obese.
“Insulin resistance and hyperinsulinemia are intrinsic to PCOS in adults and adolescents, lean or obese.”
From The Diagnosis of Polycystic Ovary Syndrome during Adolescence (2015) · cites this paper as [18]
Diagnostic criteria for polycystic ovary syndrome in adolescence are controversial because adult diagnostic features may be normal pubertal events.
“The diagnostic criteria for PCOS in adolescence are controversial, primarily because the diagnostic pathological features used in adult women, e.g.acne, irregular menses, and PCOM, may be normal pubertal physiological events [18]”
From The Diagnosis of Polycystic Ovary Syndrome during Adolescence (2015) · cites this paper as [18]
For an assay using an extraction step, total testosterone concentrations greater than 55 ng/dl are likely consistent with hyperandrogenism.
“In general, for an assay using an extraction step, total testosterone concentrations >55 ng/dl are likely consistent with hyperandrogenism [18]”
From The Diagnosis of Polycystic Ovary Syndrome during Adolescence (2015) · cites this paper as [18]
A multifollicular pattern, defined by the presence of large follicles distributed throughout the ovary, does not have a relationship with hyperandrogenism, is more common in adolescents, and should not be considered a pathological finding.
“Further, a multifollicular pattern, which is defined by the presence of large follicles distributed throughout the ovary, does not have a relationship with hyperandrogenism, is more common in adolescents, and should not be considered a pathological finding (Level C).”
From The Diagnosis of Polycystic Ovary Syndrome during Adolescence (2015) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646 · see citing works
Sample size in factor analysis.
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.1.84 · Cited by 5,497 · see citing works
Statistics notes: Cronbach's alpha: Table 1
Bland, J. M., & Altman, D. G. (1997) · BMJ · doi:10.1136/bmj.314.7080.572 · Cited by 5,300 · see citing works
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755 · see citing works
Clinical guidelines: Potential benefits, limitations, and harms of clinical guidelines
Woolf, S. H., Grol, R. P. T. M., Hutchinson, A., Eccles, M., & Grimshaw, J. (1999) · BMJ · doi:10.1136/bmj.318.7182.527 · Cited by 2,549 · see citing works
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545 · see citing works
Toutenburg, H. (1974) · Biometrische Zeitschrift · doi:10.1002/bimj.19740160814 · Cited by 1,540 · see citing works
Moher, D., Schulz, K. F., Altman, D. G., & for the CONSORT Group* (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00011 · Cited by 1,426 · see citing works
Clinical guidelines: Developing guidelines
Shekelle, P., Woolf, S. H., Eccles, M., & Grimshaw, J. (1999) · BMJ · doi:10.1136/bmj.318.7183.593 · Cited by 1,220 · see citing works
Moher, D., Jones, A., Lepage, L., & for the CONSORT Group (2001) · JAMA · doi:10.1001/jama.285.15.1992 · Cited by 1,027 · see citing works
Guidelines for Clinical Practice
Institute of Medicine (1992) · National Academies Press eBooks · doi:10.17226/1863 · Cited by 916 · see citing works
Grol, R., Dalhuijsen, J., Thomas, S., Veld, C. I. ‘., Rutten, G., & Mokkink, H. (1998) · BMJ · doi:10.1136/bmj.317.7162.858 · Cited by 877 · see citing works
Are Guidelines Following Guidelines?
Shaneyfelt, T. M., Mayo‐Smith, M. F., & Rothwangl, J. (1999) · JAMA · doi:10.1001/jama.281.20.1900 · Cited by 846 · see citing works
The measurement of interrater agreement
Fleiss, J. (1981) · Medical Entomology and Zoology · Cited by 686 · see citing works
Practice guidelines developed by specialty societies: the need for a critical appraisal
Grilli, R. G., Magrini, N., Penna, A., Mura, G., & Liberati, A. (2000) · The Lancet · doi:10.1016/s0140-6736(99)02171-6 · Cited by 663 · see citing works
The Measurement of Interrater Agreement
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley series in probability and statistics · doi:10.1002/0471445428.ch18 · Cited by 519 · see citing works
Biostatistics: The Bare Essentials
Kendrach, M. G. (1997) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/54.2.223 · Cited by 476 · see citing works
Users' Guides to the Medical Literature
Hayward, R. S. (1995) · JAMA · doi:10.1001/jama.1995.03530070068032 · Cited by 392 · see citing works
Development and application of a generic methodology to assess the quality of clinical guidelines
Cluzeau, F., Littlejohns, P., Grimshaw, J., Feder, G. S., & Moran, S. (1999) · International Journal for Quality in Health Care · doi:10.1093/intqhc/11.1.21 · Cited by 271 · see citing works
Achieving health gain through clinical guidelines. I: Developing scientifically valid guidelines.
Grimshaw, J., & Russell, I. T. (1993) · BMJ Quality & Safety · doi:10.1136/qshc.2.4.243 · Cited by 266 · see citing works
Fortnightly Review: How to ensure that guidelines are effective
Thomson, R., Lavender, M., & Madhok, R. (1995) · BMJ · doi:10.1136/bmj.311.6999.237 · Cited by 188 · see citing works
GUIDELINE DEVELOPMENT IN EUROPE An International Comparison
(2000) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300103101 · Cited by 100 · see citing works
A Provisional Instrument for Assessing Clinical Practice Guidelines
Field, M. J., & Lohr, K. N. (1992) · Cited by 65 · see citing works
Marks, J. (1985) · Springer eBooks · doi:10.1007/978-94-009-4886-0_5 · Cited by 45 · see citing works
Cluzeau, F., & Littlejohns, P. (1999) · The Joint Commission Journal on Quality Improvement · doi:10.1016/s1070-3241(16)30465-5 · Cited by 42 · see citing works
Developing Clinical Practice Guidelines
Grol, R. P. T. M., Eccles, M., Maisonneuve, H., et al. (1998) · Disease Management & Health Outcomes · doi:10.2165/00115677-199804050-00002 · Cited by 37 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Thomson R, Lavender M, Madhok R. How to ensure that guidelines are effective. BMJ 1995;311:237-42.
- Lohr KN. The quality of practice guidelines and the quality of health care. In: Guidelines in health care. Report of a WHO Conference, January 1997, Baden-Baden: Nomos Verlagsgesellschaft, 1998.
- Bland JM, Altman DG. Cronbach's alpha. Statistics notes. BMJ 1997;314:572.
Cited by
Sorted by citations
Allergic Rhinitis and its Impact on Asthma (ARIA) 2008*
Bousquet, J., Khaltaev, N., Cruz, A. A., Denburg, J., Fokkens, W. J., Togias, A., Zuberbier, T., Baena‐Cagnani, C. E., Canonica, G. W., Van Weel, C., et al. (2008) · Allergy · doi:10.1111/j.1398-9995.2007.01620.x · Cited by 4,729 · see citing works
From best evidence to best practice: effective implementation of change in patients' care
Grol, R., & Grimshaw, J. (2003) · The Lancet · doi:10.1016/s0140-6736(03)14546-1 · Cited by 4,531 · see citing works
AGREE II: advancing guideline development, reporting and evaluation in health care
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Fervers, B., Graham, I. D., Grimshaw, J., Hanna, S. E., et al. (2010) · Canadian Medical Association Journal · doi:10.1503/cmaj.090449 · Cited by 3,790 · see citing works
Mosca, L., Benjamin, E. J., Berra, K., Bezanson, J. L., Dolor, R. J., Lloyd-Jones, D. M., Newby, L. K., Piña, I. L., Roger, V. L., Shaw, L. J., et al. (2011) · Circulation · doi:10.1161/cir.0b013e31820faaf8 · Cited by 2,143 · see citing works
Zeng, X., Zhang, Y., Kwong, J. S. W., Zhang, C., Li, S., Sun, F., Niu, Y., & Du, L. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12141 · Cited by 2,082 · see citing works
Elwyn, G. (2006) · BMJ · doi:10.1136/bmj.38926.629329.ae · Cited by 1,824 · see citing works
Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines: 2010 Revision
Brożek, J. L., Bousquet, J., Baena-Cagnani, C. E., Bonini, S., Canonica, G. W., Casale, T. B., van Wijk, R. G., Ohta, K., Zuberbier, T., & Schünemann, H. J. (2010) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2010.06.047 · Cited by 1,813 · see citing works
Ma, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020) · Military Medical Research · doi:10.1186/s40779-020-00238-8 · Cited by 1,648 · see citing works
Local opinion leaders: effects on professional practice and health care outcomes
Flodgren, G., Parmelli, E., Doumit, G., Gattellari, M., O'Brien, M. A., Grimshaw, J., & Eccles, M. P. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000125.pub4 · Cited by 1,070 · see citing works
Pimentel-Nunes, P., Libânio, D., Marcos-Pinto, R., Areia, M., Leja, M., Esposito, G., Garrido, M., Kikuste, I., Megraud, F., Matysiak-Budnik, T., et al. (2019) · Endoscopy · doi:10.1055/a-0859-1883 · Cited by 993 · see citing works
Guidelines International Network: Toward International Standards for Clinical Practice Guidelines
Qaseem, A., Forland, F., Macbeth, F., Ollenschläger, G., Phillips, S., van der Wees, P., & for the Board of Trustees of the Guidelines International Network* (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-7-201204030-00009 · Cited by 902 · see citing works
AGREE II: advancing guideline development, reporting, and evaluation in health care
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Fervers, B., Graham, I. D., Grimshaw, J., Hanna, S. E., et al. (2010) · Preventive Medicine · doi:10.1016/j.ypmed.2010.08.005 · Cited by 758 · see citing works
Is There Time for Management of Patients With Chronic Diseases in Primary Care?
Ostbye, T. (2005) · The Annals of Family Medicine · doi:10.1370/afm.310 · Cited by 698 · see citing works
A systematic development process for patient decision aids
Coulter, A., Stilwell, D., Kryworuchko, J., Mullen, P. D., Ng, C. J., & van der Weijden, T. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-s2-s2 · Cited by 615 · see citing works
Use of evidence in WHO recommendations
Oxman, A. D., Lavis, J. N., & Fretheim, A. (2007) · The Lancet · doi:10.1016/s0140-6736(07)60675-8 · Cited by 287 · see citing works
Messing, S., Rütten, A., Abu-Omar, K., Ungerer-Röhrich, U., Goodwin, L., Burlacu, I., & Gediga, G. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00055 · Cited by 259 · see citing works
“One independent researcher assessed the quality of reviews using the AGREE instrument (49).”
MethodWong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02300 · Cited by 276 · see citing works
Andrade, R., Pereira, R., van Cingel, R., Staal, J. B., & Espregueira-Mendes, J. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2018-100310 · Cited by 264 · see citing works
Qaseem, A., Snow, V., Owens, D. K., Shekelle, P., & for the Clinical Guidelines Committee of the American College of Physicians (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-3-201008030-00010 · Cited by 233 · see citing works
Seixas, M., Weiss, M., & Müller, U. (2012) · Journal of Psychopharmacology · doi:10.1177/0269881111412095 · Cited by 197 · see citing works
Evaluation and adaptation of clinical practice guidelines
Graham, I. D., & Harrison, M. B. (2005) · Evidence-Based Nursing · doi:10.1136/ebn.8.3.68 · Cited by 177 · see citing works
Gauthier, S., Patterson, C., Chertkow, H., Gordon, M., Herrmann, N., Rockwood, K., Rosa-Neto, P., & Soucy, J.-P. (2012) · Canadian Geriatrics Journal · doi:10.5770/cgj.15.49 · Cited by 137 · see citing works
Ferket, B. S. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.434 · Cited by 134 · see citing works
Improving the use of research evidence in guideline development: 14. Reporting guidelines
Oxman, A. D., Schünemann, H. J., & Fretheim, A. (2006) · Health Research Policy and Systems · doi:10.1186/1478-4505-4-26 · Cited by 55 · see citing works
Meyer-Bahlburg, H. F. L., Baratz Dalke, K., Berenbaum, S. A., Cohen-Kettenis, P. T., Hines, M., & Schober, J. M. (2016) · Hormone Research in Paediatrics · doi:10.1159/000442386 · Cited by 48 · see citing works
Clinical practice guidelines for irritable bowel syndrome in Korea, 2017 revised edition
Song, K. H., Jung, H.-K., Kim, H. J., Koo, H. S., Kwon, Y. H., Shin, H. D., Lim, H. C., Shin, J. E., Kim, S. E., Cho, D. H., et al. (2018) · Journal of Neurogastroenterology and Motility · doi:10.5056/jnm17145 · Cited by 47 · see citing works
Appleyard, T., Mann, C., & Khan, K. (2006) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2006.00937.x · Cited by 34 · see citing works
From expert-driven to user-oriented communication of infection control guidelines
Verhoeven, F., Steehouder, M. F., Hendrix, R. M. G., & Van Gemert-Pijnen, J. E. W. C. (2010) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.07.003 · Cited by 24 · see citing works
Zhuang, S., Ma, X., Xiao, G., Zhao, Y., Hou, J., & Wang, Y. (2022) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2022.04.007 · Cited by 15 · see citing works
The challenge-and promise-of local clinical practice guidelines
McCulloh, R. J., & Alverson, B. K. (2012) · Pediatrics · doi:10.1542/peds.2012-2085 · Cited by 4 · see citing works
- Venue
- Quality and Safety in Health Care
- Publisher
- BMJ
- Published
- 2003-02-01
- Volume
- 12
- Issue
- 1
- Pages
- 18-23
- ISSN
- 1475-3898 (ISSN-L)
- 1475-3901
- PMC
- PMC1743672
- OpenAlex
- W2138363439
- Cited by
- 1,446
- References
- 29
- Citations by year
- 2026: 14 · 2025: 35 · 2024: 41 · 2023: 32 · 2022: 48 · 2021: 62 · 2020: 67 · 2019: 54 · 2018: 52 · 2017: 61 · 2016: 65 · 2015: 69 · 2014: 90 · 2013: 75 · 2012: 121
- FWCI
- 41.31
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- BMJ Quality & Safety (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- University of Aberdeen
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Health Education and Validation
- Keywords
- guideline quality assessment, AGREE instrument, guideline development, reporting quality, stakeholder engagement, editorial independence, international comparison of guidelines, instrument validation, reliability statistics
- MeSH terms
- Europe, Health Services Research, Humans, Program Evaluation, Program Development, Practice Guidelines as Topic, Quality Indicators, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
