The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
A checklist designed to aid consistency and reproducibility of GRADE assessments: development and pilot validation
Open accessAbstract
BACKGROUND: The grading of recommendation, assessment, development and evaluation (GRADE) approach is widely implemented in health technology assessment and guideline development organisations throughout the world. GRADE provides a transparent approach to reaching judgements about the quality of evidence on the effects of a health care intervention, but is complex and therefore challenging to apply in a consistent manner. METHODS: We developed a checklist to guide the researcher to extract the data required to make a GRADE assessment. We applied the checklist to 29 meta-analyses of randomised controlled trials on the effectiveness of health care interventions. Two reviewers used the checklist for each paper and used these data to rate the quality of evidence for a particular outcome. RESULTS: For most (70%) checklist items, there was good agreement between reviewers. The main problems were for items relating to indirectness where considerable judgement is required. CONCLUSIONS: There was consistent agreement between reviewers on most items in the checklist. The use of this checklist may be an aid to improving the consistency and reproducibility of GRADE assessments, particularly for inexperienced users or in rapid reviews without the resources to conduct assessments by two researchers independently.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology
Guyatt, G. H., Oxman, A. D., Schünemann, H. J., Tugwell, P., & Knottnerus, A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.011 · Cited by 3,094
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
GRADE guidelines 6. Rating the quality of evidence—imprecision
Guyatt, G. H., Oxman, A. D., Kunz, R., Brozek, J., Alonso-Coello, P., Rind, D., Devereaux, P. J., Montori, V. M., Freyschuss, B., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.012 · Cited by 2,693
GRADE guidelines: 7. Rating the quality of evidence—inconsistency
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Glasziou, P., Jaeschke, R., Akl, E. A., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.03.017 · Cited by 2,320
Self-citationGRADE guidelines: 2. Framing the question and deciding on important outcomes
Guyatt, G. H., Oxman, A. D., Kunz, R., Atkins, D., Brozek, J., Vist, G., Alderson, P., Glasziou, P., Falck-Ytter, Y., & Schünemann, H. J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.012 · Cited by 1,911 · cited 4×
“Population of the model was based on articles written by the GRADE working group with particular emphasis on the most recent series of articles in the JCE [1,3-15].”
MethodInfluentialCholinesterase inhibitors for Alzheimer's disease
Birks, J. S. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005593 · Cited by 1,906
GRADE guidelines: 8. Rating the quality of evidence—indirectness
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Falck-Ytter, Y., Jaeschke, R., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.014 · Cited by 1,843
GRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819
Andrews, J., Guyatt, G., Oxman, A. D., Alderson, P., Dahm, P., Falck-Ytter, Y., Nasser, M., Meerpohl, J., Post, P. N., Kunz, R., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.03.013 · Cited by 1,416
Thrombolysis for acute ischaemic stroke
Wardlaw, J. M., Murray, V., Berge, E., & del Zoppo, G. J. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000213.pub3 · Cited by 1,370
GRADE guidelines: 9. Rating up the quality of evidence
Guyatt, G. H., Oxman, A. D., Sultan, S., Glasziou, P., Akl, E. A., Alonso-Coello, P., Atkins, D., Kunz, R., Brozek, J., Montori, V., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.06.004 · Cited by 1,336
Andrews, J. C., Schünemann, H. J., Oxman, A. D., Pottie, K., Meerpohl, J. J., Coello, P. A., Rind, D., Montori, V. M., Brito, J. P., Norris, S., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.02.003 · Cited by 1,315
Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013196 · Cited by 1,311
GRADE guidelines: 12. Preparing Summary of Findings tables—binary outcomes
Guyatt, G. H., Oxman, A. D., Santesso, N., Helfand, M., Vist, G., Kunz, R., Brozek, J., Norris, S., Meerpohl, J., Djulbegovic, B., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.01.012 · Cited by 836
GRADE guidelines: 13. Preparing Summary of Findings tables and evidence profiles—continuous outcomes
Guyatt, G. H., Thorlund, K., Oxman, A. D., Walter, S. D., Patrick, D., Furukawa, T. A., Johnston, B. C., Karanicolas, P., Akl, E. A., Vist, G., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.08.001 · Cited by 772
Guyatt, G., Oxman, A. D., Sultan, S., Brozek, J., Glasziou, P., Alonso-Coello, P., Atkins, D., Kunz, R., Montori, V., Jaeschke, R., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.01.006 · Cited by 762
Eyding, D., Lelgemann, M., Grouven, U., Harter, M., Kromp, M., Kaiser, T., Kerekes, M. F., Gerken, M., & Wieseler, B. (2010) · BMJ · doi:10.1136/bmj.c4737 · Cited by 588
Mustafa, R. A., Santesso, N., Brozek, J., Akl, E. A., Walter, S. D., Norman, G., Kulasegaram, M., Christensen, R., Guyatt, G. H., Falck-Ytter, Y., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.02.004 · Cited by 370 · cited 3×
“It is also consistent with the methods used in a previous evaluation of the inter-rater agreement for GRADE assessments [18].”
BackgroundMethodInfluentialSelf-citationJohnston, B. C., Ma, S. S. Y., Goldenberg, J. Z., Thorlund, K., Vandvik, P. O., Loeb, M., & Guyatt, G. H. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-12-201212180-00563 · Cited by 367
Soomro, G. M., Altman, D. G., Rajagopal, S., & Oakley Browne, M. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001765.pub3 · Cited by 342
Lowe, C. J. M., Barker, K. L., Dewey, M., & Sackley, C. M. (2007) · BMJ · doi:10.1136/bmj.39311.460093.be · Cited by 295
Nelson, J. C., & Devanand, D. P. (2011) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2011.03355.x · Cited by 284
Cape, J., Whittington, C., Buszewicz, M., Wallace, P., & Underwood, L. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-38 · Cited by 283 · cited 2×
“[25] Brief psychological therapies for anxiety and depression in primary care: metaanalysis and meta-regression Psychological Mental health”
BackgroundMethodGould, R. L., Coulson, M. C., & Howard, R. J. (2012) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2011.03824.x · Cited by 267
Thrombolysis for acute ischaemic stroke
Wardlaw, J. M., Murray, V., Berge, E., & del Zoppo, G. J. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000213.pub2 · Cited by 267
Makani, H., Bangalore, S., Desouza, K. A., Shah, A., & Messerli, F. H. (2013) · BMJ · doi:10.1136/bmj.f360 · Cited by 244
Pinto, R. Z., Maher, C. G., Ferreira, M. L., Hancock, M., Oliveira, V. C., McLachlan, A. J., Koes, B., & Ferreira, P. H. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-12-201212180-00564 · Cited by 243
Methadone at tapered doses for the management of opioid withdrawal
Amato, L., Davoli, M., Minozzi, S., Ferroni, E., Ali, R., & Ferri, M. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003409.pub4 · Cited by 222
Antidepressants for depression in physically ill people
Rayner, L., Price, A., Evans, A., Valsraj, K., Higginson, I. J., & Hotopf, M. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007503.pub2 · Cited by 195
Escitalopram versus other antidepressive agents for depression
Cipriani, A., Santilli, C., Furukawa, T. A., Signoretti, A., Nakagawa, A., McGuire, H., Churchill, R., & Barbui, C. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006532.pub2 · Cited by 190
Harkness, E., Macdonald, W., Valderas, J., Coventry, P., Gask, L., & Bower, P. (2010) · Diabetes Care · doi:10.2337/dc09-1519 · Cited by 188
Lipid-Modifying Therapies and Risk of Pancreatitis: A Meta-analysis
Preiss, D., Tikkanen, M. J., Welsh, P., Ford, I., Lovato, L. C., Elam, M. B., LaRosa, J. C., DeMicco, D. A., Colhoun, H. M., Goldenberg, I., et al. (2012) · JAMA · doi:10.1001/jama.2012.8439 · Cited by 181
Andrews, T., Thompson, M., Buckley, D., Heneghan, C. J., Deyo, R. A., Redmond, N. M., Lucas, P. J., Blair, P. S., & Hay, A. D. (2012) · PLoS ONE · doi:10.1371/journal.pone.0030334 · Cited by 152
Eaton, L. A., Huedo-Medina, T. B., Kalichman, S. C., Pellowski, J. A., Sagherian, M. J., Warren, M., Popat, A. R., & Johnson, B. T. (2012) · American Journal of Public Health · doi:10.2105/ajph.2012.300968 · Cited by 103
Skapinakis, P., Bakola, E., Salanti, G., Lewis, G., Kyritsis, A., & Mavreas, V. (2010) · BMC Neurology · doi:10.1186/1471-2377-10-49 · Cited by 100
Care management for Type 2 diabetes in the United States: a systematic review and meta-analysis
Egginton, J. S., Ridgeway, J. L., Shah, N. D., Balasubramaniam, S. C., Emmanuel, J. R., Prokop, L. J., Montori, V. M., & Murad, M. H. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-72 · Cited by 89
Introducing GRADE across the NICE clinical guideline program
Thornton, J., Alderson, P., Tan, T., Turner, C., Latchem, S., Shaw, E. J., Ruiz, F. J., Reken, S., Mugglestone, M. A., Hill, J., et al. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.12.007 · Cited by 69
Chen, X., Liu, Y., Røe, O. D., Qian, Y., Guo, R., Zhu, L., Yin, Y., & Shu, Y. (2013) · PLoS ONE · doi:10.1371/journal.pone.0059314 · Cited by 68
Huedo-Medina, T. B., Boynton, M. H., Warren, M. R., LaCroix, J. M., Carey, M. P., & Johnson, B. T. (2010) · AIDS and Behavior · doi:10.1007/s10461-010-9763-3 · Cited by 52
Hartling, L., Fernandes, R. M., Seida, J. C., Vandermeer, B., & Dryden, D. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0034697 · Cited by 41
Zhong, C., Liu, H., Jiang, L., Zhang, W., & Yao, F. (2013) · PLoS ONE · doi:10.1371/journal.pone.0058466 · Cited by 35
Pérez-Mañá, C., Castells, X., Vidal, X., Casas, M., & Capellà, D. (2011) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2010.08.012 · Cited by 33
Ye, Q., & Chen, H. (2013) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-013-2820-1 · Cited by 26
Cited by
Selenium for preventing cancer
Vinceti, M., Filippini, T., Del Giovane, C., Dennert, G., Zwahlen, M., Brinkman, M., Zeegers, M. P., Horneber, M., D'Amico, R., & Crespi, C. M. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005195.pub4 · Cited by 649
Mead, E., Brown, T., Rees, K., Azevedo, L. B., Whittaker, V., Jones, D., Olajide, J., Mainardi, G. M., Corpeleijn, E., O'Malley, C., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012651 · Cited by 518
Al-Khudairy, L., Loveman, E., Colquitt, J. L., Mead, E., Johnson, R. E., Fraser, H., Olajide, J., Murphy, M., Velho, R. M., O'Malley, C., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012691 · Cited by 455
Communication skills training for healthcare professionals working with people who have cancer
Moore, P. M., Rivera Mercado, S., Grez Artigues, M., & Lawrie, T. A. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003751.pub3 · Cited by 452
Lymphadenectomy for the management of endometrial cancer
Frost, J. A., Webster, K. E., Bryant, A., & Morrison, J. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007585.pub4 · Cited by 430
Metadata
- Authors
- Meader, N., King, K., Llewellyn, A., Norman, G., Brown, J., Rodgers, M., Moe-Byrne, T., Higgins, J. P., Sowden, A., & Stewart, G.
- Venue
- Systematic Reviews
- Published
- 2014-07-24
- Volume
- 3
- Issue
- 1
- Pages
- 82-82
- ISSN
- 2046-4053 (ISSN-L)
- OpenAlex
- W2171111655
- Cited by
- 610
- References
- 49
- FWCI
- 32.99
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.