An algorithm was developed to assign GRADE levels of evidence to comparisons within systematic reviews
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: One recommended use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach is supporting quality assessment of evidence of comparisons included within a Cochrane overview of reviews. Within our overview, reviewers found that current GRADE guidance was insufficient to make reliable and consistent judgments. To support our ratings, we developed an algorithm to grade quality of evidence using concrete rules. METHODS: Using a pragmatic, exploratory approach, we explored the challenges of applying GRADE levels of evidence and developed an algorithm to applying GRADE levels of evidence in a consistent and transparent approach. Our methods involved application of algorithms and formulas to samples of reviews, expert panel discussion, and iterative refinement and revision. RESULTS: The developed algorithm incorporated four key criteria: number of participants, risk of bias of trials, heterogeneity, and methodological quality of the review. A formula for applying GRADE level of evidence from the number of downgrades assigned by the algorithm was agreed. CONCLUSION: Our algorithm which assigns GRADE levels of evidence using a set of concrete rules was successfully applied within our Cochrane overview. We propose that this methodological approach has implications for assessment of quality of evidence within future evidence syntheses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755 · cited 5×
“We had previously assessed quality of each included review using the AMSTAR [17,18].”
BackgroundMethodInfluentialSavović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Interventions for improving upper limb function after stroke
Pollock, A., Farmer, S. E., Brady, M. C., Langhorne, P., Mead, G. E., Mehrholz, J., & van Wijck, F. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010820.pub2 · Cited by 909 · cited 2×
“was developed specifically for our Cochrane overview [5] and with careful consideration of the type of evidence included with this overview.”
BackgroundMethodSelf-citationInfluence of trial sample size on treatment effect estimates: meta-epidemiological study
Dechartres, A., Trinquart, L., Boutron, I., & Ravaud, P. (2013) · BMJ · doi:10.1136/bmj.f2304 · Cited by 544 · cited 1×
“[11] found that trials with 1,000 patients or more had, on average, a treatment effect which was 48% smaller than trials with less than 50 patients.”
ResultAssociation Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses
Dechartres, A., Altman, D. G., Trinquart, L., Boutron, I., & Ravaud, P. (2014) · JAMA · doi:10.1001/jama.2014.8166 · Cited by 233 · cited 1×
“[12] found that the treatment outcome estimates from pooled meta-analyses within systematic reviews were generally larger than the results of a meta-analysis of the largest trials only, particularly for subjective outcomes.”
Result
Cited by
Schwingshackl, L., Knüppel, S., Schwedhelm, C., Hoffmann, G., Missbach, B., Stelmach-Mardas, M., Dietrich, S., Eichelmann, F., Kontopanteils, E., Iqbal, K., et al. (2016) · Advances in Nutrition · doi:10.3945/an.116.013052 · Cited by 360
Prehabilitation in adult patients undergoing surgery: an umbrella review of systematic reviews
McIsaac, D. I., Gill, M., Boland, L., Hutton, B., Branje, K., Shaw, J., Grudzinski, A. L., Barone, N., Gillis, C., Akhtar, S., et al. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.11.014 · Cited by 312
Solmi, M., De Toffol, M., Kim, J. Y., Choi, M. J., Stubbs, B., Thompson, T., Firth, J., Miola, A., Croatto, G., Baggio, F., et al. (2023) · BMJ · doi:10.1136/bmj-2022-072348 · Cited by 271
Pollock, M., Fernandes, R. M., Becker, L. A., Featherstone, R., & Hartling, L. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0367-5 · Cited by 195
“These methods studies examined the following: implications of including multiple SRs published on the same topic area [24]; issues related to quality assessment of SRs [30–32]; different methods for presenting outcome data [33]; methods for assessing quality of evidence using GRADE [34, 35]; and reporting conflicts of…”
MethodInfluentialLunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0784-8 · Cited by 139
“For example, Pollock 2015 incorporated the limitations of included SRs in their GRADE assessment by rating down the certainty of evidence for SRs that did not meet criteria deemed to indicate important sources of bias [31, 32].”
Influential
Metadata
- Authors
- Pollock, A., Farmer, S. E., Brady, M. C., Langhorne, P., Mead, G. E., Mehrholz, J., van Wijck, F., & Wiffen, P. J.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2016-02
- Volume
- 70
- Pages
- 106-110
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1259937382
- Cited by
- 181
- References
- 10
- FWCI
- 0.77
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.