The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Abstract
In the GRADE approach, randomized trials start as high-quality evidence and observational studies as low-quality evidence, but both can be rated down if most of the relevant evidence comes from studies that suffer from a high risk of bias. Well-established limitations of randomized trials include failure to conceal allocation, failure to blind, loss to follow-up, and failure to appropriately consider the intention-to-treat principle. More recently recognized limitations include stopping early for apparent benefit and selective reporting of outcomes according to the results. Key limitations of observational studies include use of inappropriate controls and failure to adequately adjust for prognostic imbalance. Risk of bias may vary across outcomes (e.g., loss to follow-up may be far less for all-cause mortality than for quality of life), a consideration that many systematic reviews ignore. In deciding whether to rate down for risk of bias--whether for randomized trials or observational studies--authors should not take an approach that averages across studies. Rather, for any individual outcome, when there are some studies with a high risk, and some with a low risk of bias, they should consider including only the studies with a lower risk of bias.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Certainty of evidence was assessed with use of GRADE tools for RCTs (Guyatt et al., 2011; Ryan & Hill, 2016).”
From Effectiveness of Mental Health and Wellbeing Interventions for Children and Young People in Foster, Kinship, and Residential Care: Systematic Review and Meta-Analysis (2024) · cites this paper as Guyatt et al., 2011
In the citing paper
“Additionally, the researchers (JL and WS) will assess the evidence profiles for each of the identified outcomes based on the GRADE (Grading of Recommendations, Assessment, Development and Evaluation) approach to reviewing evidence [86–88].”
From Acceptance and commitment therapy for chronic pain: Protocol of a systematic review and individual participant data meta-Analysis (2019) · cites this paper as [88]
In the citing paper
“Relevant criteria from the Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) handbook were used to assess bias in all other study designs [37].”
From Computerized cognitive behavior therapy for anxiety and depression in rural areas:a systematic review (2015) · cites this paper as [37]
In the citing paper
“The overall quality of evidence was graded using the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach, considering study limitations, inconsistency, indirectness, imprecision and publication bias [40–44].”
In the citing paper
“We also assessed each study for risk of potential bias following the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach and recorded the information in the extraction template469.”
From Health effects associated with exposure to secondhand smoke: a Burden of Proof study (2024) · cites this paper as [469]
In the citing paper
“Quality of evidence in included systematic reviews was assessed by outcome using the Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) system criteria [17-27].”
From Effects of Home Telemonitoring Interventions on Patients With Chronic Heart Failure: An Overview of Systematic Reviews (2015) · cites this paper as [20]
In the citing paper
“applied the key criteria of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidelines for observational studies to assess the methodological quality of nonrandomised studies.31”
In the citing paper
“we assessed the risk of selection bias, performance bias, selective reporting bias, detection bias, attrition bias, and potential confounding [32].”
In the citing paper
“The overall quality of research evidence for mental health indicators was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework [32].”
In the citing paper
“The overall quality of evidence was rated as “very low” according to the GRADE framework [32].”
In the citing paper
“Second, the review used a rigorous methodology following the PRISMA statement for reporting systematic reviews and meta-analyses [26] and GRADE framework [32].”
In the citing paper
“RCTs are first categorised as high-quality evidence in the GRADE system, while observational studies are classified as low-quality data supporting estimates of intervention effects.”
In the citing paper
“To determine the quality of evidence, the GRADE approach was used [29–36].”
In the citing paper
“Risk of bias was assessed using the Cochrane risk‐of‐bias tool for RCTs, and certainty of the evidence was assessed using the GRADE approach.”
Citations
Cites
Prevention of Venous Thromboembolism
Geerts, W. H., Pineo, G. F., Heit, J. A., Bergqvist, D., Lassen, M. R., Colwell, C. W., & Ray, J. G. (2004) · CHEST Journal · doi:10.1378/chest.126.3_suppl.338s · Cited by 5,889
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Prevention of Venous Thromboembolism
Geerts, W. H., Heit, J. A., Clagett, G. P., Pineo, G. F., Colwell, C. W., Anderson, F. A., & Wheeler, H. B. (2001) · CHEST Journal · doi:10.1378/chest.119.1_suppl.132s · Cited by 4,368
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308
Users' guides to the medical literature : a manual for evidence-based clinical practice
Guyatt, G. H., Rennie, D., Meade, M. O., & Cook, D. J. (2002) · Cited by 1,196
Self-citation
Cited by
Sorted by citations
Items 1,001–1,022 of 1,022
Evidence appraisal: a scoping review, conceptual framework, and research agenda
Goldstein, A., Venker, E., & Weng, C. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx050 · Cited by 30
Lin, J., Scott, W., Carpenter, L., Norton, S., Domhardt, M., Baumeister, H., & McCracken, L. M. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1044-2 · Cited by 30
School start times and academic achievement - A systematic review on grades and test scores
Biller, A. M., Meissner, K., Winnebeck, E. C., & Zerbini, G. (2022) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2021.101582 · Cited by 27
Tatalovic, M., Lehmann, R., Cheetham, M., Nowak, A., Battegay, E., & Rampini, S. K. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-028914 · Cited by 27
“We used the Cochrane risk of bias tool(30) to evaluate the risk of bias in RCTs and applied the key criteria of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidelines for observational studies to assess the methodological quality of nonrandomised studies.(31) The overall quality of ev…”
MethodGIUSTI, E. M., SPATOLA, C. A., BRUNANI, A., KUMBHARE, D., ORAL, A., ILIEVA, E., KIEKENS, C., PIETRABISSA, G., MANZONI, G. M., IMAMURA, M., et al. (2020) · European Journal of Physical and Rehabilitation Medicine · doi:10.23736/s1973-9087.20.06232-2 · Cited by 25
Kremer, F., Ditzen, B., & Wischmann, T. (2023) · PLoS ONE · doi:10.1371/journal.pone.0282065 · Cited by 22
“Hedges ´g = 1.337 Lower limit = 0.983 Upper limit = 1.820 1039 (9) ��⊝⊝ Very low event rates for fertility outcomes GRADE working group [36].”
BackgroundInfluentialRetractedTrubey, R., Evans, R., McDonald, S., Noyes, J., Robling, M., Willis, S., Boffey, M., Wooders, C., Vinnicombe, S., & Melendez-Torres, G. J. (2024) · Trauma, Violence, & Abuse · doi:10.1177/15248380241227987 · Cited by 22
“RCTs had a baseline rate of high certainty and non-randomized studies a low certainty rating.”
Martinez-Calderon, J., Flores-Cortes, M., Morales-Asencio, J. M., & Luque-Suarez, A. (2021) · Pain Medicine · doi:10.1093/pm/pnaa331 · Cited by 21
Solanes, A., Albajes-Eizagirre, A., Fullana, M. A., Fortea, L., Fusar-Poli, P., Torrent, C., Solé, B., Bonnín, C. M., Shin, J. I., Vieta, E., et al. (2021) · Revista de Psiquiatría y Salud Mental · doi:10.1016/j.rpsm.2020.08.002 · Cited by 19
Carroll, C., Scope, A., & Kaltenthaler, E. (2013) · BMC Research Notes · doi:10.1186/1756-0500-6-539 · Cited by 18
Shahinfar, H., Bazshahi, E., Amini, M. R., Payandeh, N., Pourreza, S., Noruzi, Z., & Shab‐Bidar, S. (2021) · Phytotherapy Research · doi:10.1002/ptr.7247 · Cited by 18
“…the Newcastle–Ottawa scale) (Guyatt et al., 2011), inconsistency (substantial unexplained betweenstudy heterogeneity, I2 ≥ 50% and pheterogeneity .10) (Guyatt et al., 2011c), indirectness (presence of population, intervention or comparator factors that limit the generalizability of the results)…”
BackgroundMethodInfluentialPark, S., Tang, A., Pollock, C., & Sakakibara, B. M. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-055527 · Cited by 12
Catalá-López, F., Hutton, B., Page, M. J., Ridao, M., Driver, J. A., Alonso-Arroyo, A., Forés-Martos, J., Macías Saint-Gerons, D., Vieta, E., Valencia, A., et al. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0581-9 · Cited by 11
Marí-Bauset, S., Peraita-Costa, I., Donat-Vargas, C., Llopis-González, A., Marí-Sanchis, A., Llopis-Morales, J., & Morales Suárez-Varela, M. (2022) · Autism · doi:10.1177/13623613211039950 · Cited by 10
(2021) · The Journal of Spinal Cord Medicine · doi:10.1080/10790268.2021.1863738 · Cited by 9
Association between Asthma and Caries-Related Salivary Factors: A Meta-Analysis
Hatipoğlu, Ö., & Pertek Hatipoğlu, F. (2022) · Journal of Asthma · doi:10.1080/02770903.2020.1826045 · Cited by 9
Almeida, B. A., Morales, A. P., Ribeiro, J. R. C., Sampaio-Jorge, F., Ribeiro, Y. G., Barth, T., & Ribeiro, B. G. (2024) · Current Cardiology Reviews · doi:10.2174/011573403x289842240307114736 · Cited by 9
Placebo, usual care and wait-list interventions for all mental health disorders
Faltinsen, E., Todorovac, A., Hróbjartsson, A., Gluud, C., Kongerslev, M. T., Simonsen, E., & Storebø, O. J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000050 · Cited by 9
Gordon, M., de Geus, A., Boruta, M., Banasiuk, M., Benninga, M., Borrelli, O., Darbari, A., Dore‐Stites, D., Gould, M., Hawa, J., et al. (2026) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1002/jpn3.70447 · Cited by 4
“14 Since the core outcome set was used (Table 1), indirectness of reported outcomes was not considered an issue and rated as “ not serious ” by default.”
Giusti, E. M., Spatola, C. A. M., Brunani, A., Kumbhare, D., Oral, A., Ilieva, E., Kiekens, C., Pietrabissa, G., Manzoni, G. M., Imamura, M., et al. (2020) · The Journal of the International Society of Physical and Rehabilitation Medicine · doi:10.4103/jisprm.jisprm_6_20 · Cited by 2
Chiaramonte, R., Ferrillo, M., de Sire, A., & Vecchio, M. (2026) · Journal of Manual & Manipulative Therapy · doi:10.1080/10669817.2026.2629282
Strategies for dealing with missing data in clinical trials: from design to analysis.
(2013) · PubMed Central · Cited by 288
Metadata
- Authors
- Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y.
- Affiliations
- McMaster University (CA)
- Nasjonalt Kunnskapssenter for Helsetjenesten (NO)
- University of Basel (CH)
- University at Buffalo, State University of New York (US)
- Case Western Reserve University (US)
- Corresponding author
- Guyatt, Gordon H.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2011-04
- Volume
- 64
- Issue
- 4
- Pages
- 407-415
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1981680557
- Cited by
- 2,798
- References
- 37
- Citations by year
- 2026: 130 · 2025: 182 · 2024: 238 · 2023: 199 · 2022: 264 · 2021: 280 · 2020: 284 · 2019: 198 · 2018: 184 · 2017: 180 · 2016: 176 · 2015: 179 · 2014: 120 · 2013: 95 · 2012: 65
- FWCI
- 100.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- GRADE, risk of bias, quality of evidence, observational studies, allocation concealment, blinding, loss to follow-up, intention-to-treat analysis, early stopping, selective reporting
- MeSH terms
- Female, Humans, Male, Quality Assurance, Health Care, Randomized Controlled Trials as Topic, Practice Guidelines as Topic, Publication Bias, Risk Assessment, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
