CoChat
Sign Up
Cited by 923Meta-analysis and systematic reviewsClinical practice guidelines implementationAdvanced Causal Inference Techniques

GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

Open access
Schünemann, Holger J., Cuello, Carlos, Akl, Elie A., Mustafa, Reem A., Meerpohl, Jörg J., Thayer, Kris, Morgan, Rebecca L., Gartlehner, Gerald +5 more·2019·Journal of Clinical Epidemiology
DOI 10.1016/j.jclinepi.2018.01.012PMC PMC6692166
Open this paper in CoChat — free account

Abstract

Objective To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in non-randomized studies of interventions (ROBINS-I) tool as part of GRADE’s certainty rating process. Study Design and Setting Iterative discussions, testing in systematic reviews, presentation at GRADE working group meetings with feedback from the GRADE Working Group. Results We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I, and where one would anticipate the final rating would end up. GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I. Conclusions The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from RCTs and NRS because they are placed on a common metric for risk of bias. Challenges remain that include appropriate presentation of evidence from RCTs and NRS for decision-making and how to optimally integrate RCTs and NRS in an evidence assessment.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2018.01.012

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 7 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    The authors have not identified bodies of evidence in which a ROBINS-I assessment alone leads to no rating down, or rating down by only one level.

    “We have not identified bodies of evidence in which a ROBINS-I assessment alone leads to no rating down, or rating down by only one level.”
    Quote in the paperSubstantial claim
  • Premisecitation

    A systematic review of NRS found that rofecoxib's summary RR for cardiovascular events was 1.33 (95% CI 1.00-1.79) at doses less than 25 mg/d and 2.19 (95% CI 1.64-2.91) at doses more than 25 mg/d.

    “In one of our prior articles we described that a systematic review of NRS investigating the effect of cyclooxygenase-2 inhibitors on cardiovascular events found that the summary estimate of RR with rofecoxib of 1.33 (95% CI: 1.00 to 1.79) with doses less than 25mg/d and 2.19 (95% CI: 1.64 to 2.91) with doses more than 25 mg/d.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    There is a possibility of misuse by evaluators assigning higher certainty to NRS evidence than appropriate, for example by rating risk of bias as moderate when current GRADE guidance would classify the evidence as low certainty.

    “Third, there is a possibility of misuse by those wanting to assign a higher certainty of the evidence to a body of evidence from NRS than is appropriate. Evaluators of evidence may rate the risk of bias from a group of NRS as moderate risk of bias if they are not appropriately cautious about the impact of confounding and selection bias or if reporting is poor.”
    Quote in the paperSubstantial claim
  • Interpretationnone given

    If a body of evidence from NRS is rated high certainty before considering size of effect and very large effects exist, no further rating up is possible or required.

    “With regards to large effects, if a body of evidence from NRS is indeed rated as high-certainty in the evidence prior to the consideration of size of effect and very large effects exist, no further rating up is possible or required.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    ROBINS-I users rating conventional NRS of any design will often arrive at a rating of high risk of bias after assessing confounding and selection bias.

    “This approach implies that ROBINS-I users rating conventional NRS of any design (e.g. cohort, case-control) following their assessment of confounding and selection bias, will often arrive at a rating of high risk of bias.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Randomization is the only way to fully protect against confounding, and confounding is always a concern in even the most rigorously conducted observational studies.
  • Classified as probably not a substantial claim. 15
  • May combine two claims. 12

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Until now there is no practical example on which to base a rating of initial high or even moderate certainty for a body of NRS evidence where no traditional GRADE upgrading domain applies.

    “Fifth, until now there is no practical example on which to base a rating of initial high or even moderate certainty in the evidence that comes from a body of evidence from NRS where no traditional GRADE upgrading domain applies.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    When raters use ROBINS-I with NRS beginning at high certainty, three levels for rating down for risk of bias are required so that NRS can arrive at very low certainty.

    “When raters use ROBINS-I with NRS beginning at high certainty, three levels for rating down for risk of bias are required so that NRS can arrive at a rating of very low certainty after considering risk of bias.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Given the inherent limitations of non-randomized studies, a body of evidence from NRS should generally not lead to moderate or high certainty in relation to risk of bias.

    “Given the inherent limitations of studies that do not use randomization, a body of evidence from NRS studies should generally not lead to moderate or high certainty in the evidence in relation to risk of bias.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Randomization is the only way to fully protect against confounding, and confounding is always a concern in even the most rigorously conducted observational studies.

  • May combine two claims.

    Interpretationreasoning

    A body of evidence from NRS studies can nevertheless receive a final rating of high or moderate certainty when using ROBINS-I, through rating up for large effect, dose-response, or direction of plausible confounding, or through NRS designs with greater protection against bias such as interrupted time series.

    “Using ROBINS-I it nevertheless remains possible that a body of evidence from NRS studies will receive a final rating of high or moderate certainty of evidence. This could result from rating up for large effect, dose-response, or the direction of plausible confounding. Or it could result from use of NRS designs and analyses with greater protection against risk of bias – for instance, interrupted”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    In current GRADE, NRS studies for prognosis or test accuracy questions begin with high certainty ratings; in particular, prognostic NRS begin as high certainty evidence.

    “In the current GRADE approach, NRS studies for these types of questions begin with high certainty ratings. In particular, with prognostic studies, in which the issue is association and not causation, prognostic NRS begin as high certainty evidence.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Using an approach such as ROBINS-I may harmonize GRADE approaches across different study types and questions such as prognosis or test accuracy.

    “Another potential advantage of using an approach such as ROBINS-I is that it may harmonize GRADE approaches across different study types for different types of questions such as prognosis or test accuracy.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Using ROBINS-I in GRADE assessments may allow better comparison of evidence from RCTs and NRS because they are placed on a common metric for risk of bias.

    “Second, the use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from RCTs and NRS because they are placed on a common metric for risk of bias.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    GRADE has accounted for issues that mitigate concerns about confounding and selection bias by introducing upgrading domains (large effects, dose-effect relations, and plausible residual confounders or biases increasing certainty), which need to be considered when using ROBINS-I.

    “GRADE has thus accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    Currently GRADE has only three labels for risk of bias (not serious, serious, very serious); for RCTs these correspond to high, moderate, or low certainty after risk of bias, and for NRS serious risk of bias rates down from low to very low.

    “Currently GRADE has only three labels for risk of bias: not serious, serious or very serious on the risk of bias domain levels. For RCTs, this corresponds to a rating of the body of evidence as high, moderate or low certainty of evidence after considering risk of bias; for NRS this means that when GRADE currently uses serious risk of bias for NRS they are rated down from low initial certainty to”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    For studies of interventions assessed with ROBINS-I in GRADE, an initial rating of high is suggested, with rating down for risk of bias according to ROBINS-I; in practice this will generally lead to rating down by at least two levels to low or very low certainty for NRS.

    “For studies of interventions that are assessed with ROBINS-I in the context of GRADE, we suggest that an initial rating of high is used, with appropriate consideration of the impact of lack of randomization leading to rating down for risk of bias according to the ROBINS-I tool. In practice this will generally lead to rating down by at least two levels to low or very low certainty for NRS.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Randomization is the only way to fully protect against confounding, and confounding is always a concern in even the most rigorously conducted observational studies.

  • Classified as probably not a substantial claim.

    Premisereasoning

    ROBINS-I evaluates risk of bias in estimates of intervention effects from studies that did not use randomization to allocate units to comparison groups.

    “ROBINS-I evaluates risk of bias in estimates of the effects (harm or benefit) of one or more interventions from studies that did not use randomization to allocate units (individuals or clusters of individuals) to comparison groups (in GRADE terminology observational studies).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Use of ROBINS-I may facilitate combining results of RCTs and NRS because it assesses risk of bias on an absolute scale, but conditions for combining remain uncertain.

    “Second, because of the advantage of assessing risk of bias on an absolute scale, use of ROBINS-I may facilitate combining results of RCTs and NRS. However, under what conditions one should combine results from randomized and non-randomized studies remains uncertain (this uncertainty also applies whether one uses ROBINS-I or other instruments for assessing risk of bias in NRS) (Figure 4).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Existing GRADE guidance for interventions begins a body of evidence from randomized trials at high certainty and from observational studies at low certainty, then considers eight domains, five for rating down and three for rating up.

    “If the relevant studies are randomized trials, the body of evidence begins as high certainty. If the relevant studies are observational, the body of evidence begins as low certainty. This initial rating is followed by consideration of eight domains, five of which may result in rating down certainty, and three in rating up.(8)”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [8] Guyatt, 2013, GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes, J Clin Epidemiol source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    In the for-profit hospital example, all residual plausible confounding would have decreased the observed effect, so despite biases favoring for-profit hospitals, the true effect is almost certain to be greater than the estimate.

    “The evaluation of risk of bias across studies revealed that all residual plausible confounding – the major issues being that for-profit hospitals have on average higher income patients and greater resources - would have decreased the observed effect (further towards a RR of 1.0). Despite the biases in favour of for-profit hospitals, those hospitals demonstrated higher mortality – therefore the”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premiseown data

    ROBINS-I signaling questions ask raters to assess seven domains: confounding, selection of participants, classification of interventions, departures from intended interventions, missing data, measurement of outcomes, and selection of reported results.

    “Signaling questions in the ROBINS-I instrument ask respondents to rate RoB in domains of 1) Bias due to confounding, 2) Bias in selection of participants into the study, 3) Bias in classification of interventions, 4) Bias due to departures from intended interventions, 5) Bias due to missing data, 6) Bias in measurement of outcomes, and 7) Bias in selection of reported results (Figure 2).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Whether one starts NRS evidence at low certainty and rates up or starts at high certainty and rates down, the final certainty rating should be the same.

    “Thus, we would anticipate that whether one begins with a body of evidence from NRS studies as low certainty and looks for reasons to rate up or down, or starts with that evidence as high quality and looks for reasons to rate down, the final certainty rating should be the same.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    ROBINS-I is currently the only available tool that explicitly includes a comparison against RCTs, so this guidance only applies when ROBINS-I is used.

    “First, ROBINS-I is currently the only available tool that explicitly includes a comparison against RCTs (and this situation is unlikely to change) and thus this guidance only applies to the situations when ROBINS-I is used.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    Risk of bias is best mitigated by a well-conducted RCT that balances known and unknown confounders, and by using Cochrane RoB 2.0 or similar tools for RCTs.

    “Risk of bias can be best mitigated by a well conducted RCT that balances known and unknown confounders, and using the Cochrane RoB 2.0 tool or similar assessment tools for RCTs to assess risk of bias.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: A high-quality randomized trial avoids confounding and other biases such as selection or information biases.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    For results with large effects, dose response, or results in which inference is strengthened by plausible biases, the extent of rating down may be lowered.

    “However, for results with large effects, or dose response, or results in which inference is strengthened by the plausible biases that exist, the extent of rating down may be lowered.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    ROBINS-I's underlying principles include comparing a study's risk of bias against a target RCT even if that RCT is not feasible or ethical, treating assessment of confounding and selection bias as integral, and giving an overall rating per outcome for each study after addressing multiple domains.

    “ROBINS-I’s fundamental underlying principles are that (1) the study’s risk of bias is compared against a target RCT, even if this RCT may not be feasible or ethical; (2) the assessment of confounding and selection bias are integral parts of the tool; and (3) for a given result for a specific outcome, evidence from an NRS is assessed, addressing a number of domains and then giving an overall rating”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    ROBINS-I allows review authors to assess how failure to use randomization in individual studies has impacted risk of bias, including categorizing magnitude of bias through selection and confounding domains and evaluating differences across studies.

    “Among other features, ROBINS-I allows review authors to assess how failure to use randomization in individual studies has impacted on risk of bias. For example, ROBINS-I allows a categorization of the magnitude of bias from lack of randomization through the selection and confounding bias domains, allows application of this assessment across risk of bias domains, and evaluation of how this differs”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Available evidence suggests publication bias is a greater problem in NRS than in RCTs.

    “available evidence suggests publication bias is a greater problem in NRS than in RCTs”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    An alternative approach assesses risk of bias in observational studies by comparing them with the effect that would be seen in a high-quality randomized trial, which avoids confounding and other biases.

    “An alternative approach is to determine risk of bias of observational studies in relation to the effect that would be seen in a high quality randomized trial. Such a trial avoids both confounding (through random allocation to interventions) as well as other sources of bias such as selection or information biases.”
    Quote in the paperNot a substantial claim
23 of 23

  • The paper relies on this without support.

    Randomization is the only way to fully protect against confounding, and confounding is always a concern in even the most rigorously conducted observational studies.

    “The separation of randomized and observational studies was primarily a result of recognition that randomization is the only way to fully protect against confounding (i.e. imbalance in prognostic factors between intervention and control groups), and that confounding is always a concern in even the most rigorously conducted observational studies.”

  • The paper relies on this without support.

    The potential for residual confounding in observational studies is large, and adjusted analysis provides limited protection against bias, which justified starting observational studies at low certainty.

    “The choice of starting observational studies at low rather than moderate or very low certainty followed intense discussion in the GRADE working group’s early days, and was based on the group’s assessment of the magnitude of the potential for residual confounding, and the limited protection against bias provided by adjusted analysis in observational studies.”

  • The paper relies on this without support.

    A high-quality randomized trial avoids confounding and other biases such as selection or information biases.

    “Such a trial avoids both confounding (through random allocation to interventions) as well as other sources of bias such as selection or information biases.”

  • The paper relies on this without support.

    ROBINS-I has been carefully developed by experienced investigators, tested, and begun to be validated.

    “ROBINS-I has undergone careful development by a large group of experienced investigators. It has been tested and scientists have begun to validate it, and experience will continue to accumulate.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A modified Cochrane Collaboration's Risk of Bias in Non-Randomized Studies of Interventions was used to evaluate the risk of bias in selected studies based on 6 domains.

    “A modified Cochrane Collaboration's Risk of Bias in Non-Randomized Studies of Interventions was used to evaluate the risk of bias in selected studies as low, moderate, serious, and critical based on 6 domains (Table 3) (29,30).”

    From Alterations in Heart Rate Variability Associated With Irritable Bowel Syndrome or Inflammatory Bowel Disease: A Systematic Review and Meta-Analysis (2021) · cites this paper as [30]

    The section “Rating risk of bias in individual observational studies” identifies ROBINS-I as a tool for assessing risk of bias in nonrandomized studies. It describes seven domains, however, and does not document the citing authors’ six-domain modification or its use in their selected studies. No page number is available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Risk of bias was assessed using the ROBINS-I tool.

    “the quantification of risk of bias (RoB) was determined via ROBINS-I tool [14].”

    From Periodontal Health in Patients with Self-Ligating Brackets: A Systematic Review of Clinical Studies (2022) · cites this paper as [14]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Schünemann, H. J., Cuello, C., Akl, E. A., Mustafa, R. A., Meerpohl, J. J., Thayer, K., Morgan, R. L., Gartlehner, G., Kunz, R., Katikireddi, S. V., Sterne, J., Higgins, J. P., & Guyatt, G.
Affiliations
  • McMaster University (CA)
  • University of Kansas Medical Center (US)
  • Universität für Weiterbildung Krems (AT)
  • University Hospital of Basel (CH)
  • MRC/CSO Social and Public Health Sciences Unit (GB)
  • University of Bristol (GB)
Corresponding author
Schünemann, Holger J.
ORCID
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2019-07
Volume
111
Pages
105-114
ISSN
0895-4356 (ISSN-L)
DOI
10.1016/j.jclinepi.2018.01.012
PMC
PMC6692166
OpenAlex
W2792308249
Semantic Scholar
cf0d116e848fca50508ac07e082d05e29ba05fcd
Cited by
911
References
27
Citations by year
2026: 133 · 2025: 146 · 2024: 125 · 2023: 118 · 2022: 114 · 2021: 103 · 2020: 121 · 2019: 34 · 2018: 16 · 2017: 1
FWCI
52.17
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Clinical Epidemiology (publishedVersion)
  • PubMed (publishedVersion)
  • Bristol Research (University of Bristol) (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
  • ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion, cc-by-sa)
  • edoc (University of Basel) (submittedVersion)
  • Explore Bristol Research (submittedVersion, cc-by-nc-nd)
Funders
  • National Institute for Health and Care Research (NF-SI-0611-10168, NF-SI-0616-10111, NF-SI-0617-10145)
  • Medical Research Council (MR/M025209/1, MC_UU_12017/13, MC_UU_12017/15)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords
ROBINS-I, risk of bias, non-randomized studies, GRADE, quality of evidence, residual confounding, selection bias, confounding, evidence evaluation
MeSH terms
Systematic Reviews as Topic, Clinical Trials as Topic, Humans, Risk Factors, Bias, Practice Guidelines as Topic, Risk Assessment, Evidence-Based Medicine, Uncertainty, Observational Studies as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-04.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-04.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review