The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence
Open accessAbstract
Objective To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in non-randomized studies of interventions (ROBINS-I) tool as part of GRADE’s certainty rating process. Study Design and Setting Iterative discussions, testing in systematic reviews, presentation at GRADE working group meetings with feedback from the GRADE Working Group. Results We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I, and where one would anticipate the final rating would end up. GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I. Conclusions The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from RCTs and NRS because they are placed on a common metric for risk of bias. Challenges remain that include appropriate presentation of evidence from RCTs and NRS for decision-making and how to optimally integrate RCTs and NRS in an evidence assessment.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
18 Own interpretation · 13 No basis given · 6 Not checked yet · 1 Own result
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We have not identified bodies of evidence in which a ROBINS-I assessment alone leads to no rating down, or rating down by only one level.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Summary and next steps
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In one of our prior articles we described that a systematic review of NRS investigating the effect of cyclooxygenase-2 inhibitors on cardiovascular events found that the summary estimate of RR with rofecoxib of 1.33 (95% CI: 1.00 to 1.79) with doses less than 25mg/d and 2.19 (95% CI: 1.64 to 2.91) with doses more than 25 mg/d.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > What makes us confident in results of NRS and does GRADE already account for this? > Large effects and dose responses
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, there is a possibility of misuse by those wanting to assign a higher certainty of the evidence to a body of evidence from NRS than is appropriate. Evaluators of evidence may rate the risk of bias from a group of NRS as moderate risk of bias if they are not appropriately cautious about the impact of confounding and selection bias or if reporting is poor.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Advantages and disadvantages of, in the context of GRADE, assessing risk of bias for individual studies using the ROBINS-I approach of specifying target trials > Disadvantages
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With regards to large effects, if a body of evidence from NRS is indeed rated as high-certainty in the evidence prior to the consideration of size of effect and very large effects exist, no further rating up is possible or required.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Unresolved Issues
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach implies that ROBINS-I users rating conventional NRS of any design (e.g. cohort, case-control) following their assessment of confounding and selection bias, will often arrive at a rating of high risk of bias.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Certainty of evidence for a body of evidence from NRS when using ROBINS-I for assessing risk of bias in individual NRS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Rests on an assumption: Randomization is the only way to fully protect against confounding, and confounding is always a concern in even the most rigorously conducted observational studies.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, when using ROBINS-I for assessing risk of bias in NRS, given that assessment of selection bias and confounding is an integral part of the ROBINS-I tool, the initial GRADE certainty in the evidence from a body of studies using an NRS design would be high (Figures 4 and 5).”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Certainty of evidence for a body of evidence from NRS when using ROBINS-I for assessing risk of bias in individual NRS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“GRADE suggests that large effects and dose-response relations mitigate concerns regarding residual confounding.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > What makes us confident in results of NRS and does GRADE already account for this? > Large effects and dose responses
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“GRADE allows higher certainty ratings for bodies of evidence when all plausible residual confounders or other biases increase our confidence in the range of an estimated effect, that is the effect is either larger or smaller than that observed (23, 24).”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > What makes us confident in results of NRS and does GRADE already account for this? > All plausible residual confounders or other biases increase our certainty in the estimated effect
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“One example from the public health field comes from a systematic review of NRS including a total of 38 million patients that demonstrated a very small relative increase (relative risk 1.020, 95% confidence interval 1.003-1.038) in death rates in private for-profit compared with private not-for-profit hospitals (23, 25).”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > What makes us confident in results of NRS and does GRADE already account for this? > All plausible residual confounders or other biases increase our certainty in the estimated effect
What the sources say
- Not checked yetcites [23] Guyatt, 2011, GRADE guidelines: 9. Rating up the quality of evidence, J Clin Epidemiol source
- Not checked yetcites [25] Devereaux, 2002, A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals, CMAJ
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, ROBINS-I will facilitate assessment of a study that has been described as randomized but when assessed in detail is found to be not appropriately randomized.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Advantages and disadvantages of, in the context of GRADE, assessing risk of bias for individual studies using the ROBINS-I approach of specifying target trials > Advantages
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 15
- It may state two separate claims as one. 12
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fifth, until now there is no practical example on which to base a rating of initial high or even moderate certainty in the evidence that comes from a body of evidence from NRS where no traditional GRADE upgrading domain applies.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Advantages and disadvantages of, in the context of GRADE, assessing risk of bias for individual studies using the ROBINS-I approach of specifying target trials > Disadvantages
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When raters use ROBINS-I with NRS beginning at high certainty, three levels for rating down for risk of bias are required so that NRS can arrive at a rating of very low certainty after considering risk of bias.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Unresolved Issues
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given the inherent limitations of studies that do not use randomization, a body of evidence from NRS studies should generally not lead to moderate or high certainty in the evidence in relation to risk of bias.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Summary and next steps
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Randomization is the only way to fully protect against confounding, and confounding is always a concern in even the most rigorously conducted observational studies.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using ROBINS-I it nevertheless remains possible that a body of evidence from NRS studies will receive a final rating of high or moderate certainty of evidence. This could result from rating up for large effect, dose-response, or the direction of plausible confounding. Or it could result from use of NRS designs and analyses with greater protection against risk of bias – for instance, interrupted”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Certainty of evidence for a body of evidence from NRS when using ROBINS-I for assessing risk of bias in individual NRS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the current GRADE approach, NRS studies for these types of questions begin with high certainty ratings. In particular, with prognostic studies, in which the issue is association and not causation, prognostic NRS begin as high certainty evidence.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Advantages and disadvantages of, in the context of GRADE, assessing risk of bias for individual studies using the ROBINS-I approach of specifying target trials > Advantages
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another potential advantage of using an approach such as ROBINS-I is that it may harmonize GRADE approaches across different study types for different types of questions such as prognosis or test accuracy.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Advantages and disadvantages of, in the context of GRADE, assessing risk of bias for individual studies using the ROBINS-I approach of specifying target trials > Advantages
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, the use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from RCTs and NRS because they are placed on a common metric for risk of bias.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > ROBINS-I and GRADE
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“GRADE has thus accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > What makes us confident in results of NRS and does GRADE already account for this? > Large effects and dose responses
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Currently GRADE has only three labels for risk of bias: not serious, serious or very serious on the risk of bias domain levels. For RCTs, this corresponds to a rating of the body of evidence as high, moderate or low certainty of evidence after considering risk of bias; for NRS this means that when GRADE currently uses serious risk of bias for NRS they are rated down from low initial certainty to”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Unresolved Issues
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For studies of interventions that are assessed with ROBINS-I in the context of GRADE, we suggest that an initial rating of high is used, with appropriate consideration of the impact of lack of randomization leading to rating down for risk of bias according to the ROBINS-I tool. In practice this will generally lead to rating down by at least two levels to low or very low certainty for NRS.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Summary and next steps
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Randomization is the only way to fully protect against confounding, and confounding is always a concern in even the most rigorously conducted observational studies.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ROBINS-I evaluates risk of bias in estimates of the effects (harm or benefit) of one or more interventions from studies that did not use randomization to allocate units (individuals or clusters of individuals) to comparison groups (in GRADE terminology observational studies).”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Rating risk of bias in individual observational studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, because of the advantage of assessing risk of bias on an absolute scale, use of ROBINS-I may facilitate combining results of RCTs and NRS. However, under what conditions one should combine results from randomized and non-randomized studies remains uncertain (this uncertainty also applies whether one uses ROBINS-I or other instruments for assessing risk of bias in NRS) (Figure 4).”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Advantages and disadvantages of, in the context of GRADE, assessing risk of bias for individual studies using the ROBINS-I approach of specifying target trials > Disadvantages
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“If the relevant studies are randomized trials, the body of evidence begins as high certainty. If the relevant studies are observational, the body of evidence begins as low certainty. This initial rating is followed by consideration of eight domains, five of which may result in rating down certainty, and three in rating up.(8)”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > GRADE’s approach to the certainty of the evidence from observational studies
What the sources say
- Not checked yetcites [8] Guyatt, 2013, GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes, J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evaluation of risk of bias across studies revealed that all residual plausible confounding – the major issues being that for-profit hospitals have on average higher income patients and greater resources - would have decreased the observed effect (further towards a RR of 1.0). Despite the biases in favour of for-profit hospitals, those hospitals demonstrated higher mortality – therefore the”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > What makes us confident in results of NRS and does GRADE already account for this? > All plausible residual confounders or other biases increase our certainty in the estimated effect
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Signaling questions in the ROBINS-I instrument ask respondents to rate RoB in domains of 1) Bias due to confounding, 2) Bias in selection of participants into the study, 3) Bias in classification of interventions, 4) Bias due to departures from intended interventions, 5) Bias due to missing data, 6) Bias in measurement of outcomes, and 7) Bias in selection of reported results (Figure 2).”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Rating risk of bias in individual observational studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, we would anticipate that whether one begins with a body of evidence from NRS studies as low certainty and looks for reasons to rate up or down, or starts with that evidence as high quality and looks for reasons to rate down, the final certainty rating should be the same.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Certainty of evidence for a body of evidence from NRS when using ROBINS-I for assessing risk of bias in individual NRS
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, ROBINS-I is currently the only available tool that explicitly includes a comparison against RCTs (and this situation is unlikely to change) and thus this guidance only applies to the situations when ROBINS-I is used.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Advantages and disadvantages of, in the context of GRADE, assessing risk of bias for individual studies using the ROBINS-I approach of specifying target trials > Disadvantages
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Risk of bias can be best mitigated by a well conducted RCT that balances known and unknown confounders, and using the Cochrane RoB 2.0 tool or similar assessment tools for RCTs to assess risk of bias.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Summary and next steps
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: A high-quality randomized trial avoids confounding and other biases such as selection or information biases.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, for results with large effects, or dose response, or results in which inference is strengthened by the plausible biases that exist, the extent of rating down may be lowered.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Summary and next steps
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ROBINS-I’s fundamental underlying principles are that (1) the study’s risk of bias is compared against a target RCT, even if this RCT may not be feasible or ethical; (2) the assessment of confounding and selection bias are integral parts of the tool; and (3) for a given result for a specific outcome, evidence from an NRS is assessed, addressing a number of domains and then giving an overall rating”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Rating risk of bias in individual observational studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Among other features, ROBINS-I allows review authors to assess how failure to use randomization in individual studies has impacted on risk of bias. For example, ROBINS-I allows a categorization of the magnitude of bias from lack of randomization through the selection and confounding bias domains, allows application of this assessment across risk of bias domains, and evaluation of how this differs”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Advantages and disadvantages of, in the context of GRADE, assessing risk of bias for individual studies using the ROBINS-I approach of specifying target trials > Advantages
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“available evidence suggests publication bias is a greater problem in NRS than in RCTs”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Unresolved Issues
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“An alternative approach is to determine risk of bias of observational studies in relation to the effect that would be seen in a high quality randomized trial. Such a trial avoids both confounding (through random allocation to interventions) as well as other sources of bias such as selection or information biases.”
§ GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence > Rating risk of bias in individual observational studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Randomization is the only way to fully protect against confounding, and confounding is always a concern in even the most rigorously conducted observational studies.
“The separation of randomized and observational studies was primarily a result of recognition that randomization is the only way to fully protect against confounding (i.e. imbalance in prognostic factors between intervention and control groups), and that confounding is always a concern in even the most rigorously conducted observational studies.”
The paper relies on this without support.
The potential for residual confounding in observational studies is large, and adjusted analysis provides limited protection against bias, which justified starting observational studies at low certainty.
“The choice of starting observational studies at low rather than moderate or very low certainty followed intense discussion in the GRADE working group’s early days, and was based on the group’s assessment of the magnitude of the potential for residual confounding, and the limited protection against bias provided by adjusted analysis in observational studies.”
The paper relies on this without support.
A high-quality randomized trial avoids confounding and other biases such as selection or information biases.
“Such a trial avoids both confounding (through random allocation to interventions) as well as other sources of bias such as selection or information biases.”
The paper relies on this without support.
ROBINS-I has been carefully developed by experienced investigators, tested, and begun to be validated.
“ROBINS-I has undergone careful development by a large group of experienced investigators. It has been tested and scientists have begun to validate it, and experience will continue to accumulate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A modified Cochrane Collaboration's Risk of Bias in Non-Randomized Studies of Interventions was used to evaluate the risk of bias in selected studies as low, moderate, serious, and critical based on 6 domains (Table 3) (29,30).”
From Alterations in Heart Rate Variability Associated With Irritable Bowel Syndrome or Inflammatory Bowel Disease: A Systematic Review and Meta-Analysis (2021) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“the quantification of risk of bias (RoB) was determined via ROBINS-I tool [14].”
From Periodontal Health in Patients with Self-Ligating Brackets: A Systematic Review of Clinical Studies (2022) · cites this paper as [14]
Citations
Cites
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,586
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationGRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681
Self-citationGrading quality of evidence and strength of recommendations
(2004) · BMJ · doi:10.1136/bmj.328.7454.1490 · Cited by 8,477
Self-citationNorderyd, O. M., Koch, G., Papias, A., Köhler, A. A., Helkimo, A. N., Brahm, C., Lindmark, U., Lindfors, N., Mattsson, A., Rolander, B., et al. (2015) · PubMed · Cited by 3,584
GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology
Guyatt, G. H., Oxman, A. D., Schünemann, H. J., Tugwell, P., & Knottnerus, A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.011 · Cited by 3,094
Self-citationPuhan, M. A., Schunemann, H. J., Murad, M. H., Li, T., Brignardello-Petersen, R., Singh, J. A., Kessels, A. G., Guyatt, G. H., & for the GRADE Working Group (2014) · BMJ · doi:10.1136/bmj.g5630 · Cited by 1,812
Self-citationGrading quality of evidence and strength of recommendations for diagnostic tests and strategies
Schünemann, H. J., Oxman, A. D., Brozek, J., Glasziou, P., Jaeschke, R., Vist, G. E., Williams, J. W., Kunz, R., Craig, J., Montori, V. M., et al. (2008) · BMJ · doi:10.1136/bmj.39500.677199.ae · Cited by 1,782
Self-citationSanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531
Self-citationGRADE guidelines: 9. Rating up the quality of evidence
Guyatt, G. H., Oxman, A. D., Sultan, S., Glasziou, P., Akl, E. A., Alonso-Coello, P., Atkins, D., Kunz, R., Brozek, J., Montori, V., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.06.004 · Cited by 1,336
Self-citation
Cited by
Sorted by citations
A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E)
Higgins, J. P. T., Morgan, R. L., Rooney, A. A., Taylor, K. W., Thayer, K. A., Silva, R. A., Lemeris, C., Akl, E. A., Bateson, T. F., Berkman, N. D., et al. (2024) · Environment International · doi:10.1016/j.envint.2024.108602 · Cited by 1,022
“An example of such a framework is the GRADE approach (Guyatt et al., 2011) where users may start at high certainty in the “study limitations” (risk of bias) domain of GRADE and use their ROBINS-E evaluation to determine how many levels to rate down this certainty (Schunemann et al., 2018).”
Self-citationNußbaumer-Streit, B., Mayr, V., Dobrescu, A. I., Chapman, A. L., Persad, E., Klerings, I., Wagner, G., Siebert, U., Christof, C., Zachariah, C., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013574 · Cited by 987
Self-citationCompleting ‘Summary of findings’ tables and grading the certainty of the evidence
Schünemann, H. J., Higgins, J. P., Vist, G. E., Glasziou, P., Akl, E. A., Skoetz, N., Guyatt, G. H., & on behalf of the Cochrane GRADEing Methods Group (formerly Applicability and Recommendations Methods Group) and the Cochrane Statistical Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch14 · Cited by 669
Self-citationNußbaumer-Streit, B., Mayr, V., Dobrescu, A. I., Chapman, A. L., Persad, E., Klerings, I., Wagner, G., Siebert, U., Ledinger, D., Zachariah, C., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013574.pub2 · Cited by 548
Self-citationUse of the GRADE approach in systematic reviews and guidelines
Granholm, A., Alhazzani, W. A., & Møller, M. H. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2019.08.015 · Cited by 470
Sultan, S., Altayar, O., Siddique, S. M., Davitkov, P., Feuerstein, J. D., Lim, J. K., Falck-Ytter, Y., & El-Serag, H. B. (2020) · Gastroenterology · doi:10.1053/j.gastro.2020.05.001 · Cited by 437
Introduction to the GRADE tool for rating certainty in evidence and recommendations
Prasad, M. (2024) · Clinical Epidemiology and Global Health · doi:10.1016/j.cegh.2023.101484 · Cited by 404
Piechotta, V., Chai, K. L., Valk, S. J., Doree, C. J., Monsef, I., Wood, E. M., Lamikanra, A., Kimber, C., McQuilten, Z., So‐Osman, C., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013600.pub2 · Cited by 372
von Philipsborn, P., Stratil, J. M., Burns, J., Busert, L. K., Pfadenhauer, L. M., Polus, S., Holzapfel, C., Hauner, H., & Rehfuess, E. A. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012292.pub2 · Cited by 320
Zhang, Y., Alonso‐Coello, P., Guyatt, G. H., Yepes-Núñez, J. J., Akl, E. A., Hazlewood, G. S., Hernandez, H. P., Etxeandia‐Ikobaltzeta, I., Qaseem, A., Williams, J. W., et al. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.011 · Cited by 294
Self-citationGrimani, A., Aboagye, E., & Kwak, L. (2019) · BMC Public Health · doi:10.1186/s12889-019-8033-1 · Cited by 269
Using systematic reviews in guideline development: The GRADE approach
Zhang, Y., Akl, E. A., & Schünemann, H. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1313 · Cited by 260
Self-citationJayedi, A., Soltani, S., Motlagh, S. Z.-T., Emadi, A., Shahinfar, H., Moosavi, H., & Shab-Bidar, S. (2022) · BMJ · doi:10.1136/bmj-2021-067516 · Cited by 253
Morgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Ansari, M., Akl, E. A., et al. (2019) · Environment International · doi:10.1016/j.envint.2018.11.004 · Cited by 248
Self-citationPapageorgiou, S. N., Koletsi, D., Iliadi, A., Peltomäki, T., & Eliades, T. (2019) · European Journal of Orthodontics · doi:10.1093/ejo/cjz094 · Cited by 243
Guidelines for the Neurocritical Care Management of Aneurysmal Subarachnoid Hemorrhage
Treggiari, M. M., Rabinstein, A. A., Busl, K. M., Caylor, M. M., Citerio, G., Deem, S. A., Diringer, M. N., Fox, E., Livesay, S. L., Sheth, K. N., et al. (2023) · Neurocritical Care · doi:10.1007/s12028-023-01713-5 · Cited by 225
Igelström, E., Campbell, M., Craig, P., & Katikireddi, S. V. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.08.022 · Cited by 217
Self-citationMahamat‐Saleh, Y., Fiolet, T., Rebeaud, M. E., Mulot, M., Guihur, A., Fatouhi, D. E., Laouali, N., Peiffer‐Smadja, N., Aune, D., & Severi, G. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-052777 · Cited by 205
Stuart, A. J., Romano, V., Virgili, G., & Shortt, A. J. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012097.pub2 · Cited by 196
Safety and effectiveness of acellular pertussis vaccination during pregnancy: a systematic review
Vygen-Bonnet, S., Hellenbrand, W., Garbe, E., von Kries, R., Bogdan, C., Heininger, U., Röbl-Mathieu, M., & Harder, T. (2020) · BMC Infectious Diseases · doi:10.1186/s12879-020-4824-3 · Cited by 189
Piechotta, V., Iannizzi, C., Chai, K. L., Valk, S. J., Kimber, C., Dorando, E., Monsef, I., Wood, E. M., Lamikanra, A., Roberts, D. J., et al. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013600.pub4 · Cited by 190
Cuello‐García, C. A., Santesso, N. A., Morgan, R. L., Verbeek, J. H. A. M., Thayer, K. A., Ansari, M., Meerpohl, J. J., Schwingshackl, L., Katikireddi, S. V., Brożek, J., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.11.026 · Cited by 178
Self-citationIncluding non‐randomized studies on intervention effects
Reeves, B. C., Deeks, J. J., Higgins, J. P. T., Shea, B., Tugwell, P., Wells, G. A., & on behalf of the Cochrane Non‐Randomized Studies of Interventions Methods Group (2019) · Wiley eBooks · doi:10.1002/9781119536604.ch24 · Cited by 167
Self-citationConceptualising natural and quasi experiments in public health
de Vocht, F., Katikireddi, S. V., McQuire, C., Tilling, K. M., Hickman, M., & Craig, P. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01224-x · Cited by 168
Self-citationChai, K. L., Valk, S. J., Piechotta, V., Kimber, C., Monsef, I., Doree, C. J., Wood, E. M., Lamikanra, A., Roberts, D. J., McQuilten, Z., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013600.pub3 · Cited by 150
Stroke risk, phenotypes, and death in COVID-19
Fridman, S., Bullrich, M. B., Jiménez‐Ruiz, A., Costantini, P., Shah, P. A., Just, C., Vela‐Duarte, D., Linfante, I., Razavi, A. S., Karimi, N., et al. (2020) · Neurology · doi:10.1212/wnl.0000000000010851 · Cited by 146
Ludwig‐Walz, H., Dannheim, I., Pfadenhauer, L. M., Fegert, J. M., & Bujard, M. (2022) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-022-00546-y · Cited by 151
Kiely, B. T., Croke, A., O'Shea, M., Boland, F., O’Shea, E., Connolly, D., & Smith, S. M. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-062951 · Cited by 138
Balalle, H., & Pannilage, S. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.101299 · Cited by 137
Neuenschwander, M., Stadelmaier, J., Eble, J., Grummich, K., Szczerba, E., Kiesswetter, E., Schlesinger, S., & Schwingshackl, L. (2023) · BMC Medicine · doi:10.1186/s12916-023-03093-1 · Cited by 136
Neuenschwander, M., Barbaresko, J., Pischke, C. R., Iser, N., Beckhaus, J., Schwingshackl, L., & Schlesinger, S. (2020) · PLoS Medicine · doi:10.1371/journal.pmed.1003347 · Cited by 130
Naenni, N., Lim, H., Papageorgiou, S. N., & Hämmerle, C. H. F. (2019) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13052 · Cited by 128
International travel-related control measures to contain the COVID-19 pandemic: a rapid review
Burns, J., Movsisyan, A., Stratil, J. M., Biallas, R., Coenen, M., Emmert-Fees, K. M. F., Geffert, K., Hoffmann, S., Horstick, O., Laxy, M., et al. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013717.pub2 · Cited by 127
Chi, C., Shao, S., Kuo, L., Huang, Y., & Lai, P.-C. (2023) · Dermatologica Sinica · doi:10.4103/ds.ds-d-22-00154 · Cited by 121
Therriault, J., Brum, W. S., Trudel, L., Macedo, A. C., Bitencourt, F. V., Martins, C. C., Nakouzi, M., Pola, I., Wong, M. W. K., Kac, P. R., et al. (2025) · The Lancet Neurology · doi:10.1016/s1474-4422(25)00227-3 · Cited by 118
Piggott, T., Morgan, R. L., Cuello‐García, C. A., Santesso, N. A., Mustafa, R. A., Meerpohl, J. J., & Schünemann, H. J. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.11.019 · Cited by 110
Self-citationLam, P. P. Y., Du, R. Y., Peng, S., McGrath, C. P. J., & Yiu, C. K. Y. (2020) · Autism · doi:10.1177/1362361319877337 · Cited by 115
Travel-related control measures to contain the COVID-19 pandemic: a rapid review
Burns, J., Movsisyan, A., Stratil, J. M., Coenen, M., Emmert-Fees, K. M. F., Geffert, K., Hoffmann, S., Horstick, O., Laxy, M., Pfadenhauer, L. M., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013717 · Cited by 110
Morgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Guyatt, G., & Schünemann, H. J. (2018) · Environment International · doi:10.1016/j.envint.2018.08.018 · Cited by 106
Self-citationMartín, C., Celis, B., Ambrosio, N., Bollaín, J., Antonoglou, G. N., & Figuero, E. (2021) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13487 · Cited by 102
Schlesinger, S., Neuenschwander, M., Lang, A., Pafili, K., Kuß, O., Herder, C., & Roden, M. (2021) · Diabetologia · doi:10.1007/s00125-021-05458-8 · Cited by 100
Rosado, M. E. B., Suárez, L. M. C., de Zevallos, G. F. D. O., & Espinosa‐Cristia, J. F. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1173425 · Cited by 115
Asiimwe, J. B., Nagendrappa, P. B., Atukunda, E. C., Kamatenesi, M. M., Nambozi, G., Tolo, C. U., Ogwang, P. E., & Sarki, A. M. (2021) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2021/9963038 · Cited by 99
Community-level interventions for improving access to food in low- and middle-income countries
Durão, S., Visser, M. E., Ramokolo, V., de Oliveira, J. M., Schmidt, B., Balakrishna, Y., Brand, A., Kristjansson, E., & Schoonees, A. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011504.pub3 · Cited by 101
García‐Elorrio, E., Rowe, S., Teijeiro, M. E., Ciapponi, A., & Rowe, A. K. (2019) · PLoS ONE · doi:10.1371/journal.pone.0221919 · Cited by 99
Piechotta, V., Siemens, W., Thielemann, I., Toews, M., Koch, J., Vygen-Bonnet, S., Kothari, K. U., Grummich, K., Braun, C., Kapp, P., et al. (2023) · The Lancet Child & Adolescent Health · doi:10.1016/s2352-4642(23)00078-0 · Cited by 98
Self-citationWong, S., Le, G. H., Phan, L., Rhee, T. G., Ho, R., Meshkat, S., Teopiz, K. M., Kwan, A. T. H., Mansur, R. B., Rosenblat, J. D., et al. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.04.086 · Cited by 95
Emergency cerclage in singleton pregnancies with painless cervical dilatation: A meta‐analysis
Chatzakis, C., Efthymiou, A., Sotiriadis, A., & Makrydimas, G. (2020) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.13968 · Cited by 91
Assessing the certainty of the evidence in systematic reviews: importance, process, and use
Brignardello‐Petersen, R., & Guyatt, G. H. (2024) · American Journal of Epidemiology · doi:10.1093/aje/kwae332 · Cited by 91
Self-citationInterventions to reduce ambient particulate matter air pollution and their effect on health
Burns, J., Boogaard, H., Polus, S., Pfadenhauer, L. M., Rohwer, A. C., van Erp, A. M., Turley, R., & Rehfuess, E. A. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010919.pub2 · Cited by 93
Iliodromiti, S., McLaren, J., Ghouri, N., Miller, M. R., Leinhard, O. D., Linge, J., Ballantyne, S. A., Platt, J., Foster, J., Hanvey, S., et al. (2022) · Diabetologia · doi:10.1007/s00125-022-05803-5 · Cited by 95
Prevention of Cisplatin-Induced Acute Kidney Injury: A Systematic Review and Meta-Analysis
Hamroun, A., Lenain, R., Bigna, J. J., Speyer, É., Bui, L., Chamley, P., Pottier, N., Cauffiez, C., Dewaeles, E., Dhalluin, X., et al. (2019) · Drugs · doi:10.1007/s40265-019-01182-1 · Cited by 88
Risk factors for pre‐eclampsia in clinical practice guidelines: Comparison with the evidence
Elawad, T., Scott, G., Bone, J. N., Elwell, H., Lopez, C. E., Filippi, V., Green, M., Khalil, A. A., Kinshella, M. W., Mistry, H. D., et al. (2022) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17320 · Cited by 92
Ramai, D., Singh, J. K., Lester, J., Khan, S. R., Chandan, S., Tartaglia, N., Ambrosi, A., Serviddio, G., & Facciorusso, A. (2021) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.16335 · Cited by 88
Methods to Address Confounding and Other Biases in Meta-Analyses: Review and Recommendations
Mathur, M. B., & VanderWeele, T. J. (2022) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-051920-114020 · Cited by 85
Shields, M., Dimov, S., Kavanagh, A. M., Milner, A. J., Spittal, M. J., & King, T. L. (2021) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-021-02077-x · Cited by 85
Cannabis Use in Pregnancy and Neonatal Outcomes: A Systematic Review and Meta-Analysis
Lo, J. O.-T., Shaw, B., Robalino, S., Ayers, C. K., Durbin, S., Rushkin, M. C., Olyaei, A. F., Kansagara, D. L., & Harrod, C. S. (2023) · Cannabis and Cannabinoid Research · doi:10.1089/can.2022.0262 · Cited by 85
Zeraatkar, D., Cooper, M. A., Agarwal, A., Vernooij, R. W. M., Leung, G., Loniewski, K., Dookie, J., Ahmed, M. M., Hong, B. Y., Hong, C. J. H., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-054282 · Cited by 87
Li, D., Zhang, Y., Li, X., Zhang, K., Lü, Y., & Brown, R. D. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.164435 · Cited by 88
Kazemi, A., Soltani, S., Aune, D., Hosseini, E., Mokhtari, Z., Hassanzadeh, Z., Jayedi, A., Pitanga, F. J. G., & Akhlaghi, M. (2024) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-024-01593-8 · Cited by 92
Olaniran, A., Briggs, J., Pradhan, A., Bogue, E., Schreiber, B., Dini, H. S., Hurkchand, H., & Ballard, M. (2022) · Human Resources for Health · doi:10.1186/s12960-022-00755-8 · Cited by 84
Boon, M. H., Thomson, H., Shaw, B., Akl, E. A., Lhachimi, S. K., López‐Alcalde, J., Klugar, M., Choi, L., Saz‐Parkinson, Z., Mustafa, R. A., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.001 · Cited by 81
Self-citationRISK PREDICTORS OF EARLY CHILDHOOD CARIES INCREMENT—A SYSTEMATIC REVIEW AND META-ANALYSIS
Lam, P. P. Y., Chua, H., Ekambaram, M., Lo, E. C. M., & Yiu, C. K. Y. (2022) · Journal of Evidence Based Dental Practice · doi:10.1016/j.jebdp.2022.101732 · Cited by 83
Guyatt, G. H., Wang, Y., Eachempati, P., Iorio, A., Murad, M. H., Hultcrantz, M., Chu, D. K., Flórez, I. D., Hemkens, L. G., Agoritsas, T., et al. (2025) · BMJ · doi:10.1136/bmj-2024-083864 · Cited by 85
Self-citationCement plant emissions and health effects in the general population: a systematic review
Raffetti, E., Treccani, M., & Donato, F. (2018) · Chemosphere · doi:10.1016/j.chemosphere.2018.11.088 · Cited by 76
Abbing, A., Koretsi, V., Eliades, T., & Papageorgiou, S. N. (2020) · Progress in Orthodontics · doi:10.1186/s40510-020-00334-4 · Cited by 79
Rijkse, E., Ceuppens, S., Qi, H., IJzermans, J. N. M., Hesselink, D. A., & Minnee, R. C. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.106021 · Cited by 76
Community-level interventions for improving access to food in low- and middle-income countries
Durão, S., Visser, M. E., Ramokolo, V., de Oliveira, J. M., Schmidt, B., Balakrishna, Y., Brand, A., Kristjansson, E., & Schoonees, A. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011504.pub2 · Cited by 77
Drukker, M., Weltens, I., van Hooijdonk, C. F. M., Vandenberk, E., & Bak, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.675071 · Cited by 75
Papageorgiou, S. N., Antonoglou, G. N., Michelogiannakis, D., Kakali, L., Eliades, T., & Madianos, P. (2022) · Journal of Clinical Periodontology · doi:10.1111/jcpe.13529 · Cited by 75
Schwingshackl, L., Schünemann, H. J., & Meerpohl, J. J. (2020) · European Journal of Nutrition · doi:10.1007/s00394-020-02464-1 · Cited by 77
Self-citationMontgomery, P., Movsisyan, A., Grant, S., Macdonald, G. M., & Rehfuess, E. A. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000848 · Cited by 72
Malocclusions and quality of life among adolescents: a systematic review and meta-analysis
Göranson, E., Sonesson, M., Naimi‐Akbar, A., & Dimberg, L. (2023) · European Journal of Orthodontics · doi:10.1093/ejo/cjad009 · Cited by 76
Brennan, S., McDonald, S. J., Page, M. J., Reid, J. E. A., Ward, S. A., Forbes, A. B., & McKenzie, J. E. (2020) · Systematic Reviews · doi:10.1186/s13643-019-1220-4 · Cited by 75
Young, I., Greig, J., Wilhelm, B. J., & Waddell, L. A. (2019) · Journal of Food Protection · doi:10.4315/0362-028x.jfp-19-108 · Cited by 68
Behavioral interventions to extend sleep duration: A systematic review and meta-analysis
Baron, K. G., Duffecy, J. L., Reutrakul, S., Levenson, J. C., McFarland, M., Lee, S., & Qeadan, F. (2021) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2021.101532 · Cited by 70
Brignardello‐Petersen, R., TOMLINSON, G. A., Flórez, I. D., Rind, D. M., Chu, D. K., Morgan, R. L., Mustafa, R. A., Schünemann, H. J., & Guyatt, G. H. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.06.010 · Cited by 72
Self-citationDevasahayam, A. J., Farwell, K., Lim, B., Morton, A., Fleming, N., Jagroop, D., Aryan, R., Saumur, T. M., & Mansfield, A. (2022) · Physical Therapy · doi:10.1093/ptj/pzac154 · Cited by 68
Biologicals in atopic disease in pregnancy: An EAACI position paper
Pfaller, B., Yepes-Núñez, J. J., Agache, I. O., Akdiş, C. A., Alsalamah, M., Bavbek, S., Bossios, A., Boyman, O., Chaker, A. M., Chan, S., et al. (2020) · Allergy · doi:10.1111/all.14282 · Cited by 66
Incorvaia, C., Al-Ahmad, M. S., Ansotegui, I. J., Arasi, S., Bachert, C., Bos, C., Bousquet, J. J., Bożek, A., Caimmi, D. P., Calderón, M. A., et al. (2020) · Allergy · doi:10.1111/all.14575 · Cited by 65
The evolution of GRADE (part 3): A framework built on science or faith?
Mercuri, M., & Gafni, A. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13016 · Cited by 65
MacKinnon, M. C., Sargeant, J. M., Pearl, D. L., Reid‐Smith, R. J., Carson, C. A., Parmley, E. J., & McEwen, S. A. (2020) · Antimicrobial Resistance and Infection Control · doi:10.1186/s13756-020-00863-x · Cited by 64
Research Note: Interpreting findings of a systematic review using GRADE methods
Brennan, S., & Johnston, R. V. (2023) · Journal of physiotherapy · doi:10.1016/j.jphys.2023.05.012 · Cited by 65
Storage of an avulsed tooth prior to replantation: A systematic review and meta‐analysis
Brier, N. D., Dorien, O., Borra, V., Singletary, E. M., Zideman, D. A., & Buck, E. D. J. D. (2020) · Dental Traumatology · doi:10.1111/edt.12564 · Cited by 64
Schwingshackl, L., Balduzzi, S., Beyerbach, J., Bröckelmann, N., Werner, S. S., Zähringer, J., Nagavci, B., & Meerpohl, J. J. (2021) · BMJ · doi:10.1136/bmj.n1864 · Cited by 64
Self-citationIribarne-Durán, L. M., Peinado, F. M., Freire, C., Castillero-Rosales, I., Artacho‐Cordón, F., & Olea, N. (2021) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2021.150437 · Cited by 63
Azegami, T., Kounoue, N., Sofue, T., Yazawa, M., Tsujita, M., Masutani, K., Kataoka, Y., & Oguchi, H. (2023) · Renal Failure · doi:10.1080/0886022x.2023.2169618 · Cited by 63
Cai, C., Xiong, S., Millett, C. J., Xu, J., Tian, M., & Hone, T. (2023) · Health Policy and Planning · doi:10.1093/heapol/czad058 · Cited by 63
Barbaresko, J., Bröder, J., Conrad, J., Szczerba, E., Lang, A., & Schlesinger, S. (2025) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2024.2317877 · Cited by 60
Talebi, S., Zeraattalab-Motlagh, S., Rahimlou, M., Naeini, F., Ranjbar, M., Talebi, A., & Mohammadi, H. (2023) · Advances in Nutrition · doi:10.1016/j.advnut.2023.05.008 · Cited by 62
Lampousi, A., Carlsson, S., & Löfvenborg, J. E. (2021) · EBioMedicine · doi:10.1016/j.ebiom.2021.103633 · Cited by 61
HESS, D. R. (2023) · Respiratory Care · doi:10.4187/respcare.11170 · Cited by 65
Edmunds, K., Tuffaha, H., Galvao, D. A., Scuffham, P., & Newton, R. U. (2020) · Supportive Care in Cancer · doi:10.1007/s00520-019-05255-5 · Cited by 63
Papageorgiou, S. N., Eliades, T., & Hämmerle, C. H. F. (2018) · Clinical Oral Implants Research · doi:10.1111/clr.13291 · Cited by 62
Stratil, J. M., Biallas, R., Burns, J., Arnold, L., Geffert, K., Kunzler, A. M., Monsef, I., Stadelmaier, J., Wabnitz, K. J. P., Litwin, T., et al. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015085.pub2 · Cited by 59
Hedman, L., Galanti, M. R., Ryk, C., Gilljam, H., & Adermark, L. (2021) · Tobacco Prevention & Cessation · doi:10.18332/tpc/142320 · Cited by 58
Ludwig‐Walz, H., Dannheim, I., Pfadenhauer, L. M., Fegert, J. M., & Bujard, M. (2023) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-023-00612-z · Cited by 61
Benzene Exposure and Lung Cancer Risk: A Systematic Review and Meta-Analysis of Human Studies
Chiavarini, M., Rosignoli, P., Sorbara, B., Giacchetta, I., & Fabiani, R. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21020205 · Cited by 58
Verbeek, J. H. A. M., Whaley, P., Morgan, R. L., Taylor, K. W., Rooney, A. A., Schwingshackl, L., Hoving, J. L., Katikireddi, S. V., Shea, B., Mustafa, R. A., et al. (2021) · Environment International · doi:10.1016/j.envint.2021.106868 · Cited by 55
Self-citationLudwig‐Walz, H., Siemens, W., Heinisch, S., Dannheim, I., Loss, J., & Bujard, M. (2023) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-023-01542-x · Cited by 62
Metadata
- Authors
- Schünemann, H. J., Cuello, C., Akl, E. A., Mustafa, R. A., Meerpohl, J. J., Thayer, K., Morgan, R. L., Gartlehner, G., Kunz, R., Katikireddi, S. V., Sterne, J., Higgins, J. P., & Guyatt, G.
- Affiliations
- McMaster University (CA)
- University of Kansas Medical Center (US)
- Universität für Weiterbildung Krems (AT)
- University Hospital of Basel (CH)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- University of Bristol (GB)
- Corresponding author
- Schünemann, Holger J.
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 111
- Pages
- 105-114
- ISSN
- 0895-4356 (ISSN-L)
- PMC
- PMC6692166
- OpenAlex
- W2792308249
- Semantic Scholar
- cf0d116e848fca50508ac07e082d05e29ba05fcd
- Cited by
- 911
- References
- 27
- Citations by year
- 2026: 133 · 2025: 146 · 2024: 125 · 2023: 118 · 2022: 114 · 2021: 103 · 2020: 121 · 2019: 34
- FWCI
- 52.17
- Type
- journal article
- Language
- en
- Open access
- Licence
- cc-by-nc-nd
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion, cc-by-sa)
- edoc (University of Basel) (submittedVersion)
- Explore Bristol Research (submittedVersion, cc-by-nc-nd)
- Funders
- National Institute for Health and Care Research (NF-SI-0611-10168, NF-SI-0616-10111, NF-SI-0617-10145)
- Medical Research Council (MR/M025209/1, MC_UU_12017/13, MC_UU_12017/15)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- ROBINS-I, risk of bias, non-randomized studies, GRADE, quality of evidence, residual confounding, selection bias, confounding, evidence evaluation
- MeSH terms
- Systematic Reviews as Topic, Clinical Trials as Topic, Humans, Risk Factors, Bias, Practice Guidelines as Topic, Risk Assessment, Evidence-Based Medicine, Uncertainty, Observational Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
