The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
A risk of bias instrument for non-randomized studies of exposures: A users' guide to its application in the context of GRADE
Open accessAbstract
The objective of this paper is to explain how to apply, interpret, and present the results of a new instrument to assess the risk of bias (RoB) in non-randomized studies (NRS) dealing with effects of environmental exposures on health outcomes. This instrument is modeled on the Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) instrument. The RoB instrument for NRS of exposures assesses RoB along a standardized comparison to a randomized target experiment, instead of the study-design directed RoB approach. We provide specific guidance for the integral steps of developing a research question and target experiment, distinguishing issues of indirectness from RoB, making individual-study judgments, and performing and interpreting sensitivity analyses for RoB judgments across a body of evidence. Also, we present an approach for integrating the RoB assessments within the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework to assess the certainty of the evidence in the systematic review. Finally, we guide the reader through an overall assessment to support the rating of all domains that determine the certainty of a body of evidence using the GRADE approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The Risk of Bias in Non-randomized Studies—of Exposures (ROBINS-E) tool was used for nonrandomized comparative studies [19, 20].”
From Evidence for Human-to-Human Transmission of Hantavirus: A Systematic Review (2022) · cites this paper as [19]
In the citing paper
“Early work by a subset of the authors considered the extent to which the ROBINS-I tool was suitable for observational studies of exposures and suggested modifications to the tool (Morgan et al., 2018; Morgan et al., 2019).”
From A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E) (2024) · cites this paper as Morgan et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationGRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Self-citationUsing Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available: Table 1.
Hernán, M. A., & Robins, J. M. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kwv254 · Cited by 3,115
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Self-citationSmoking and Carcinoma of the Lung
Doll, R., & Hill, A. B. (1950) · BMJ · doi:10.1136/bmj.2.4682.739 · Cited by 1,999
GRADE guidelines: 8. Rating the quality of evidence—indirectness
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Falck-Ytter, Y., Jaeschke, R., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.014 · Cited by 1,843
Self-citationMorgan, R. L., Whaley, P., Thayer, K. A., & Schünemann, H. J. (2018) · Environment International · doi:10.1016/j.envint.2018.07.015 · Cited by 1,424
Self-citationGRADE guidelines: 9. Rating up the quality of evidence
Guyatt, G. H., Oxman, A. D., Sultan, S., Glasziou, P., Akl, E. A., Alonso-Coello, P., Atkins, D., Kunz, R., Brozek, J., Montori, V., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.06.004 · Cited by 1,336 · cited 6×
“These latter two domains contribute to increasing the CoE in a body of evidence of NRS in GRADE; however, within the RoB instrument for NRS of exposures they may also influence the study-level judgments (Guyatt et al., 2011c).”
BackgroundResultInfluentialSelf-citation(2016) · Cochrane Database of Systematic Reviews · Cited by 985
Research works printed but not linked: 1
- Thayer, 2013, Draft protocol for systematic review to evaluate the evidence for an association between bisphenol A (BPA) exposure and obesity, Natl. Toxicol. Program
Cited by
Sorted by citations
A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E)
Higgins, J. P. T., Morgan, R. L., Rooney, A. A., Taylor, K. W., Thayer, K. A., Silva, R. A., Lemeris, C., Akl, E. A., Bateson, T. F., Berkman, N. D., et al. (2024) · Environment International · doi:10.1016/j.envint.2024.108602 · Cited by 1,022
“Early work by a subset of the authors considered the extent to which the ROBINS-I tool was suitable for observational studies of exposures and suggested modifications to the tool (Morgan et al., 2018; Morgan et al., 2019).”
Self-citationOrellano, P. W., Reynoso, J. I., QUARANTA, N. E., Bardach, A., & Ciapponi, A. (2020) · Environment International · doi:10.1016/j.envint.2020.105876 · Cited by 857
Huangfu, P., & Atkinson, R. W. (2020) · Environment International · doi:10.1016/j.envint.2020.105998 · Cited by 548
Li, L. Z., Yang, P., Singer, S. J., Pfeffer, J., Mathur, M. B., & Shanafelt, T. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.43059 · Cited by 501
Air pollution exposure and depression: A comprehensive updated systematic review and meta-analysis
Borroni, E., Pesatori, A. C., Bollati, V., Buoli, M., & Carugno, M. (2021) · Environmental Pollution · doi:10.1016/j.envpol.2021.118245 · Cited by 287
Metadata
- Authors
- Morgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Ansari, M., Akl, E. A., Filippini, T., Hansell, A., Meerpohl, J. J., Mustafa, R. A., Verbeek, J., Vinceti, M., Whaley, P., & Schünemann, H. J.
- Affiliations
- McMaster University (CA)
- Environmental Protection Agency (US)
- ICF International (United States) (US)
- University of Ottawa (CA)
- University of Modena and Reggio Emilia (IT)
- Imperial College Healthcare NHS Trust (GB)
- University of Freiburg (DE)
- University of Kansas Medical Center (US)
- Finnish Institute of Occupational Health (FI)
- Boston University (US)
- Lancaster University (GB)
- Corresponding author
- Schünemann, Holger J.
- ORCID
- Venue
- Environment International
- Publisher
- Elsevier BV
- Published
- 2019-01
- Volume
- 122
- Pages
- 168-184
- ISSN
- 0160-4120 (ISSN-L)
- PMC
- PMC8221004
- OpenAlex
- W2901788566
- Cited by
- 248
- References
- 33
- Citations by year
- 2026: 20 · 2025: 34 · 2024: 31 · 2023: 45 · 2022: 45 · 2021: 41 · 2020: 25 · 2019: 7
- FWCI
- 13.92
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Environment International (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Lancaster EPrints (Lancaster University) (acceptedVersion)
- PubMed Central (acceptedVersion, cc-by-nc-nd)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Leicester Research Archive (University of Leicester) (submittedVersion, other-oa)
- Funders
- Intramural Research Program of the National Institute of Environmental Health Sciences
- MacGRADE Centre at the McMaster University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- risk of bias, non-randomized studies, environmental exposure, ROBINS-I, GRADE, quality of evidence, indirectness, sensitivity analysis, systematic reviews
- MeSH terms
- Environmental Exposure, Humans, Random Allocation, Research Design, Bias, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
