The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
A risk of bias instrument for non-randomized studies of exposures: A users' guide to its application in the context of GRADE
Open accessAbstract
The objective of this paper is to explain how to apply, interpret, and present the results of a new instrument to assess the risk of bias (RoB) in non-randomized studies (NRS) dealing with effects of environmental exposures on health outcomes. This instrument is modeled on the Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) instrument. The RoB instrument for NRS of exposures assesses RoB along a standardized comparison to a randomized target experiment, instead of the study-design directed RoB approach. We provide specific guidance for the integral steps of developing a research question and target experiment, distinguishing issues of indirectness from RoB, making individual-study judgments, and performing and interpreting sensitivity analyses for RoB judgments across a body of evidence. Also, we present an approach for integrating the RoB assessments within the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework to assess the certainty of the evidence in the systematic review. Finally, we guide the reader through an overall assessment to support the rating of all domains that determine the certainty of a body of evidence using the GRADE approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The Risk of Bias in Non-randomized Studies—of Exposures (ROBINS-E) tool was used for nonrandomized comparative studies [19, 20].”
From Evidence for Human-to-Human Transmission of Hantavirus: A Systematic Review (2022) · cites this paper as [19]
In the citing paper
“Early work by a subset of the authors considered the extent to which the ROBINS-I tool was suitable for observational studies of exposures and suggested modifications to the tool (Morgan et al., 2018; Morgan et al., 2019).”
From A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E) (2024) · cites this paper as Morgan et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationGRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Self-citationUsing Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available: Table 1.
Hernán, M. A., & Robins, J. M. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kwv254 · Cited by 3,115
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Self-citationSmoking and Carcinoma of the Lung
Doll, R., & Hill, A. B. (1950) · BMJ · doi:10.1136/bmj.2.4682.739 · Cited by 1,999
GRADE guidelines: 8. Rating the quality of evidence—indirectness
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Falck-Ytter, Y., Jaeschke, R., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.014 · Cited by 1,843
Self-citationMorgan, R. L., Whaley, P., Thayer, K. A., & Schünemann, H. J. (2018) · Environment International · doi:10.1016/j.envint.2018.07.015 · Cited by 1,424
Self-citationGRADE guidelines: 9. Rating up the quality of evidence
Guyatt, G. H., Oxman, A. D., Sultan, S., Glasziou, P., Akl, E. A., Alonso-Coello, P., Atkins, D., Kunz, R., Brozek, J., Montori, V., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.06.004 · Cited by 1,336 · cited 6×
“These latter two domains contribute to increasing the CoE in a body of evidence of NRS in GRADE; however, within the RoB instrument for NRS of exposures they may also influence the study-level judgments (Guyatt et al., 2011c).”
BackgroundResultInfluentialSelf-citation(2016) · Cochrane Database of Systematic Reviews · Cited by 985
Schünemann, H. J., Cuello, C., Akl, E. A., Mustafa, R. A., Meerpohl, J. J., Thayer, K., Morgan, R. L., Gartlehner, G., Kunz, R., Katikireddi, S. V., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.012 · Cited by 911
Self-citationMortality in Relation to Smoking: Ten Years' Observations of British Doctors
Doll, R., & Hill, A. B. (1964) · BMJ · doi:10.1136/bmj.1.5396.1460 · Cited by 683
The Planning of Observational Studies of Human Populations
Cochran, W. G., & Chambers, S. P. (1965) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344179 · Cited by 569
Mortality in Relation to Smoking: Ten Years' Observations of British Doctors
Doll, R., & Hill, A. (1964) · BMJ · doi:10.1136/bmj.1.5395.1399 · Cited by 545
Woodruff, T. J., & Sutton, P. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307175 · Cited by 427
Proper interpretation of non-differential misclassification effects: expectations vs observations
Jurek, A. M., Greenland, S., Maldonado, G., & Church, T. R. (2005) · International Journal of Epidemiology · doi:10.1093/ije/dyi060 · Cited by 426
Urinary bisphenol A and obesity: NHANES 2003–2006
Carwile, J. L., & Michels, K. B. (2011) · Environmental Research · doi:10.1016/j.envres.2011.05.014 · Cited by 388
Pharmacokinetics of bisphenol A in humans following a single oral administration
Thayer, K. A., Doerge, D. R., Hunt, D., Schurman, S. H., Twaddle, N. C., Churchwell, M. I., Garantziotis, S., Kissling, G. E., Easterling, M. R., Bucher, J. R., et al. (2015) · Environment International · doi:10.1016/j.envint.2015.06.008 · Cited by 328
Self-citationRancière, F., Lyons, J. G., Loh, V. H. Y., Botton, J., Galloway, T. S., Wang, T., Shaw, J. E., & Magliano, D. J. (2015) · Environmental Health · doi:10.1186/s12940-015-0036-5 · Cited by 313
GRADE: Assessing the quality of evidence in environmental and occupational health
Morgan, R. L., Thayer, K. A., Bero, L. A., Bruce, N., Falck-Ytter, Y. T., Ghersi, D., Guyatt, G. H., Hooijmans, C. R., Langendam, M. W., Mandrioli, D., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.01.004 · Cited by 287
Self-citationPrenatal and Postnatal Bisphenol A Exposure and Body Mass Index in Childhood in the CHAMACOS Cohort
Harley, K. G., Schall, R. A., Chevrier, J., Tyler, K., Aguirre, H., Bradman, A., Holland, N., Lustig, R. H., Calafat, A. M., & Eskenazi, B. (2013) · Environmental Health Perspectives · doi:10.1289/ehp.1205548 · Cited by 270
Blair, A., Stewart, P., Lubin, J. H., & Forastiere, F. (2007) · American Journal of Industrial Medicine · doi:10.1002/ajim.20281 · Cited by 251
Human exposures to bisphenol A: mismatches between data and assumptions
Vandenberg, L. N., Hunt, P. A., Myers, J. P., & Saal, F. S. V. (2013) · Reviews on Environmental Health · doi:10.1515/reveh-2012-0034 · Cited by 212
Spurious effects from an extraneous variable
Bross, I. D. J. (1966) · Journal of Chronic Diseases · doi:10.1016/0021-9681(66)90062-2 · Cited by 184
Urine Bisphenol-A Level in Relation to Obesity and Overweight in School-Age Children
Li, D., Miao, M., Zhou, Z., Wu, C., Shi, H., Liu, X., Wang, S., & Yuan, W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0065399 · Cited by 159
Wang, H., Zhou, Y., Tang, C., Wu, J., Chen, Y., & Jiang, Q. (2012) · Environmental Health · doi:10.1186/1476-069x-11-79 · Cited by 138
Morgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Guyatt, G., & Schünemann, H. J. (2018) · Environment International · doi:10.1016/j.envint.2018.08.018 · Cited by 106
Self-citationRooney, A. A., Cooper, G. S., Jahnke, G. D., Lam, J., Morgan, R. L., Boyles, A. L., Ratcliffe, J. M., Kraft, A. D., Schünemann, H. J., Schwingl, P. J., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.01.005 · Cited by 95
Self-citationExposure Assessment in Environmental Epidemiology
(2015) · Oxford University Press eBooks · doi:10.1093/med/9780199378784.001.0001 · Cited by 89
Topics in Environmental Epidemiology
Steenland, K., & Savitz, D. A. (1998) · Epidemiology · doi:10.1097/00001648-199803000-00021 · Cited by 53
Cox, K. J., Porucznik, C. A., Anderson, D. J., Brozek, E., Szczotka, K. M., Bailey, N. M., Wilkins, D. G., & Stanford, J. B. (2015) · Environmental Health Perspectives · doi:10.1289/ehp.1509752 · Cited by 40
Kogevinas, M. (2011) · Environmental Health · doi:10.1186/1476-069x-10-s1-s3 · Cited by 36
Topics in Environmental Epidemiology
(1997) · doi:10.1093/oso/9780195095647.001.0001 · Cited by 29
Research works printed but not linked: 1
- Thayer, 2013, Draft protocol for systematic review to evaluate the evidence for an association between bisphenol A (BPA) exposure and obesity, Natl. Toxicol. Program
Cited by
Sorted by citations
A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E)
Higgins, J. P. T., Morgan, R. L., Rooney, A. A., Taylor, K. W., Thayer, K. A., Silva, R. A., Lemeris, C., Akl, E. A., Bateson, T. F., Berkman, N. D., et al. (2024) · Environment International · doi:10.1016/j.envint.2024.108602 · Cited by 1,022
“Early work by a subset of the authors considered the extent to which the ROBINS-I tool was suitable for observational studies of exposures and suggested modifications to the tool (Morgan et al., 2018; Morgan et al., 2019).”
Self-citationOrellano, P. W., Reynoso, J. I., QUARANTA, N. E., Bardach, A., & Ciapponi, A. (2020) · Environment International · doi:10.1016/j.envint.2020.105876 · Cited by 857
Huangfu, P., & Atkinson, R. W. (2020) · Environment International · doi:10.1016/j.envint.2020.105998 · Cited by 548
Li, L. Z., Yang, P., Singer, S. J., Pfeffer, J., Mathur, M. B., & Shanafelt, T. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.43059 · Cited by 501
Air pollution exposure and depression: A comprehensive updated systematic review and meta-analysis
Borroni, E., Pesatori, A. C., Bollati, V., Buoli, M., & Carugno, M. (2021) · Environmental Pollution · doi:10.1016/j.envpol.2021.118245 · Cited by 287
Metadata
- Authors
- Morgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Ansari, M., Akl, E. A., Filippini, T., Hansell, A., Meerpohl, J. J., Mustafa, R. A., Verbeek, J., Vinceti, M., Whaley, P., & Schünemann, H. J.
- Affiliations
- McMaster University (CA)
- Environmental Protection Agency (US)
- ICF International (United States) (US)
- University of Ottawa (CA)
- University of Modena and Reggio Emilia (IT)
- Imperial College Healthcare NHS Trust (GB)
- University of Freiburg (DE)
- University of Kansas Medical Center (US)
- Finnish Institute of Occupational Health (FI)
- Boston University (US)
- Lancaster University (GB)
- Corresponding author
- Schünemann, Holger J.
- ORCID
- Venue
- Environment International
- Publisher
- Elsevier BV
- Published
- 2019-01
- Volume
- 122
- Pages
- 168-184
- ISSN
- 0160-4120 (ISSN-L)
- PMC
- PMC8221004
- OpenAlex
- W2901788566
- Cited by
- 248
- References
- 33
- Citations by year
- 2026: 20 · 2025: 34 · 2024: 31 · 2023: 45 · 2022: 45 · 2021: 41 · 2020: 25 · 2019: 7
- FWCI
- 13.92
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Environment International (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Lancaster EPrints (Lancaster University) (acceptedVersion)
- PubMed Central (acceptedVersion, cc-by-nc-nd)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Leicester Research Archive (University of Leicester) (submittedVersion, other-oa)
- Funders
- Intramural Research Program of the National Institute of Environmental Health Sciences
- MacGRADE Centre at the McMaster University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- risk of bias, non-randomized studies, environmental exposure, ROBINS-I, GRADE, quality of evidence, indirectness, sensitivity analysis, systematic reviews
- MeSH terms
- Environmental Exposure, Humans, Random Allocation, Research Design, Bias, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
