The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
A risk of bias instrument for non-randomized studies of exposures: A users' guide to its application in the context of GRADE
Open accessAbstract
The objective of this paper is to explain how to apply, interpret, and present the results of a new instrument to assess the risk of bias (RoB) in non-randomized studies (NRS) dealing with effects of environmental exposures on health outcomes. This instrument is modeled on the Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) instrument. The RoB instrument for NRS of exposures assesses RoB along a standardized comparison to a randomized target experiment, instead of the study-design directed RoB approach. We provide specific guidance for the integral steps of developing a research question and target experiment, distinguishing issues of indirectness from RoB, making individual-study judgments, and performing and interpreting sensitivity analyses for RoB judgments across a body of evidence. Also, we present an approach for integrating the RoB assessments within the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework to assess the certainty of the evidence in the systematic review. Finally, we guide the reader through an overall assessment to support the rating of all domains that determine the certainty of a body of evidence using the GRADE approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The Risk of Bias in Non-randomized Studies—of Exposures (ROBINS-E) tool was used for nonrandomized comparative studies [19, 20].”
From Evidence for Human-to-Human Transmission of Hantavirus: A Systematic Review (2022) · cites this paper as [19]
In the citing paper
“Early work by a subset of the authors considered the extent to which the ROBINS-I tool was suitable for observational studies of exposures and suggested modifications to the tool (Morgan et al., 2018; Morgan et al., 2019).”
From A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E) (2024) · cites this paper as Morgan et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationGRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Self-citationUsing Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available: Table 1.
Hernán, M. A., & Robins, J. M. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kwv254 · Cited by 3,115
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Self-citationSmoking and Carcinoma of the Lung
Doll, R., & Hill, A. B. (1950) · BMJ · doi:10.1136/bmj.2.4682.739 · Cited by 1,999
GRADE guidelines: 8. Rating the quality of evidence—indirectness
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Falck-Ytter, Y., Jaeschke, R., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.014 · Cited by 1,843
Self-citationMorgan, R. L., Whaley, P., Thayer, K. A., & Schünemann, H. J. (2018) · Environment International · doi:10.1016/j.envint.2018.07.015 · Cited by 1,424
Self-citationGRADE guidelines: 9. Rating up the quality of evidence
Guyatt, G. H., Oxman, A. D., Sultan, S., Glasziou, P., Akl, E. A., Alonso-Coello, P., Atkins, D., Kunz, R., Brozek, J., Montori, V., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.06.004 · Cited by 1,336 · cited 6×
“These latter two domains contribute to increasing the CoE in a body of evidence of NRS in GRADE; however, within the RoB instrument for NRS of exposures they may also influence the study-level judgments (Guyatt et al., 2011c).”
BackgroundResultInfluentialSelf-citation(2016) · Cochrane Database of Systematic Reviews · Cited by 985
Flagged first, then by citations
Research works printed but not linked: 1
- Thayer, 2013, Draft protocol for systematic review to evaluate the evidence for an association between bisphenol A (BPA) exposure and obesity, Natl. Toxicol. Program
Cited by
Sorted by citations
Items 201–247 of 247
Chen, Q. J., Mozaffari, H., Ananthapavan, J., Smith, B. T., Wong, G. W. K., & Jessri, M. (2025) · Systematic Reviews · doi:10.1186/s13643-025-02978-x · Cited by 2
Parenting Autism: Exploring Social Support and Religiosity
Utami, R., Krisnatuti, D., & Johan, I. R. (2025) · Indigenous Jurnal Ilmiah Psikologi · doi:10.23917/indigenous.v10i2.10520 · Cited by 2
Yang, W., & Li, J. (2026) · Environmental Health and Preventive Medicine · doi:10.1265/ehpm.25-00424 · Cited by 1
Aung, Y. K., Zhang, Y., Baxter, N. N., Jenkins, M. A., & Win, A. K. (2026) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-26-0386 · Cited by 1
The three horizons model applied to medical science
Román-Belmonte, J. M., la Corte‐Rodríguez, H. D., Rodriguez-Merchan, E. C., Vázquez-Sasot, A., Rodríguez-Damiani, B. A., Luis, C. R., & Sánchez-Laguna, F. J. (2022) · Postgraduate Medicine · doi:10.1080/00325481.2022.2124086 · Cited by 1
Lee, H., Joung, D., Kwon, Y. J., & Park, B. (2023) · Journal of people, plants, and environment · doi:10.11628/ksppe.2023.26.5.535 · Cited by 1
Masri, A. A., Schiffner, U., Mourad, M. S., Schmoeckel, J., Joseph, P., & Splieth, C. H. (2023) · European Archives of Paediatric Dentistry · doi:10.1007/s40368-023-00854-7 · Cited by 1
Long-Term Outcomes of Patients With COVID-19 Who Are Critically Ill
Mazzochi, K., Silva, S. D., Chan, N. L. M., McGann, E., Robertson, T., Hellings, T., Hodgson, C. L., & Higgins, A. M. (2024) · CHEST Critical Care · doi:10.1016/j.chstcc.2024.100108 · Cited by 1
Ludwig‐Walz, H., Dannheim, I., Pfadenhauer, L. M., Fegert, J. M., & Bujard, M. (2022) · Research Square · doi:10.21203/rs.3.rs-1638547/v1 · Cited by 1
Is education a risk factor for Parkinson’s disease?: A systematic review and meta-analysisPreprint
Lammer, L., Pflanz, C. P., Kirk, M., Shahabuddin, A. A., & Bauermeister, S. (2025) · medRxiv · doi:10.1101/2025.07.02.25330781 · Cited by 1
Chen, J. (2021) · Utrecht University Repository (Utrecht University) · doi:10.33540/453
Chalak, M. H., Shekaftik, S. O., Jafari, H., Ghaffari, H. R., Dadipoor, S., & Popov, T. I. (2021) · Research Square · doi:10.21203/rs.3.rs-1173417/v1
Swain, C. T. V., Drummond, A. E., Milne, R. L., English, D. R., Brown, K. A., Lou, M. W. C., Boing, L., Bageley, A., Skinner, T. L., van Roekel, E. H., et al. (2023) · doi:10.1158/1055-9965.c.6491050
Lou, M. W. C., Drummond, A. E., Swain, C. T. V., Milne, R. L., English, D. R., Brown, K. A., van Roekel, E. H., Skinner, T. L., Moore, M. M., Gaunt, T. R., et al. (2023) · doi:10.1158/1055-9965.c.6491033.v1
Swain, C. T. V., Drummond, A. E., Milne, R. L., English, D. R., Brown, K. A., Lou, M. W. C., Boing, L., Bageley, A., Skinner, T. L., van Roekel, E. H., et al. (2023) · doi:10.1158/1055-9965.c.6491050.v1
Lou, M. W. C., Drummond, A. E., Swain, C. T. V., Milne, R. L., English, D. R., Brown, K. A., van Roekel, E. H., Skinner, T. L., Moore, M. M., Gaunt, T. R., et al. (2023) · doi:10.1158/1055-9965.22354157
Swain, C. T. V., Drummond, A. E., Milne, R. L., English, D. R., Brown, K. A., Lou, M. W. C., Boing, L., Bageley, A., Skinner, T. L., van Roekel, E. H., et al. (2023) · doi:10.1158/1055-9965.c.6491050.v2
Sexual Risk Behavior of Children and Adolescents With a History of Sexual Violence
Schulte, K. L., Szota, K., & Christiansen, H. (2023) · Kindheit und Entwicklung · doi:10.1026/0942-5403/a000411
Lou, M. W. C., Drummond, A. E., Swain, C. T. V., Milne, R. L., English, D. R., Brown, K. A., van Roekel, E. H., Skinner, T. L., Moore, M. M., Gaunt, T. R., et al. (2023) · doi:10.1158/1055-9965.22725240
Lou, M. W. C., Drummond, A. E., Swain, C. T. V., Milne, R. L., English, D. R., Brown, K. A., van Roekel, E. H., Skinner, T. L., Moore, M. M., Gaunt, T. R., et al. (2023) · doi:10.1158/1055-9965.c.6491033.v3
Berg, T., Flunkert, S., & Brenner, E. (2025) · Zeitschrift für Geburtshilfe und Neonatologie · doi:10.1055/a-2506-9511
Is vaping nicotine a gateway to smoking tobacco? Rules on how to review and design studies
Králíková, E., & Pavlovska, I. (2025) · Hygiena · doi:10.21101/hygiena.a1884
Caffeine in Preterm Infants and its Effect on Neonatal Sleep: A Systematic Review
Denisova, K., Diamandis, N., Tang, J., Eng-Kohn, M., Willson, G., Shrestha, B. V., DeStefano, A., Lee, J., Merrin, J., Lin, Z., et al. (2025) · The Journal of Clinical Pharmacology · doi:10.1002/jcph.70096
Hejazian, S. S., Shahjouei, S., Sadr, A. V., Vemuri, A. K., Bahrami, S., Abedi, V., & Zand, R. (2025) · Journal of the American Heart Association · doi:10.1161/jaha.125.042175
Bertoldi, C. S., Nunes, M. V. M. B., Moreira, M. F. S., & Drehmer, M. (2025) · Journal of Human Lactation · doi:10.1177/08903344251389600
Ren, Q., Wang, H., Zeng, Y., Fang, X., Wang, M., Li, D., Huang, W., & Xu, Y. (2022) · Figshare · doi:10.6084/m9.figshare.19298438
Ludwig‐Walz, H., Dannheim, I., Pfadenhauer, L. M., Fegert, J. M., & Bujard, M. (2024) · Figshare · doi:10.6084/m9.figshare.26558672
Orellano, P. W., Reynoso, J. I., & QUARANTA, N. E. (2023) · Figshare · doi:10.6084/m9.figshare.22623877
Ludwig‐Walz, H., Dannheim, I., Pfadenhauer, L. M., Fegert, J. M., & Bujard, M. (2024) · Figshare · doi:10.6084/m9.figshare.26581511
Chen, H., Zhang, J., Lai, J., Zhou, Y., Lin, X., Deng, G., Zhang, Z., & Li, L. (2021) · Figshare · doi:10.6084/m9.figshare.16305660
Ren, Q., Wang, H., Zeng, Y., Fang, X., Wang, M., Li, D., Huang, W., & Xu, Y. (2022) · Figshare · doi:10.6084/m9.figshare.19298432
尹杨雪, Xie, S., Xu, Q. S., Liao, L., Chen, H., & Zhou, R. (2024) · Figshare · doi:10.6084/m9.figshare.24416211
Sato, S., Nakanishi, M., Ogawa, M., Abe, M., Yasuma, N., Kono, T., Igarashi, M., Iwanaga, M., Kawaguchi, T., & Yamaguchi, S. (2023) · Figshare · doi:10.6084/m9.figshare.24452430
Urbano, T., Vinceti, M., Wise, L. A., & Filippini, T. (2021) · Figshare · doi:10.6084/m9.figshare.16822557
Self-citationShad, N. A., Li, W., Yu, Y., He, Y., Shi, S., Lu, M., Luo, H., Lin, Z., Chen, B., Li, Z., et al. (2026) · Ecotoxicology and Environmental Safety · doi:10.1016/j.ecoenv.2025.119656
A framework for developing PICO research questions with subgroups in clinical practice guidelines
Vella, E. T., Florez, I. D., Kellett, S., Varela, N. P., Zwaal, C., Sussman, J., & Yao, X. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112271
Radke, E. G., Mehta, S., Christensen, K. L. Y., Glenn, B., Rooney, A. A., Whaley, P., Morgan, R. L., Beverly, B., Thayer, K., Taylor, K. W., et al. (2026) · Environment International · doi:10.1016/j.envint.2026.110310
Self-citationMetals and Extremophilic Bacteria in Mining Environments: A Systematic Review
Jiménez-Venegas, J., Zamora-Leiva, L., Román‐Figueroa, C., Tapia, Y., & Paneque, M. (2026) · Microorganisms · doi:10.3390/microorganisms14061312
Ambient air pollution and depression: a systematic review and meta-analysis
Mohamed, H. A. S., Olstrup, H., Honore, J., Lu, S. S. M., Andersson, J., Oudin, A., Sigsgaard, T., & Schullehner, J. (2026) · Frontiers in Environmental Health · doi:10.3389/fenvh.2026.1809298
Xu, J., Zhang, P., Wang, Y., Yang, J., & Li, S. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1873054
Relationship between internalised homophobia and physical activity
Reynes, É. (2026) · Sustainability and Sports Science Journal · doi:10.55860/lkle1923
Flórez‐García, V., Guevara-Romero, E., Rod, J. E., Santacruz‐Salas, E., Noronha, N. Y., Schettino, M. J. T., Acosta, T., Molina, R. T., Acosta-Reyes, J., Navarro, É., et al. (2026) · Ecotoxicology and Environmental Safety · doi:10.1016/j.ecoenv.2026.120710
Melo, S. M. D. A., do Amaral, M. T. A., Santana, M. L. L., Nogueira, T. E., & Roriz, V. M. (2026) · International Journal of Dentistry · doi:10.1155/ijod/5546489
Tohari, A. I., Hidayat, M. R. F., Athoillah, N., Nugraha, M. Y., Sakinah, E. N., Supangat, S., Bakri, S., & Nandakumar, A. (2026) · Siriraj Medical Journal/San Sirirat · doi:10.33192/smj.v78i3.279093
Periodontal disease in patients with thyroid disorders: a systematic review and meta-analysis
Santos, P. G. S. T. D., Shazo, S., & Vellone, V. (2026) · Odontology · doi:10.1007/s10266-026-01591-1
Baldiotti, A. L. P., SANTOS-SILVA, P., Ferreira, F. M., Fernandes, I. B., & Martins, C. C. (2026) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2026.07.007
Hernández-Mariano, J. Á., Domínguez-Aliphat, A., Mercado-Martínez, M. N., López-Jardón, J. E., Ángel-López, J., Salinas-Cruz, J., & Terán-Godínez, C. (2026) · Women · doi:10.3390/women6040071
Metadata
- Authors
- Morgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Ansari, M., Akl, E. A., Filippini, T., Hansell, A., Meerpohl, J. J., Mustafa, R. A., Verbeek, J., Vinceti, M., Whaley, P., & Schünemann, H. J.
- Affiliations
- McMaster University (CA)
- Environmental Protection Agency (US)
- ICF International (United States) (US)
- University of Ottawa (CA)
- University of Modena and Reggio Emilia (IT)
- Imperial College Healthcare NHS Trust (GB)
- University of Freiburg (DE)
- University of Kansas Medical Center (US)
- Finnish Institute of Occupational Health (FI)
- Boston University (US)
- Lancaster University (GB)
- Corresponding author
- Schünemann, Holger J.
- ORCID
- Venue
- Environment International
- Publisher
- Elsevier BV
- Published
- 2019-01
- Volume
- 122
- Pages
- 168-184
- ISSN
- 0160-4120 (ISSN-L)
- PMC
- PMC8221004
- OpenAlex
- W2901788566
- Cited by
- 248
- References
- 33
- Citations by year
- 2026: 20 · 2025: 34 · 2024: 31 · 2023: 45 · 2022: 45 · 2021: 41 · 2020: 25 · 2019: 7
- FWCI
- 13.92
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Environment International (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Lancaster EPrints (Lancaster University) (acceptedVersion)
- PubMed Central (acceptedVersion, cc-by-nc-nd)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Leicester Research Archive (University of Leicester) (submittedVersion, other-oa)
- Funders
- Intramural Research Program of the National Institute of Environmental Health Sciences
- MacGRADE Centre at the McMaster University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- risk of bias, non-randomized studies, environmental exposure, ROBINS-I, GRADE, quality of evidence, indirectness, sensitivity analysis, systematic reviews
- MeSH terms
- Environmental Exposure, Humans, Random Allocation, Research Design, Bias, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review