Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools
Open accessAbstract
BACKGROUND: Systematic reviews are increasingly prevalent in environmental health due to their ability to synthesize evidence while reducing bias. Different systematic review methods have been developed by the US National Toxicology Program's Office of Health Assessment and Translation (OHAT), the US Environmental Protection Agency's (EPA) Integrated Risk Information System (IRIS), and by the US EPA under the Toxic Substances Control Act (TSCA), including the approach to assess risk of bias (ROB), one of the most vital steps which is used to evaluate internal validity of the studies. Our objective was to compare the performance of three tools (OHAT, IRIS, TSCA) in assessing ROB. METHODS: We selected a systematic review on polybrominated diphenyl ethers and intelligence quotient and/or attention deficit hyperactivity disorder because it had been endorsed by the National Academy of Sciences. Two reviewers followed verbatim instructions from the tools and independently applied each tool to assess ROB in 15 studies previously identified. We documented the time to apply each tool and the impact the ROB ratings for each tool had on the final rating of the quality of the overall body of evidence. RESULTS: The time to complete the ROB assessments varied widely (mean = 20, 32, and 40 min per study for the OHAT, IRIS, and TSCA tools, respectively). All studies were rated overall "low" or "uninformative" using IRIS, due to "deficient" or "critically deficient" ratings in one or two domains. Similarly, all studies were rated "unacceptable" using the TSCA tool because of one "unacceptable" rating in a metric related to statistical power. Approximately half of the studies had "low" or "probably low ROB" ratings across all domains with the OHAT and Navigation Guide tools. CONCLUSIONS: Tools that use overall ROB or study quality ratings, such as IRIS and TSCA, may reduce the available evidence to assess the harms of environmental exposures by erroneously excluding studies, which leads to inaccurate conclusions about the quality of the body of evidence. We recommend using ROB tools that circumvents these issues, such as OHAT and Navigation Guide. SYSTEMATIC REVIEW REGISTRATION: This review has not been registered as it is not a systematic review.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Different risk-of-bias tools, and different ways of addressing risk-of-bias assessments, can lead to differences in conclusions of systematic reviews.
“different tools, and different ways of addressing risk-of-bias assessments, can lead to differences in conclusions of systematic reviews”
From A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E) (2024) · cites this paper as Eick et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,577 · see citing works
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · see citing works
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613 · see citing works
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · see citing works
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2014) · International Journal of Surgery · doi:10.1016/j.ijsu.2014.07.014 · Cited by 2,969 · see citing works
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Johnson, P. I., Sutton, P., Atchley, D. S., Koustas, E., Lam, J., Sen, S., Robinson, K. A., Axelrad, D. A., & Woodruff, T. J. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307893 · Cited by 449 · see citing works
Self-citationRooney, A. A., Boyles, A. L., Wolfe, M. S., Bucher, J. R., & Thayer, K. A. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307972 · Cited by 434 · see citing works
Woodruff, T. J., & Sutton, P. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307175 · Cited by 427 · see citing works
Self-citationRoze, E., Meijer, L., Bakker, A., Braeckel, K. N. J. A. V., Sauer, P. J. J., & Bos, A. F. (2009) · Environmental Health Perspectives · doi:10.1289/ehp.0901015 · Cited by 392 · see citing works
Koustas, E., Lam, J., Sutton, P., Johnson, P. I., Atchley, D. S., Sen, S., Robinson, K. A., Axelrad, D. A., & Woodruff, T. J. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307177 · Cited by 349 · see citing works
Self-citationPage, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159267 · Cited by 284 · see citing works
Bero, L. A., Chartres, N., Diong, J. H. L., Fabbri, A., Ghersi, D., Lam, J., Lau, A., McDonald, S., Mintzes, B. J., Sutton, P., et al. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0915-2 · Cited by 279 · see citing works
Self-citationO'Connor, S. R., Tully, M. A., Ryan, B., Bradley, J. M., Baxter, D. G., & McDonough, S. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1181-1 · Cited by 248 · see citing works
A Systematic Review and Meta-Analysis of Multiple Airborne Pollutants and Autism Spectrum Disorder
Lam, J., Sutton, P., Kalkbrenner, A., Windham, G., Halladay, A., Koustas, E., Lawler, C., Davidson, L., Daniels, N., Newschaffer, C., et al. (2016) · PLOS ONE · doi:10.1371/journal.pone.0161851 · Cited by 247 · see citing works
Self-citationThe National Academies of SCIENCES • ENGINEERING • MEDICINE
Earth, D. O. (2016) · Cited by 233 · see citing works
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · see citing works
Developmental PBDE Exposure and IQ/ADHD in Childhood: A Systematic Review and Meta-analysis
Lam, J., Lanphear, B., Bellinger, D., Axelrad, D., McPartland, J., Sutton, P. M., Davidson, L., Daniels, N., Sen, Ś., & Woodruff, T. (2017) · Environmental Health Perspectives · doi:10.1289/ehp1632 · Cited by 194 · see citing works
Self-citationHootman, J. M., Driban, J. B., Sitler, M. R., Harris, K. P., & Cattano, N. M. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.41 · Cited by 176 · see citing works
Review of EPA's Integrated Risk Information System (IRIS) Process
National Research Council (2014) · National Academies Press eBooks · doi:10.17226/18764 · Cited by 132 · see citing works
Johnson, P. I., Koustas, E., Vesterinen, H., Sutton, P. M., Atchley, D. S., Kim, A., Campbell, M. A., Donald, J. M., Sen, Ś., Bero, L., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.03.009 · Cited by 130 · see citing works
Self-citationLevels of Breast Milk PBDEs From Southern Taiwan and Their Potential Impact on Neurodevelopment
Chao, H., Tsou, T., Huang, H.-L., et al. (2011) · Pediatric Research · doi:10.1203/pdr.0b013e3182320b9b · Cited by 113 · see citing works
Three risk of bias tools lead to opposite conclusions in observational research synthesis
Losilla, J.-M., Oliveras, I., Marin-Garcia, J. A., & Vives, J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.021 · Cited by 111 · see citing works
Hoffman, K., Adgent, M. A., Goldman, B. D., et al. (2012) · Environmental Health Perspectives · doi:10.1289/ehp.1205100 · Cited by 100 · see citing works
Rooney, A. A., Cooper, G. S., Jahnke, G. D., Lam, J., Morgan, R. L., Boyles, A. L., Ratcliffe, J. M., Kraft, A. D., Schünemann, H. J., Schwingl, P. J., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.01.005 · Cited by 95 · see citing works
Self-citationPrenatal exposure to polybrominated diphenyl ethers and child attention problems at 3–7 years
Cowell, W., Lederman, S. A., Sjödin, A., Jones, R. S., Wang, S., Perera, F. P., Wang, R., Rauh, V., & Herbstman, J. B. (2015) · Neurotoxicology and Teratology · doi:10.1016/j.ntt.2015.08.009 · Cited by 88 · see citing works
Whiting, P. F., Rutjes, A. W., Dinnes, J., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.04.008 · Cited by 79 · see citing works
Board on Environmental Studies and Toxicology, Studies, D. O. E. A. L., & National Academies of Sciences, E. (2017) · National Academies Press eBooks · doi:10.17226/24758 · Cited by 77 · see citing works
Brominated Flame Retardants in Breast Milk and Behavioural and Cognitive Development at 36 Months
Adgent, M. A., Hoffman, K., Goldman, B. D., et al. (2013) · Paediatric and Perinatal Epidemiology · doi:10.1111/ppe.12078 · Cited by 67 · see citing works
Vesterinen, H., Morello-Frosch, R., Sen, Ś., Zeise, L., & Woodruff, T. (2017) · PLoS ONE · doi:10.1371/journal.pone.0176331 · Cited by 67 · see citing works
Self-citationNeurodevelopment of Infants with Prenatal Exposure to Polybrominated Diphenyl Ethers
Shy, C.-G., Huang, H.-L., Chang-Chien, G.-P., et al. (2011) · Bulletin of Environmental Contamination and Toxicology · doi:10.1007/s00128-011-0422-9 · Cited by 54 · see citing works
A systematic review: Tools for assessing methodological quality of human observational studiesPreprint
Wang, Z., Taylor, K. W., Allman-Farinelli, M. A., Armstrong, B. K., Askie, L. M., Ghersi, D., McKenzie, J. E., Norris, S. L., Page, M. J., Rooney, A. A., et al. (2019) · doi:10.31222/osf.io/pnqmy · Cited by 52 · see citing works
Self-citationFetal growth and maternal glomerular filtration rate: a systematic review
Vesterinen, H., Johnson, P. I., Atchley, D. S., Sutton, P. M., Lam, J., Zlatnik, M., Sen, Ś., & Woodruff, T. (2015) · Journal of Maternal-Fetal & Neonatal Medicine · doi:10.3109/14767058.2014.980809 · Cited by 49 · see citing works
Self-citationGump, B. B., Yun, S., & Kannan, K. (2014) · Environmental Research · doi:10.1016/j.envres.2014.04.009 · Cited by 34 · see citing works
Review of DOD's Approach to Deriving an Occupational Exposure Level for Trichloroethylene
Board on Environmental Studies and Toxicology, Studies, D. O. E. A. L., & National Academies of Sciences, E. (2019) · National Academies Press eBooks · doi:10.17226/25610 · Cited by 5 · see citing works
Lin, D.-Y., Chao, H., Gou, Y.-Y., et al. (2010) · doi:10.1109/iccceng.2010.5560424 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- [13]Rooney AA Cooper GS Jahnke GD Lam J Morgan RL Boyles AL How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards Environ Int. 2016 92-93 617 629 10.1016/j.envint.2016.01.005 26857180 PMC4902751 · DOI
- [22]National Research Council Review of EPA’s integrated risk information system (IRIS) process 2014 Washington, DC The National Academies Press 170 25101400
- [29]Committee to Review the IRIS Process; Board on Environmental Studies and Toxicology; Division on Earth and Life Studies; National Research Council. Review of EPA’s integrated risk information system (IRIS) process. 2014. Available from: https://www.ncbi.nlm.nih.gov/books/NBK230060/. 25101400
- [39]Wang Z, Taylor K, Allman-Farinelli M, Armstrong B, Askie L, Ghersi D, et al. A systematic review: tools for assessing methodological quality of human observational studies. MetaArXiv. 2019.
- [41]Higgins JPT Green S Cochrane C Cochrane handbook for systematic reviews of interventions 2008 Chichester, England; Hoboken, NJ Wiley-Blackwell
- [43]Ntp. Handbook for conducting a literature-based health assessment using OHAT approach for systematic review and evidence integration. U.S. Dept. of Health and Human Services, National Toxicology Program; 2015.
- [45]Losilla J-M Oliveras I Marin-Garcia JA Vives J Three risk of bias tools lead to opposite conclusions in observational research synthesis J Clin Epidemiol. 2018 101 61 72 10.1016/j.jclinepi.2018.05.021 29864541 · DOI · Paper page
Cited by
Sorted by citations
A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E)
Higgins, J. P. T., Morgan, R. L., Rooney, A. A., Taylor, K. W., Thayer, K. A., Silva, R. A., Lemeris, C., Akl, E. A., Bateson, T. F., Berkman, N. D., et al. (2024) · Environment International · doi:10.1016/j.envint.2024.108602 · Cited by 1,022 · see citing works
“Comparisons between these tools have been published previously (National Toxicology Program, 2015; Steenland et al., 2020; Eick et al., 2020; Radke et al., 2021; Eick et al., 2022) and have observed that different tools, and different ways of addressing risk-of-bias assessments, can lead to…”
Rosato, I., Bonato, T., Fletcher, T., Batzella, E., & Canova, C. (2024) · Environmental Research · doi:10.1016/j.envres.2023.117743 · Cited by 273 · see citing works
Radke, E. G., Glenn, B. S., & Kraft, A. D. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01783-6 · Cited by 3 · see citing works
- Authors
- Eick, S. M., Goin, D. E., Chartres, N., Lam, J., & Woodruff, T. J.
- Affiliations
- University of California, San Francisco (US)
- Corresponding author
- Woodruff, Tracey J.
- Venue
- Systematic Reviews
- Published
- 2020-10-29
- Volume
- 9
- Issue
- 1
- Pages
- 249-249
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC7596989
- OpenAlex
- W3097377405
- Cited by
- 82
- References
- 39
- Citations by year
- 2026: 13 · 2025: 14 · 2024: 18 · 2023: 17 · 2022: 10 · 2021: 10
- FWCI
- 5.64
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- JPB Foundation (681)
- Passport Foundation
- Clarence E. Heller Charitable Foundation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Environmental Science › Health, Toxicology and Mutagenesis › Health, Environment, Cognitive Aging
- Keywords
- risk of bias assessment, environmental health, iris, Toxic Substances Control Act, study quality ratings, polybrominated diphenyl ethers, intelligence quotient, attention deficit hyperactivity disorder, internal validity, statistical power
- MeSH terms
- Environmental Health, Humans, Research Design, Bias, Risk Assessment, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
