Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 4 of the 28 sources its claims cite were checked for support.
Abstract
Background: Systematic-review methodologies provide objectivity and transparency to the process of collecting and synthesizing scientific evidence in reaching conclusions on specific research questions. There is increasing interest in applying these procedures to address environmental health questions. Objectives: The goal was to develop a systematic-review framework to address environmental health questions by extending approaches developed for clinical medicine to handle the breadth of data relevant to environmental health sciences (e.g., human, animal, and mechanistic studies). Methods: The Office of Health Assessment and Translation (OHAT) adapted guidance from authorities on systematic-review and sought advice during development of the OHAT Approach through consultation with technical experts in systematic review and human health assessments, as well as scientific advisory groups and the public. The method was refined by considering expert and public comments and through application to case studies. Results and Discussion: Here we present a seven-step framework for systematic review and evidence integration for reaching hazard identification conclusions: 1) problem formulation and protocol development, 2) search for and select studies for inclusion, 3) extract data from studies, 4) assess the quality or risk of bias of individual studies, 5) rate the confidence in the body of evidence, 6) translate the confidence ratings into levels of evidence, and 7) integrate the information from different evidence streams (human, animal, and “other relevant data” including mechanistic or in vitro studies) to develop hazard identification conclusions. Conclusion: The principles of systematic review can be successfully applied to environmental health questions to provide greater objectivity and transparency to the process of developing conclusions. Citation: Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. 2014. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect 122:711–718; http://dx.doi.org/10.1289/ehp.1307972
Overview
CoChat Summary
TL;DR
This paper presents the OHAT seven-step framework for systematic review and evidence integration in environmental health assessments, extending clinical systematic-review methods to human, animal, and other relevant data. The approach is designed to improve objectivity, transparency, and consistency in hazard identification conclusions.
Key findings
- The OHAT framework is a flexible seven-step process: 1) problem formulation and protocol development, 2) search for and select studies, 3) extract data, 4) assess study quality/risk of bias, 5) rate confidence in the body of evidence, 6) translate confidence ratings into levels of evidence, and 7) integrate evidence streams to develop hazard identification conclusions.
- In applying the OHAT Approach to case studies, steps 2–4 were the most time intensive: selecting studies, extracting data, and assessing the quality of individual studies.
- The framework defines five hazard identification conclusion categories: known, presumed, suspected, not classifiable, and not identified as a hazard to humans.
- The paper states that high confidence means further research is very unlikely to change the apparent relationship, while very low confidence means further research is very likely to change it; for a conclusion of evidence of no health effect, the paper says this is reached only when there is high confidence.
Methods
- The authors developed the OHAT Approach by adapting guidance from authoritative systematic-review groups and consulting technical experts, scientific advisory groups, and the public; the method was refined through comments and application to case studies.
- The framework was built to cover human, animal, and other relevant data (including mechanistic or in vitro studies) in literature-based environmental health evaluations.
- The paper describes the method as a seven-step systematic-review and evidence-integration protocol, including prespecified procedures for study selection, data extraction, risk-of-bias assessment, confidence rating, and evidence integration.
Limitations
- The paper states that systematic-review methods may add time and investment to literature-based evaluations, and notes that steps 2–4 were the most time intensive.
- It also says that the scientific judgments used in confidence ratings are inherently subjective; this is the paper’s own stated limitation.
- Evident from the text: the framework is not a report of results from an empirical study with a sample, so there are no quantitative effect estimates or outcome findings to assess here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
3 Valid · 0 Not valid · 16 Not checked · 52 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It includes all of the recommended elements for conducting and reporting a systematic review [outlined in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement (Moher et al. 2009)].”
§ Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments > Results
What the sources say
- Partly supportedcites Moher et al. 2009 sourceExperimental — accuracy not yet measured
Section 4 (The PRISMA Statement) confirms that PRISMA outlines recommended elements — a 27-item checklist plus a four-phase flow diagram — but frames them strictly as reporting guidance: 'The aim of the PRISMA Statement is to help authors improve the reporting of systematic reviews and meta-analyses,' and explicitly adds that 'the PRISMA checklist is not a quality assessment instrument to gauge the quality of a systematic review.' The dedicated section 'Conduct and Reporting Research Are Distinct Concepts' maintains that distinction (noting only that the two are closely intertwined for systematic reviews), so the claim's 'elements for conducting and reporting' overstates the source on the 'conducting' half; Table 1 likewise lists 'Checklist of items to include when reporting a systematic review or meta-analysis.' Whether OHAT covers all of those elements is the citing authors' own comparison, not something this source addresses. (LLM, conf=0.75)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
In the paper
“In applying the OHAT Approach to case studies (NTP 2013a, 2013f), the NTP found that steps 2–4 were the most time intensive: selecting studies, extracting data, and assessing the quality of individual studies.”
§ Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments > Discussion
What the sources say
- Not checked yetcites NTP 2013a, 2013f
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, these studies will usually have all four features and receive an initial rating of “high confidence.””
§ Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments > Results > Step 5: Rate the Confidence in the Body of Evidence
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Therefore, it is worth noting that a single well-conducted study may provide evidence of toxicity or a health effect associated with exposure to the substance in question [e.g., see Germolec (2009) and Foster (2009) for explanations of the NTP levels of evidence for determination of “toxicity” for individual studies].”
§ Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments > Results > Step 5: Rate the Confidence in the Body of Evidence
What the sources say
- Not checked yetcites Germolec (2009)
- Not checked yetcites Foster (2009)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The principles of systematic review can be successfully applied to environmental health questions to provide greater objectivity and transparency to the process of developing conclusions.”
§ Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
7 more claims
Create a free account to see all 71 claims and to verify the 23 references still open.
- Probably not a substantial claim. 54
- It may state two separate claims as one. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The study design is sometimes made explicit in the key elements (e.g. 'PICO'-type questions may be stated as 'PICOD' or 'PICOS' in the scientific literature, where 'D' (design) or 'S' (study) indicates that study design is being considered) [11].”
From Eligibility screening in evidence synthesis of environmental management topics (2017) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“such methods have only recently been developed and utilized in environmental health sciences [23–30].”
From A Systematic Review and Meta-Analysis of Multiple Airborne Pollutants and Autism Spectrum Disorder (2016) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“To ensure reliability of the eligibility screening process, articles providing guidance on conducting systematic reviews in environmental research [2, 11, 12] and health research [14, 15, 18] recommend that eligibility screening should be performed where possible by at least two people.”
From Eligibility screening in evidence synthesis of environmental management topics (2017) · cites this paper as [11]
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.005 · Cited by 19,575
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,285
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974
Norderyd, O. M., Koch, G., Papias, A., Köhler, A. A., Helkimo, A. N., Brahm, C., Lindmark, U., Lindfors, N., Mattsson, A., Rolander, B., et al. (2015) · PubMed · Cited by 3,584
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
GRADE guidelines 6. Rating the quality of evidence—imprecision
Guyatt, G. H., Oxman, A. D., Kunz, R., Brozek, J., Alonso-Coello, P., Rind, D., Devereaux, P. J., Montori, V. M., Freyschuss, B., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.012 · Cited by 2,693 · cited 2×
“In brief, OHAT reviewed guidance from authoritative systematic-review groups (AHRQ 2013; Guyatt et al. 2011a; Higgins and Green 2011) in developing an initial draft and sought additional advice through web-based discussions and consultation with technical experts, the NTP Executive Committee, the…”
GRADE guidelines: 8. Rating the quality of evidence—indirectness
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Falck-Ytter, Y., Jaeschke, R., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.014 · Cited by 1,843
GRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898
Chapter 12: Interpreting results and drawing conclusions
Schünemann, H. J., Oxman, A. D., Jonathan, J., & Glasziou, P. (2011) · Cited by 870
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818
Authority, E. G. F. T. C. O. S. R. E. F. S. (2010) · EFSA Journal · doi:10.2903/j.efsa.2010.1637 · Cited by 693
Ahmed, I., Sutton, A. J., & Riley, R. D. (2012) · BMJ · doi:10.1136/bmj.d7762 · Cited by 404
The GRADE approach and Bradford Hill's criteria for causation
Schünemann, H., Hill, S., Guyatt, G., Akl, E. A., & Ahmed, F. (2011) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2010.119933 · Cited by 262
Woodruff, T. J., Sutton, P., & The Navigation Guide Work Group (2011) · Health Affairs · doi:10.1377/hlthaff.2010.1219 · Cited by 134
Whitlock, E. P., Lopez, S. A., Chang, S., Helfand, M., Eder, M. L., & Floyd, N. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.03.008 · Cited by 109
A survey of frameworks for best practices in weight-of-evidence analyses
Rhomberg, L. R., Goodman, J. E., Bailey, L. A., Prueitt, R. L., Beck, N. B., Bevan, C., Honeycutt, M., Kaminski, N. E., Paoli, G., Pottenger, L. H., et al. (2013) · Critical Reviews in Toxicology · doi:10.3109/10408444.2013.832727 · Cited by 96
Implementing Systematic Review at the National Toxicology Program: Status and Next Steps
Birnbaum, L. S., Thayer, K. A., Bucher, J. R., & Wolfe, M. S. (2013) · Environmental Health Perspectives · doi:10.1289/ehp.1306711 · Cited by 33
Self-citationCenter for the Evaluation of Risks to Human Reproduction?The First Five Years
Jahnke, G. D., Iannucci, A. R., Scialli, A. R., & Shelby, M. D. (2005) · Birth Defects Research Part B Developmental and Reproductive Toxicology · doi:10.1002/bdrb.20028 · Cited by 18
Bucher, J. R., Thayer, K. A., & Birnbaum, L. S. (2011) · Environmental Health Perspectives · doi:10.1289/ehp.1103645 · Cited by 14
Self-citationShelby, M. D. (2005) · Birth Defects Research Part B Developmental and Reproductive Toxicology · doi:10.1002/bdrb.20029 · Cited by 13
Medlin, J. (2003) · Environmental Health Perspectives · doi:10.1289/ehp.111-a696 · Cited by 2
New NTP Centers Meet the Need to Know
Twombly, R. (1998) · Environmental Health Perspectives · doi:10.1289/ehp.98106a480 · Cited by 2
Research works printed but not linked: 2
- CLARITY Group at McMaster University. 2013. Systematic and Literature Review Resources. Instruments.Available: http://www.evidencepartners.com/resources/ [accessed 15 January 2013].
- NRC (National Research Council). 2013b. Project Title: Review of the IRIS Process.Available: http://www8.nationalacademies.org/cp/projectview.aspx?key=49458 [accessed 1 November 2013].
Cited by
Rochester, J. R., & Bolden, A. L. (2015) · Environmental Health Perspectives · doi:10.1289/ehp.1408989 · Cited by 1,524
Fedak, K. M., Bernal, A., Capshaw, Z. A., & Gross, S. (2015) · Emerging Themes in Epidemiology · doi:10.1186/s12982-015-0037-4 · Cited by 680
Key Characteristics of Carcinogens as a Basis for Organizing Data on Mechanisms of Carcinogenesis
Smith, M. T., Guyton, K. Z., Gibbons, C. F., Fritz, J. M., Portier, C. J., Rusyn, I., DeMarini, D. M., Caldwell, J. C., Kavlock, R. J., Lambert, P. F., et al. (2016) · Environmental Health Perspectives · doi:10.1289/ehp.1509912 · Cited by 650
Self-citationLevy, K., Woster, A. P., Goldstein, R. S., & Carlton, E. J. (2016) · Environmental Science & Technology · doi:10.1021/acs.est.5b06186 · Cited by 535
New Look at BTEX: Are Ambient Levels a Problem?
Bolden, A. L., Kwiatkowski, C. F., & Colborn, T. (2015) · Environmental Science & Technology · doi:10.1021/es505316f · Cited by 456
Metadata
- Authors
- Rooney, A. A., Boyles, A. L., Wolfe, M. S., Bucher, J. R., & Thayer, K. A.
- Affiliations
- National Institutes of Health (US)
- Corresponding author
- Rooney, Andrew A.
- Venue
- Environmental Health Perspectives
- Publisher
- American Chemical Society (ACS)
- Published
- 2014-04-22
- Volume
- 122
- Issue
- 7
- Pages
- 711-718
- ISSN
- 0091-6765 (ISSN-L)
- 1552-9924
- PMC
- PMC4080517
- OpenAlex
- W2066218383
- Semantic Scholar
- f30c4b27530fb3fa34898432b9850e3fbcdbde0e
- Cited by
- 434
- References
- 31
- Citations by year
- 2026: 34 · 2025: 33 · 2024: 43 · 2023: 34 · 2022: 39 · 2021: 53 · 2020: 40 · 2019: 29 · 2018: 33 · 2017: 25 · 2016: 31 · 2015: 28 · 2014: 12
- FWCI
- 8.81
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/licenses/by/4.0/ (vor, 1347 d)
- Locations
- Environmental Health Perspectives (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Environmental Science › Health, Toxicology and Mutagenesis › Health, Environment, Cognitive Aging
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- systematic reviews, evidence integration, environmental health risk assessment, hazard identification, level of evidence, confidence in evidence, risk of bias assessment, mechanistic studies, animal studies, human subjects research
- MeSH terms
- Systematic Reviews as Topic, Animals, Environmental Health, Humans, United States, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
