How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Environmental health hazard assessments are routinely relied upon for public health decision-making. The evidence base used in these assessments is typically developed from a collection of diverse sources of information of varying quality. It is critical that literature-based evaluations consider the credibility of individual studies used to reach conclusions through consistent, transparent and accepted methods. Systematic review procedures address study credibility by assessing internal validity or “risk of bias” — the assessment of whether the design and conduct of a study compromised the credibility of the link between exposure/intervention and outcome. This paper describes the commonalities and differences in risk-of-bias methods developed or used by five groups that conduct or provide methodological input for performing environmental health hazard assessments: the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group, the Navigation Guide, the National Toxicology Program’s (NTP) Office of Health Assessment and Translation (OHAT) and Office of the Report on Carcinogens (ORoC), and the Integrated Risk Information System of the U.S. Environmental Protection Agency (EPA-IRIS). Each of these groups have been developing and applying rigorous assessment methods for integrating across a heterogeneous collection of human and animal studies to inform conclusions on potential environmental health hazards. There is substantial consistency across the groups in the consideration of risk-of-bias issues or “domains” for assessing observational human studies. There is a similar overlap in terms of domains addressed for animal studies; however, the groups differ in the relative emphasis placed on different aspects of risk of bias. Future directions for the continued harmonization and improvement of these methods are also discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Self-citationGRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
GRADE guidelines: 7. Rating the quality of evidence—inconsistency
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Glasziou, P., Jaeschke, R., Akl, E. A., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.03.017 · Cited by 2,320
Self-citationPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147 · cited 2×
“…ability of funding source to influence study outcome, from studies on health impacts of tobacco (Barnes and Bero, 1997, 1998), to safety and efficacy of pharmaceuticals (Bero et al., 2007; Lexchin et al., 2003; Lundh et al., 2012; Perlis et al., 2005), and medical procedures (Popelut et al., 2010).”
GRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819
Self-citationIndustry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Atkins, D., Eccles, M., Flottorp, S., Guyatt, G. H., Henry, D., Hill, S., Liberati, A., O'Connell, D., Oxman, A. D., Phillips, B., et al. (2004) · BMC Health Services Research · doi:10.1186/1472-6963-4-38 · Cited by 1,235 · cited 4×
“1) (Atkins et al., 2004; Balshem et al., 2011).”
InfluentialSelf-citationAuthority, E. G. F. T. C. O. S. R. E. F. S. (2010) · EFSA Journal · doi:10.2903/j.efsa.2010.1637 · Cited by 693
Industry sponsorship and research outcome
Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O. A., & Bero, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub2 · Cited by 605
Rooney, A. A., Boyles, A. L., Wolfe, M. S., Bucher, J. R., & Thayer, K. A. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307972 · Cited by 434
Self-citation
Cited by
Bero, L., Chartres, N., Diong, J., Fabbri, A., Ghersi, D., Lam, J., Lau, A., McDonald, S., Mintzes, B., Sutton, P., et al. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0915-2 · Cited by 278
Self-citationMorgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Ansari, M., Akl, E. A., et al. (2019) · Environment International · doi:10.1016/j.envint.2018.11.004 · Cited by 248
“labelling of study designs, and the use of signaling questions [4, 5].”
BackgroundMethodInfluentialSelf-citationMorgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Guyatt, G., & Schünemann, H. J. (2018) · Environment International · doi:10.1016/j.envint.2018.08.018 · Cited by 106
Self-citationWhaley, P., Aiassa, E., Beausoleil, C., Beronius, A., Bilotta, G., Boobis, A., de Vries, R., Hanberg, A., Hoffmann, S., Hunt, N., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105926 · Cited by 91
Self-citationEick, S. M., Goin, D. E., Chartres, N., Lam, J., & Woodruff, T. J. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01490-8 · Cited by 82
Self-citation
Metadata
- Authors
- Rooney, A. A., Cooper, G. S., Jahnke, G. D., Lam, J., Morgan, R. L., Boyles, A. L., Ratcliffe, J. M., Kraft, A. D., Schünemann, H. J., Schwingl, P., Walker, T. D., Thayer, K. A., & Lunn, R. M.
- Venue
- Environment International
- Publisher
- Elsevier BV
- Published
- 2016-07
- Volume
- 92-93
- Pages
- 617-629
- ISSN
- 0160-4120 (ISSN-L)
- OpenAlex
- W2255858454
- Cited by
- 95
- References
- 19
- FWCI
- 12.22
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. https://app.cochat.ai/papers/10.1016/j.envint.2016.01.005
- The paper
- Rooney, A. A., Cooper, G. S., Jahnke, G. D., Lam, J., Morgan, R. L., Boyles, A. L., Ratcliffe, J. M., Kraft, A. D., Schünemann, H. J., Schwingl, P., Walker, T. D., Thayer, K. A., & Lunn, R. M. (2016). How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environment International. https://doi.org/10.1016/j.envint.2016.01.005
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.