Study sensitivity: Evaluating the ability to detect effects in systematic reviews of chemical exposures
Open accessAbstract
A critical step in systematic reviews of potential health hazards is the structured evaluation of the strengths and weaknesses of the included studies; risk of bias is a term often used to represent this process, specifically with respect to the evaluation of systematic errors that can lead to inaccurate (biased) results (i.e. focusing on internal validity). Systematic review methods developed in the clinical medicine arena have been adapted for use in evaluating environmental health hazards; this expansion raises questions about the scope of risk of bias tools and the extent to which they capture the elements that can affect the interpretation of results from environmental and occupational epidemiology studies and in vivo animal toxicology studies, (the studies typically available for assessment of risk of chemicals). One such element, described here as "sensitivity", is a measure of the ability of a study to detect a true effect or hazard. This concept is similar to the concept of the sensitivity of an assay; an insensitive study may fail to show a difference that truly exists, leading to a false conclusion of no effect. Factors relating to study sensitivity should be evaluated in a systematic manner with the same rigor as the evaluation of other elements within a risk of bias framework. We discuss the importance of this component for the interpretation of individual studies, examine approaches proposed or in use to address it, and describe how it relates to other evaluation components. The evaluation domains contained within a risk of bias tool can include, or can be modified to include, some features relating to study sensitivity; the explicit inclusion of these sensitivity criteria with the same rigor and at the same stage of study evaluation as other bias-related criteria can improve the evaluation process. In some cases, these and other features may be better addressed through a separate sensitivity domain. The combined evaluation of risk of bias and sensitivity can be used to identify the most informative studies, to evaluate the confidence of the findings from individual studies and to identify those study elements that may help to explain heterogeneity across the body of literature.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study population must be defined such that the exposure-outcome association can be validly studied.
“The study population must be defined such that the exposure–outcome association can be validly studied [23].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [23]
Cooper and colleagues compiled a list of questions relating to study sensitivity.
“Cooper and colleagues compiled a list of questions relating to study sensitivity [23].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · see citing works
SYRCLE’s risk of bias tool for animal studies
Hooijmans, C. R., Rovers, M. M., de Vries, R. B., Leenaars, M., Ritskes-Hoitinga, M., & Langendam, M. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-43 · Cited by 4,415 · see citing works
Rice, D., & Barone, S. (2000) · Environmental Health Perspectives · doi:10.1289/ehp.00108s3511 · Cited by 2,944 · see citing works
Semple, B. D., Blomgren, K., Gimlin, K., Ferriero, D. M., & Noble-Haeusslein, L. J. (2013) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2013.04.001 · Cited by 2,132 · see citing works
State-of-the-science of endocrine disrupting chemicals, 2012
Bergman, Å., Heindel, J., Jobling, S., Kidd, K., & Zoeller, R. T. (2012) · Toxicology Letters · doi:10.1016/j.toxlet.2012.03.020 · Cited by 1,388 · see citing works
Finding What Works in Health Care: Standards for Systematic Reviews
Eden, J., Levit, L. A., Berg, A. O., & Morton, S. C. (2013) · Cited by 943 · see citing works
Fu, R., Gartlehner, G., Grant, M., Shamliyan, T., Sedrakyan, A., Wilt, T. J., Griffith, L., Oremus, M., Raina, P., Ismaila, A., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.08.010 · Cited by 730 · see citing works
Arsenic, metals, fibres, and dusts.
(2012) · PubMed Central · Cited by 601 · see citing works
“ToxRTool”, a new tool to assess the reliability of toxicological data
Schneider, K., Schwarz, M., Burkholder, I., Kopp‐Schneider, A., Edler, L., Kinsner‐Ovaskainen, A., Härtung, T., & Hoffmann, S. (2009) · Toxicology Letters · doi:10.1016/j.toxlet.2009.05.013 · Cited by 469 · see citing works
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448 · see citing works
Rooney, A. A., Boyles, A. L., Wolfe, M. S., Bucher, J. R., & Thayer, K. A. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307972 · Cited by 434 · see citing works
Woodruff, T. J., & Sutton, P. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307175 · Cited by 427 · see citing works
Bergman, Å., Heindel, J. J., Jobling, S., Kidd, K. A., & Zoeller, R. T. (2013) · Cited by 426 · see citing works
Hemminki, K., Kyyrönen, P., & Lindbohm, M.-L. (1985) · Journal of Epidemiology & Community Health · doi:10.1136/jech.39.2.141 · Cited by 320 · see citing works
Spontaneous abortions in hospital staff engaged in sterilising instruments with chemical agents.
Hemminki, K., Mutanen, P., Saloniemi, I., Niemi, M.-L., & Vainio, H. (1982) · BMJ · doi:10.1136/bmj.285.6353.1461 · Cited by 141 · see citing works
Whaley, P., Halsall, C. J., Ågerstrand, M., Aiassa, E., Benford, D., Bilotta, G. S., Coggon, D., Collins, C. D., Dempsey, C., Duarte‐Davidson, R., et al. (2015) · Environment International · doi:10.1016/j.envint.2015.11.002 · Cited by 91 · see citing works
Self-citationMarsh, G. M., Youk, A. O., Stone, R. A., et al. (2001) · Journal of Occupational and Environmental Medicine · doi:10.1097/00043764-200109000-00004 · Cited by 70 · see citing works
Beronius, A., Molander, L., Rudén, C., & Hanberg, A. (2014) · Journal of Applied Toxicology · doi:10.1002/jat.2991 · Cited by 64 · see citing works
Molander, L., Ågerstrand, M., Beronius, A., Hanberg, A., & Rudén, C. (2014) · Human and Ecological Risk Assessment An International Journal · doi:10.1080/10807039.2014.928104 · Cited by 52 · see citing works
Self-citationReview of the Formaldehyde Assessment in the National Toxicology Program 12th Report on Carcinogens
National Research Council (2014) · National Academies Press eBooks · doi:10.17226/18948 · Cited by 35 · see citing works
Review of the Styrene Assessment in the National Toxicology Program 12th Report on Carcinogens
Board on Environmental Studies and Toxicology, & Studies, D. O. E. A. L. (2014) · National Academies Press eBooks · doi:10.17226/18725 · Cited by 20 · see citing works
Comparison Bias and Dilution Effect in Occupational Cohort Studies
Parodi, S., Gennaro, V., Ceppi, M., & Cocco, P. (2007) · International Journal of Occupational and Environmental Health · doi:10.1179/oeh.2007.13.2.143 · Cited by 15 · see citing works
Waskey, A. J. (2015) · The SAGE Encyclopedia of World Poverty · doi:10.4135/9781483345727.n586 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Higgins JPT Green S Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 2011 The Cochrane Collaboration Chapter 8: Assessing risk of bias in included studies. [updated March 2011] available from www.cochrane-handbook.org
- NRC (National Research Council) Assessing the Human Health Risks of Trichloroethylene: Key Scientific Issues, National Resource Council 2006 379 National Academies Press Washington, D.C.
- NTP (National Toxicology Program) Report on Carcinogens Monograph on Trichloroethyelne – January 2015 2015a Office of the Report on Carcinogens RTP, NC Available: http://ntp.niehs.nih.gov/go/37899 34941221
Cited by
Sorted by citations
Dekkers, O. M., Vandenbroucke, J. P., Cevallos, M., Renehan, A. G., Altman, D. G., & Egger, M. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002742 · Cited by 525 · see citing works
Morgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Guyatt, G., & Schünemann, H. J. (2018) · Environment International · doi:10.1016/j.envint.2018.08.018 · Cited by 106 · see citing works
- Authors
- Cooper, G. S., Lunn, R. M., Ågerstrand, M., Glenn, B. S., Kraft, A. D., Luke, A. M., & Ratcliffe, J. M.
- Affiliations
- Environmental Protection Agency (US)
- National Institutes of Health (US)
- Stockholm University (SE)
- Integrated Laboratory Systems, Inc. (US)
- Corresponding author
- Cooper, Glinda S.
- Venue
- Environment International
- Publisher
- Elsevier BV
- Published
- 2016-07
- Volume
- 92-93
- Pages
- 605-610
- ISSN
- 0160-4120 (ISSN-L)
- PMC
- PMC5110036
- OpenAlex
- W2346453203
- Cited by
- 33
- References
- 23
- Citations by year
- 2026: 5 · 2025: 3 · 2024: 1 · 2023: 2 · 2022: 3 · 2021: 2 · 2020: 6 · 2019: 5 · 2018: 4 · 2016: 2
- FWCI
- 1.29
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Environment International (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Institutes of Health (Z99-ES999999)
- National Institute of Environmental Health Sciences
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Environmental Science › Health, Toxicology and Mutagenesis › Health, Environment, Cognitive Aging
- Keywords
- risk of bias, systematic reviews, chemical exposure, environmental health hazards, internal validity, study power, sensitivity analysis, risk of bias tools, occupational epidemiology, heterogeneity
- MeSH terms
- Animals, Environmental Exposure, Environmental Pollutants, Humans, Review Literature as Topic, Bias, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
