Negative Controls: A Tool for Detecting Confounding and Bias in Observational Studies
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
Noncausal associations between exposures and outcomes are a threat to validity of causal inference in observational studies. Many techniques have been developed for study design and analysis to identify and eliminate such errors. Such problems are not expected to compromise experimental studies, where careful standardization of conditions (for laboratory work) and randomization (for population studies) should, if applied properly, eliminate most such noncausal associations. We argue, however, that a routine precaution taken in the design of biologic laboratory experiments--the use of "negative controls"--is designed to detect both suspected and unsuspected sources of spurious causal inference. In epidemiology, analogous negative controls help to identify and resolve confounding as well as other sources of error, including recall bias or analytic flaws. We distinguish 2 types of negative controls (exposure controls and outcome controls), describe examples of each type from the epidemiologic literature, and identify the conditions for the use of such negative controls to detect confounding. We conclude that negative controls should be more commonly employed in observational studies, and that additional work is needed to specify the conditions under which negative controls will be sensitive detectors of other sources of error in observational studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, in order to evaluate the specificity of the association to ADHD medication, the tenth used selective serotonin reuptake inhibitors (SSRIs) as a negative control medication exposure (27, 30).”
From ADHD Medication and Substance-Related Problems (2017) · cites this paper as [30]
In the citing paper
“The use of ‘negative controls’ for outcomes or exposures can be helpful to assess the likely presence of unmeasured or residual confounding in observational research [51].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [51]
In the citing paper
“In a third approach–negative control studies–we would expect an unbiased negative control study to be null and the ‘real’ study not-null if there is a true causal effect and the assumptions of this approach are met (see Table 1).39,40”
From Triangulation in aetiological epidemiology (2017) · cites this paper as [39]
Citations
Cites
(1979) · Medical Entomology and Zoology · Cited by 11,252
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,285
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · cited 1×
“19 Hill argued that causal inferences were more credible if the exposure (in his example, nickel mining) was associated with only certain types of outcomes (death from lung and nose cancer but not death from other cancers), and if the outcome was associated with one kind of exposure (nickel mining)…”
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844
An introduction to instrumental variables for epidemiologists
Greenland, S. (2000) · International Journal of Epidemiology · doi:10.1093/ije/29.4.722 · Cited by 1,156
Instruments for Causal Inference: An Epidemiologist's Dream?
Hernán, M. A., & Robins, J. M. (2006) · Epidemiology · doi:10.1097/01.ede.0000222409.00878.37 · Cited by 961
Vaccines for preventing influenza in the elderly
Demicheli, V., Jefferson, T., Di Pietrantonj, C., Ferroni, E., Thorning, S., Thomas, R. E., & Rivetti, A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004876.pub4 · Cited by 695
Impact of Influenza Vaccination on Seasonal Mortality in the US Elderly Population
(2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.3.265 · Cited by 543 · cited 1×
“8 To test this hypothesis, Jackson et al 7 reproduced earlier estimates of the protective effect of influenza vaccination, but then repeated the analysis for two sets of negative control outcomes, and showed that the protective effect was observed even in circumstances where the vaccine could not…”
Mortality benefits of influenza vaccination in elderly people: an ongoing controversy
Simonsen, L., Taylor, R. J., Viboud, C., Miller, M. A., & Jackson, L. A. (2007) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(07)70236-0 · Cited by 520 · cited 1×
“However, older age is associated with a less robust immune response to influenza vaccination, and ecological data suggest that the benefits measured in observational studies far exceed the corresponding benefits expected at the population level when influenza vaccination rates have increased among the elderly.(6) Impo…”
Interleukin-17A Mediates Acquired Immunity to Pneumococcal Colonization
Lu, Y.-J., Gross, J., Bogaert, D., Finn, A., Bagrade, L., Zhang, Q., Kolls, J. K., Srivastava, A., Lundgren, A., Forte, S., et al. (2008) · PLoS Pathogens · doi:10.1371/journal.ppat.1000159 · Cited by 493
Self-citationEvidence of bias in estimates of influenza vaccine effectiveness in seniors
Jackson, L. A., Jackson, M. L., Nelson, J. C., Neuzil, K. M., & Weiss, N. S. (2006) · International Journal of Epidemiology · doi:10.1093/ije/dyi274 · Cited by 493 · cited 5×
“8 To test this hypothesis, Jackson et al 7 reproduced earlier estimates of the protective effect of influenza vaccination, but then repeated the analysis for two sets of negative control outcomes, and showed that the protective effect was observed even in circumstances where the vaccine could not have caused the prote…”
InfluentialInstrumental Variables: Application and Limitations
Martens, E. P., Pestman, W. R., de Boer, A., Belitser, S. V., & Klungel, O. H. (2006) · Epidemiology · doi:10.1097/01.ede.0000215160.88317.cb · Cited by 464 · cited 1×
“11 , 12 13 – 15 An instrumental variable is any variable that is connected causally to A but free of any of the confounding connections to Y from which A suffers.”
Brookhart, M. A., Wang, P. S., Solomon, D. H., & Schneeweiss, S. (2006) · Epidemiology · doi:10.1097/01.ede.0000193606.58671.c5 · Cited by 341 · cited 1×
“11 , 12 13 – 15 An instrumental variable is any variable that is connected causally to A but free of any of the confounding connections to Y from which A suffers.”
Essentials of Epidemiology in Public Health
Aschengrau, A., & Seage, G. R. (2013) · Cited by 307
Essentials of Epidemiology in Public Health
Stadtlander, C. T. K.-H. (2022) · Epidemiology, Biostatistics, and Public Health · doi:10.2427/12261 · Cited by 212
Vaccines for preventing influenza in the elderly
Rivetti, D., Jefferson, T., Thomas, R. E., Rudin, M., Rivetti, A., Di Pietrantonj, C., & Demicheli, V. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004876.pub2 · Cited by 192
Harris, A. D., Samore, M. H., Lipsitch, M., Kaye, K. S., Perencevich, E., & Carmeli, Y. (2002) · Clinical Infectious Diseases · doi:10.1086/340533 · Cited by 171 · cited 1×
“This difficulty is reduced if multiple control groups are used, as is occasionally done for other reasons.(11,12) A useful contrast can be drawn between variables that can serve as negative controls and those that can be used as instruments.”
Self-citationIntroduction to Regression Modeling
Abraham, B., & Ledolter, J. (2005) · WU Research · Cited by 168
Causation and causal inference
Hoggatt, K. J., Greenland, S., & VanderWeele, T. J. (2015) · doi:10.1093/med/9780199661756.003.0117 · Cited by 98
Weiss, N. S. (2002) · Epidemiology · doi:10.1097/00001648-200201000-00003 · Cited by 91
Bell, C. M., Bajcar, J., Bierman, A. S., Li, P., Mamdani, M. M., & Urbach, D. R. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.22.2525 · Cited by 70
Selective association of multiple sclerosis with infectious mononucleosis
Zaadstra, B., Chorus, A., van Buuren, S., Kalsbeek, H., & van Noort, J. (2008) · Multiple Sclerosis Journal · doi:10.1177/1352458507084265 · Cited by 41 · cited 1×
“A recent study 9 tested the association between multiple sclerosis (MS) and a variety of common childhood infections assessed by self-report.”
Bias in observational study of the effectiveness of nasal corticosteroids in asthma
Suissa, S., & Ernst, P. (2005) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2004.12.1118 · Cited by 39 · cited 2×
“Inclusion of the “immortal” time is dependent on both being exposed during that time and on not having the outcome during that time 10 ; hence a (negative) association is induced between exposure and outcome.”
Cited by
Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available: Table 1.
Hernán, M. A., & Robins, J. M. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kwv254 · Cited by 3,115
“A second approach is to consider outcome controls (18) for which no causal effect is expected, for example, brain cancer or pneumonia in our hormone therapy example.”
BackgroundBNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting
Dagan, N., Barda, N., Kepten, E., Miron, O., Perchik, S., Katz, M. A., Hernán, M. A., Lipsitch, M., Reis, B., & Balicer, R. D. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2101765 · Cited by 2,527
Self-citationLong-term cardiovascular outcomes of COVID-19
Xie, Y., Xu, E., Bowe, B., & Al-Aly, Z. (2022) · Nature Medicine · doi:10.1038/s41591-022-01689-3 · Cited by 2,029
Bjornevik, K., Cortese, M., Healy, B. C., Kuhle, J., Mina, M. J., Leng, Y., Elledge, S. J., Niebuhr, D. W., Scher, A. I., Munger, K. L., et al. (2022) · Science · doi:10.1126/science.abj8222 · Cited by 2,049
High-dimensional characterization of post-acute sequelae of COVID-19
Al-Aly, Z., Xie, Y., & Bowe, B. (2021) · Nature · doi:10.1038/s41586-021-03553-9 · Cited by 1,518
Metadata
- Authors
- Lipsitch, M., Tchetgen Tchetgen, E., & Cohen, T.
- Affiliations
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Lipsitch, Marc
- Venue
- Epidemiology
- Published
- 2010-05
- Volume
- 21
- Issue
- 3
- Pages
- 383-388
- ISSN
- 1044-3983 (ISSN-L)
- PMC
- PMC3053408
- OpenAlex
- W2009428399
- Cited by
- 1,583
- References
- 25
- Citations by year
- 2026: 278 · 2025: 215 · 2024: 183 · 2023: 138 · 2022: 153 · 2021: 132 · 2020: 107 · 2019: 75 · 2018: 72 · 2017: 71 · 2016: 56 · 2015: 32 · 2014: 26 · 2013: 25 · 2012: 11
- FWCI
- 11.64
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Funders
- National Institutes of Health (5u01gm076497-03, 1dp2od006663-01)
- National Institute of General Medical Sciences (U54GM088558, U01GM076497, 5U01GM076497)
- National Institute of Environmental Health Sciences (T32ES007142)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- negative control, confounding, spurious association, exposure control, outcome control, observational study validity, recall bias, experimental design, causal inference
- MeSH terms
- Humans, Bias, Causality, Confounding Factors, Epidemiologic, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.