Negative Controls: A Tool for Detecting Confounding and Bias in Observational Studies
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
Noncausal associations between exposures and outcomes are a threat to validity of causal inference in observational studies. Many techniques have been developed for study design and analysis to identify and eliminate such errors. Such problems are not expected to compromise experimental studies, where careful standardization of conditions (for laboratory work) and randomization (for population studies) should, if applied properly, eliminate most such noncausal associations. We argue, however, that a routine precaution taken in the design of biologic laboratory experiments--the use of "negative controls"--is designed to detect both suspected and unsuspected sources of spurious causal inference. In epidemiology, analogous negative controls help to identify and resolve confounding as well as other sources of error, including recall bias or analytic flaws. We distinguish 2 types of negative controls (exposure controls and outcome controls), describe examples of each type from the epidemiologic literature, and identify the conditions for the use of such negative controls to detect confounding. We conclude that negative controls should be more commonly employed in observational studies, and that additional work is needed to specify the conditions under which negative controls will be sensitive detectors of other sources of error in observational studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, in order to evaluate the specificity of the association to ADHD medication, the tenth used selective serotonin reuptake inhibitors (SSRIs) as a negative control medication exposure (27, 30).”
From ADHD Medication and Substance-Related Problems (2017) · cites this paper as [30]
In the citing paper
“The use of ‘negative controls’ for outcomes or exposures can be helpful to assess the likely presence of unmeasured or residual confounding in observational research [51].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [51]
In the citing paper
“In a third approach–negative control studies–we would expect an unbiased negative control study to be null and the ‘real’ study not-null if there is a true causal effect and the assumptions of this approach are met (see Table 1).39,40”
From Triangulation in aetiological epidemiology (2017) · cites this paper as [39]
Citations
Cites
(1979) · Medical Entomology and Zoology · Cited by 11,252
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,285
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · cited 1×
“19 Hill argued that causal inferences were more credible if the exposure (in his example, nickel mining) was associated with only certain types of outcomes (death from lung and nose cancer but not death from other cancers), and if the outcome was associated with one kind of exposure (nickel mining)…”
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844
An introduction to instrumental variables for epidemiologists
Greenland, S. (2000) · International Journal of Epidemiology · doi:10.1093/ije/29.4.722 · Cited by 1,156
Instruments for Causal Inference: An Epidemiologist's Dream?
Hernán, M. A., & Robins, J. M. (2006) · Epidemiology · doi:10.1097/01.ede.0000222409.00878.37 · Cited by 961
Vaccines for preventing influenza in the elderly
Demicheli, V., Jefferson, T., Di Pietrantonj, C., Ferroni, E., Thorning, S., Thomas, R. E., & Rivetti, A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004876.pub4 · Cited by 695
Impact of Influenza Vaccination on Seasonal Mortality in the US Elderly Population
(2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.3.265 · Cited by 543 · cited 1×
“8 To test this hypothesis, Jackson et al 7 reproduced earlier estimates of the protective effect of influenza vaccination, but then repeated the analysis for two sets of negative control outcomes, and showed that the protective effect was observed even in circumstances where the vaccine could not…”
Mortality benefits of influenza vaccination in elderly people: an ongoing controversy
Simonsen, L., Taylor, R. J., Viboud, C., Miller, M. A., & Jackson, L. A. (2007) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(07)70236-0 · Cited by 520 · cited 1×
“However, older age is associated with a less robust immune response to influenza vaccination, and ecological data suggest that the benefits measured in observational studies far exceed the corresponding benefits expected at the population level when influenza vaccination rates have increased among the elderly.(6) Impo…”
Cited by
Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available: Table 1.
Hernán, M. A., & Robins, J. M. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kwv254 · Cited by 3,115
“A second approach is to consider outcome controls (18) for which no causal effect is expected, for example, brain cancer or pneumonia in our hormone therapy example.”
BackgroundBNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting
Dagan, N., Barda, N., Kepten, E., Miron, O., Perchik, S., Katz, M. A., Hernán, M. A., Lipsitch, M., Reis, B., & Balicer, R. D. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2101765 · Cited by 2,527
Self-citationLong-term cardiovascular outcomes of COVID-19
Xie, Y., Xu, E., Bowe, B., & Al-Aly, Z. (2022) · Nature Medicine · doi:10.1038/s41591-022-01689-3 · Cited by 2,029
Bjornevik, K., Cortese, M., Healy, B. C., Kuhle, J., Mina, M. J., Leng, Y., Elledge, S. J., Niebuhr, D. W., Scher, A. I., Munger, K. L., et al. (2022) · Science · doi:10.1126/science.abj8222 · Cited by 2,049
High-dimensional characterization of post-acute sequelae of COVID-19
Al-Aly, Z., Xie, Y., & Bowe, B. (2021) · Nature · doi:10.1038/s41586-021-03553-9 · Cited by 1,518
Guidelines for performing Mendelian randomization investigations
Burgess, S., Davey Smith, G., Davies, N. M., Dudbridge, F., Gill, D., Glymour, M. M., Hartwig, F. P., Holmes, M. V., Minelli, C., Relton, C. L., et al. (2020) · Wellcome Open Research · doi:10.12688/wellcomeopenres.15555.2 · Cited by 1,273
Triangulation in aetiological epidemiology
Lawlor, D. A., Tilling, K., & Davey Smith, G. (2017) · International Journal of Epidemiology · doi:10.1093/ije/dyw314 · Cited by 1,129
“In a third approach–negative control studies–we would expect an unbiased negative control study to be null and the ‘real’ study not-null if there is a true causal effect and the assumptions of this approach are met (see Table 1).(39,40) In many negative control studies there will not be a simple dichotomy in which the…”
BackgroundLong COVID after breakthrough SARS-CoV-2 infection
Al-Aly, Z., Bowe, B., & Xie, Y. (2022) · Nature Medicine · doi:10.1038/s41591-022-01840-0 · Cited by 866
Mendelian Randomization: Concepts and Scope
Richmond, R. C., & Davey Smith, G. (2022) · Cold Spring Harbor Perspectives in Medicine · doi:10.1101/cshperspect.a040501 · Cited by 841
Collider bias undermines our understanding of COVID-19 disease risk and severity
Griffith, G. J., Morris, T. T., Tudball, M. J., Herbert, A., Mancano, G., Pike, L., Sharp, G. C., Sterne, J., Palmer, T. M., Davey Smith, G., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-19478-2 · Cited by 826
Guidelines for performing Mendelian randomization investigations: update for summer 2023
Burgess, S., Davey Smith, G., Davies, N. M., Dudbridge, F., Gill, D., Glymour, M. M., Hartwig, F. P., Kutalik, Z., Holmes, M. V., Minelli, C., et al. (2023) · Wellcome Open Research · doi:10.12688/wellcomeopenres.15555.3 · Cited by 721
Acute and postacute sequelae associated with SARS-CoV-2 reinfection
Bowe, B., Xie, Y., & Al-Aly, Z. (2022) · Nature Medicine · doi:10.1038/s41591-022-02051-3 · Cited by 639
Patone, M., Mei, X. W., Handunnetthi, L., Dixon, S., Zaccardi, F., Shankar-Hari, M., Watkinson, P., Khunti, K., Harnden, A., Coupland, C. A. C., et al. (2022) · Nature Medicine · doi:10.1038/s41591-021-01630-0 · Cited by 637
Exposure to Greenness and Mortality in a Nationwide Prospective Cohort Study of Women
James, P., Hart, J. E., Banay, R. F., & Laden, F. (2016) · Environmental Health Perspectives · doi:10.1289/ehp.1510363 · Cited by 591
“Other deaths [including accidental (ICD-9 codes E800– E999)] were included as negative control outcomes to detect potential confounding bias (Lipsitch et al. 2010).”
BackgroundInfluentialRisks and burdens of incident diabetes in long COVID: a cohort study
Xie, Y., & Al-Aly, Z. (2022) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(22)00044-4 · Cited by 564
“36,37 Robust sandwich estimators were used to estimate variances when weightings were applied.”
MethodLong-term neurologic outcomes of COVID-19
Xu, E., Xie, Y., & Al-Aly, Z. (2022) · Nature Medicine · doi:10.1038/s41591-022-02001-z · Cited by 530
Dekkers, O. M., Vandenbroucke, J. P., Cevallos, M., Renehan, A. G., Altman, D. G., & Egger, M. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002742 · Cited by 525
Patil, S. R., Arnold, B. F., Salvatore, A. L., Briceno, B., Ganguly, S., Colford, J. M., & Gertler, P. J. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001709 · Cited by 487
“We also measured bruising/abrasions and itchy skin/scalp to serve as negative control outcomes [30] to check for differential reporting bias in this unblinded trial [31,32].”
MethodRoshanov, P. S., Rochwerg, B., Patel, A., Salehian, O., Duceppe, E., Belley-Côté, E. P., Guyatt, G. H., Sessler, D. I., Le Manach, Y., Borges, F. K., et al. (2017) · Anesthesiology · doi:10.1097/aln.0000000000001404 · Cited by 423
Risk of clinical sequelae after the acute phase of SARS-CoV-2 infection: retrospective cohort study
Daugherty, S. E., Guo, Y., Heath, K., Dasmariñas, M. C., Jubilo, K. G., Samranvedhya, J., Lipsitch, M., & Cohen, K. (2021) · BMJ · doi:10.1136/bmj.n1098 · Cited by 435
Self-citationWang, S. V., Schneeweiss, S., Initiative, R.-D., Franklin, J. M., Desai, R. J., Feldman, W., Garry, E. M., Glynn, R. J., Lin, K. J., Paik, J., et al. (2023) · JAMA · doi:10.1001/jama.2023.4221 · Cited by 428
Pickering, A. J., Djebbari, H., Lopez, C., Coulibaly, M., & Alzua, M. L. (2015) · The Lancet Global Health · doi:10.1016/s2214-109x(15)00144-8 · Cited by 391
Maciejewski, M. L. (2020) · Biostatistics & Epidemiology · doi:10.1080/24709360.2018.1477468 · Cited by 259
Exposure to ambient air pollution and the incidence of dementia: a population-based cohort study
Chen, H., Kwong, J. C., Copes, R., Hystad, P., van Donkelaar, A., Tu, K., Brook, J. R., Goldberg, M. S., Martin, R. V., Murray, B. J., et al. (2017) · Environment International · doi:10.1016/j.envint.2017.08.020 · Cited by 394
Neurological complications after first dose of COVID-19 vaccines and SARS-CoV-2 infection
Patone, M., Handunnetthi, L., Saatci, D., Pan, J., Katikireddi, S. V., Razvi, S., Hunt, D., Mei, X. W., Dixon, S., Zaccardi, F., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01556-7 · Cited by 394
Hippisley-Cox, J., Patone, M., Mei, X. W., Saatci, D., Dixon, S., Khunti, K., Zaccardi, F., Watkinson, P., Shankar-Hari, M., Doidge, J., et al. (2021) · BMJ · doi:10.1136/bmj.n1931 · Cited by 371
The 2016 global and national burden of diabetes mellitus attributable to PM 2·5 air pollution
Bowe, B., Xie, Y., Li, T., Yan, Y., Xian, H., & Al-Aly, Z. (2018) · The Lancet Planetary Health · doi:10.1016/s2542-5196(18)30140-2 · Cited by 371
Raz, R., Roberts, A. L., Lyall, K., Hart, J. E., Just, A. C., Laden, F., & Weisskopf, M. G. (2015) · Environmental Health Perspectives · doi:10.1289/ehp.1408133 · Cited by 334
“In this way, PM 2.5 exposure before and after pregnancy (because no association is seen with them in mutually adjusted models) acts as a negative control (Flanders et al. 2011; Lipsitch et al. 2010) and rules out confounding by many— even unmeasured—potential confounders.”
BackgroundInfluentialBurden of Cause-Specific Mortality Associated With PM 2.5 Air Pollution in the United States
Bowe, B., Xie, Y., Yan, Y., & Al-Aly, Z. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.15834 · Cited by 367
Particulate Matter Air Pollution and the Risk of Incident CKD and Progression to ESRD
Bowe, B., Xie, Y., Li, T., Yan, Y., Xian, H., & Al-Aly, Z. (2018) · Journal of the American Society of Nephrology · doi:10.1681/asn.2017030253 · Cited by 354
Causality and causal inference in epidemiology: the need for a pluralistic approach
Vandenbroucke, J. P., Broadbent, A., & Pearce, N. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv341 · Cited by 327
“The two main types of negative controls are: exposure controls and outcome controls.(43) An interesting example was the finding of an influence of smoking habits on pregnancy outcomes.”
BackgroundPostacute sequelae of COVID-19 at 2 years
Bowe, B., Xie, Y., & Al-Aly, Z. (2023) · Nature Medicine · doi:10.1038/s41591-023-02521-2 · Cited by 350
Bowe, B., Xie, Y., Xu, E., & Al-Aly, Z. (2021) · Journal of the American Society of Nephrology · doi:10.1681/asn.2021060734 · Cited by 333
“To examine the robustness of results to study-design specifications, we examined a set of positive and negative outcome controls.(25) We examined the association of COVID-19 with positive outcome controls of all-cause mortality and hospitalization, where, on the basis of prior www.”
MethodRisks of mental health outcomes in people with covid-19: cohort study
Xie, Y., Xu, E., & Al-Aly, Z. (2022) · BMJ · doi:10.1136/bmj-2021-068993 · Cited by 323
“21 The successful application of both positive and negative controls might lessen concern about the presence of spurious biases related to the cohort construction, study design, covariate selection, analytic approach, outcome ascertainment, residual confounding, and other sources of latent biases.”
BackgroundOutcome-Wide Longitudinal Designs for Causal Inference: A New Template for Empirical Studies
VanderWeele, T. J., Mathur, M. B., & Chen, Y. (2020) · Statistical Science · doi:10.1214/19-sts728 · Cited by 310
“positive results obtained are not due solely to unmeasured confounding, if some of the outcomes might plausibly serve as negative controls (Lipsitch et al., 2010).”
BackgroundTransparent Reporting of Observational Studies Emulating a Target Trial—The TARGET Statement
Cashin, A. G., Hansford, H. J., Hernán, M. A., Swanson, S. A., Lee, H., Jones, M. D., Dahabreh, I. J., Dickerman, B. A., Egger, M., Garcia-Albeniz, X., et al. (2025) · JAMA · doi:10.1001/jama.2025.13350 · Cited by 260
Big Data in Public Health: Terminology, Machine Learning, and Privacy
Mooney, S. J., & Pejaver, V. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014208 · Cited by 325
“Second, wider datasets with more measured covariates offer opportunities to use negative controls (76) more extensively to estimate the potential magnitude of residual confounding, measurement error, or selection bias.”
BackgroundMethodXie, Y., Bowe, B., & Al-Aly, Z. (2021) · Nature Communications · doi:10.1038/s41467-021-26513-3 · Cited by 301
Burgess, S., Foley, C. N., & Zuber, V. (2018) · Annual Review of Genomics and Human Genetics · doi:10.1146/annurev-genom-083117-021731 · Cited by 295
“Alternative ways of assessing the IV assumptions are (a) considering positive and negative control variables (20, 72) (for example, CRP and interleukin-6 may be considered positive controls for intervention on interleukin-1); (b) subsetting the population, particularly if there are subsets in the population that may h…”
BackgroundPostacute Sequelae of SARS-CoV-2 Infection in the Pre-Delta, Delta, and Omicron Eras
Xie, Y., Choi, T., & Al-Aly, Z. (2024) · New England Journal of Medicine · doi:10.1056/nejmoa2403211 · Cited by 267
Wei, Y., Wang, Y., Di, Q., Choirat, C., Wang, Y., Koutrakis, P., Zanobetti, A., Dominici, F., & Schwartz, J. D. (2019) · BMJ · doi:10.1136/bmj.l6258 · Cited by 279
Fourth Dose of BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting
Magen, O., Waxman, J. G., Makov-Assif, M., Vered, R., Dicker, D., Hernán, M. A., Lipsitch, M., Reis, B. Y., Balicer, R. D., & Dagan, N. (2022) · New England Journal of Medicine · doi:10.1056/nejmoa2201688 · Cited by 279
Self-citationThree-year outcomes of post-acute sequelae of COVID-19
Cai, M., Xie, Y., Topol, E. J., & Al-Aly, Z. (2024) · Nature Medicine · doi:10.1038/s41591-024-02987-8 · Cited by 254
Identifying causal effects with proxy variables of an unmeasured confounder
Miao, W., Geng, Z., & Tchetgen Tchetgen, E. J. (2018) · Biometrika · doi:10.1093/biomet/asy038 · Cited by 231
Placebo Tests for Causal Inference
Eggers, A. C., Tuñón, G., & Dafoe, A. (2024) · American Journal of Political Science · doi:10.1111/ajps.12818 · Cited by 219
Causal inference with observational data: the need for triangulation of evidence
Hammerton, G., & Munafò, M. R. (2021) · Psychological Medicine · doi:10.1017/s0033291720005127 · Cited by 262
“Lipsitch and colleagues (Lipsitch, Tchetgen Tchetgen, & Cohen, 2010) described their use as a means whereby we can ‘ detect both suspected and unsuspected sources of spurious causal inference ’ .”
BackgroundAssociation Between Nursing Home Crowding and COVID-19 Infection and Mortality in Ontario, Canada
Brown, K. A., Jones, A., Daneman, N., Chan, A. K., Schwartz, K. L., Garber, G. E., Costa, A. P., & Stall, N. M. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2020.6466 · Cited by 245
Epstein–Barr virus as a leading cause of multiple sclerosis: mechanisms and implications
Bjornevik, K., Münz, C., Cohen, J. I., & Ascherio, A. (2023) · Nature Reviews Neurology · doi:10.1038/s41582-023-00775-5 · Cited by 246
Comparative Effectiveness of BNT162b2 and mRNA-1273 Vaccines in U.S. Veterans
Dickerman, B. A., Gerlovin, H., Madenci, A. L., Kurgansky, K. E., Ferolito, B. R., Figueroa Muñiz, M. J., Gagnon, D. R., Gaziano, J. M., Cho, K., Casas, J. P., et al. (2022) · New England Journal of Medicine · doi:10.1056/nejmoa2115463 · Cited by 249
Does water kill? A call for less casual causal inferences
Hernán, M. A. (2016) · Annals of Epidemiology · doi:10.1016/j.annepidem.2016.08.016 · Cited by 219
Causal Inference About the Effects of Interventions From Observational Studies in Medical Journals
Dahabreh, I. J., & Bibbins-Domingo, K. (2024) · JAMA · doi:10.1001/jama.2024.7741 · Cited by 242
An Introduction to Proximal Causal Inference
Tchetgen, E. J. T., Ying, A., Cui, Y., Shi, X., & Miao, W. (2024) · Statistical Science · doi:10.1214/23-sts911 · Cited by 317
Interpreting observational studies: why empirical calibration is needed to correct p‐values
Schuemie, M. J., Ryan, P. B., DuMouchel, W., Suchard, M. A., & Madigan, D. (2014) · Statistics in Medicine · doi:10.1002/sim.5925 · Cited by 234
“For our negative controls, we could either pick different drugs that are known not to cause the outcome of interest or pick outcomes that are known not to be caused by our drug of interest [17].”
BackgroundXie, Y., Bowe, B., Yan, Y., Xian, H., Li, T., & Al-Aly, Z. (2019) · BMJ · doi:10.1136/bmj.l1580 · Cited by 228
Trade-offs of Personal Versus More Proxy Exposure Measures in Environmental Epidemiology
Weisskopf, M. G., & Webster, T. F. (2017) · Epidemiology · doi:10.1097/ede.0000000000000686 · Cited by 213
Handgun Ownership and Suicide in California
Studdert, D. M., Zhang, Y., Swanson, S. A., Prince, L., Rodden, J. A., Holsinger, E. E., Spittal, M. J., Wintemute, G. J., & Miller, M. (2020) · New England Journal of Medicine · doi:10.1056/nejmsa1916744 · Cited by 224
Association of Varicose Veins With Incident Venous Thromboembolism and Peripheral Artery Disease
Chang, S.-L., Huang, Y.-L., Lee, M.-C., Hu, S., Hsiao, Y.-C., Chang, S.-W., Chang, C. J., & Chen, P.-C. (2018) · JAMA · doi:10.1001/jama.2018.0246 · Cited by 217
Cohen, K., Ren, S., Heath, K., Dasmariñas, M. C., Jubilo, K. G., Guo, Y., Lipsitch, M., & Daugherty, S. E. (2022) · BMJ · doi:10.1136/bmj-2021-068414 · Cited by 223
Self-citationAssociation of Short Interpregnancy Interval With Pregnancy Outcomes According to Maternal Age
Schummers, L., Hutcheon, J. A., Hernandez-Diaz, S., Williams, P. L., Hacker, M. R., VanderWeele, T. J., & Norman, W. V. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.4696 · Cited by 205
“We (1) adjusted for additional measured baseline clinical and sociodemographic covariates that could be potential confounding variables (eg, rural residence, obesity, infertility treatment); (2) assessed potential confounding by pregnancy intention by restricting to pregnancies following an infertility diagnosis on th…”
MethodNørgaard, M., Ehrenstein, V., & Vandenbroucke, J. P. (2017) · Clinical Epidemiology · doi:10.2147/clep.s129879 · Cited by 210
“49 Using negative exposure, or outcome controls, does not control confounding, but it is a way to at least address the potential magnitude of uncontrolled confounding.”
Prenatal exposure to acetaminophen and risk of ADHD
Ystrom, E., Gustavson, K., Brandlistuen, R. E., Knudsen, G. P., Magnus, P., Susser, E., Davey Smith, G., Stoltenberg, C., Surén, P., Håberg, S. E., et al. (2017) · Pediatrics · doi:10.1542/peds.2016-3840 · Cited by 217
Gene–obesogenic environment interactions in the UK Biobank study
Tyrrell, J., Wood, A. R., Ames, R. M., Yaghootkar, H., Beaumont, R. N., Jones, S. E., Tuke, M. A., Ruth, K. S., Freathy, R. M., Davey Smith, G., et al. (2017) · International Journal of Epidemiology · doi:10.1093/ije/dyw337 · Cited by 208
“These tests represented ‘impossible by the proposed mechanism’ negative controls.(37,38) These analyses also help to test whether or not statistical artefacts were influencing our results, such as different variances in BMI.”
Target Trial Emulation to Improve Causal Inference from Observational Data: What, Why, and How?
Fu, E. L. (2023) · Journal of the American Society of Nephrology · doi:10.1681/asn.0000000000000152 · Cited by 184
“68,69 These are outcomes for which treatment is not expected to have any effect.”
Cavalcanti, D. M., de Oliveira Ferreira de Sales, L., da Silva, A. F., Basterra, E. L., Pena, D., Monti, C., Barreix, G., Silva, N. J., Vaz, P., Saute, F., et al. (2025) · The Lancet · doi:10.1016/s0140-6736(25)01186-9 · Cited by 178
Yip, T. C.-F., Wong, V. W.-S., Chan, H. L.-Y., Tse, Y.-K., Lui, G. C.-Y., & Wong, G. L.-H. (2020) · Gastroenterology · doi:10.1053/j.gastro.2019.09.025 · Cited by 203
A Selective Review of Negative Control Methods in Epidemiology
Shi, X., Miao, W., & Tchetgen, E. T. (2020) · Current Epidemiology Reports · doi:10.1007/s40471-020-00243-4 · Cited by 188
“A powerful tool increasingly recognized to mitigate bias is negative control study design and analysis [3–5].”
BackgroundMethodInfluentialLevine, S. Z., Kodesh, A., Viktorin, A., Smith, L., Uher, R., Reichenberg, A., & Sandin, S. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.4050 · Cited by 198
Bowe, B., Xie, Y., Li, T., Yan, Y., Xian, H., & Al-Aly, Z. (2017) · The Lancet Planetary Health · doi:10.1016/s2542-5196(17)30117-1 · Cited by 198
Causal Inference Methods for Combining Randomized Trials and Observational Studies: A Review
Colnet, B., Mayer, I., Chen, G., Dieng, A., Li, R., Varoquaux, G. P., Vert, J., Josse, J., & Yang, S. (2024) · Statistical Science · doi:10.1214/23-sts889 · Cited by 302
“The analogy of this principle in causal inference is detailed in (Lipsitch et al., 2010).”
BackgroundEffectiveness of BNT162b2 Vaccine against Critical Covid-19 in Adolescents
Olson, S. M., Newhams, M. M., Halasa, N. B., Price, A. M., Boom, J. A., Sahni, L. C., Pannaraj, P. S., Irby, K., Walker, T. C., Schwartz, S. P., et al. (2022) · New England Journal of Medicine · doi:10.1056/nejmoa2117995 · Cited by 189
Simonsen, L., Taylor, R. J., Schuck-Paim, C., Lustig, R., Haber, M., & Klugman, K. P. (2014) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(14)70032-3 · Cited by 195
Fu, E. L., Evans, M., Clase, C. M., Tomlinson, L. A., van Diepen, M., Dekker, F. W., & Carrero, J. J. (2021) · Journal of the American Society of Nephrology · doi:10.1681/asn.2020050682 · Cited by 192
Taylor, A. E., Jones, H. J., Sallis, H., Euesden, J., Stergiakouli, E., Davies, N. M., Zammit, S., Lawlor, D. A., Munafò, M. R., Davey Smith, G., et al. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyy060 · Cited by 192
“Negative control exposures and/or outcomes can be used to see if associations between genetic variants and outcomes exist that are not biologically plausible and should only arise through selection bias.(39) Similarly, where there is a well characterized association (replicated in a number of studies) of known magnitu…”
BackgroundLabrecque, J., & Swanson, S. A. (2018) · Current Epidemiology Reports · doi:10.1007/s40471-018-0152-1 · Cited by 190
Waaijenborg, S., Hahne, S. J. M., Mollema, L., Smits, G. P., Berbers, G. A. M., van der Klis, F. R. M., de Melker, H. E., & Wallinga, J. (2013) · Journal of Infectious Diseases · doi:10.1093/infdis/jit143 · Cited by 178
“In this sense, the analysis for varicella serves as a negative control [29].”
BackgroundADHD Medication and Substance-Related Problems
Quinn, P. D., Chang, Z., Hur, K., Gibbons, R. D., Lahey, B. B., Rickert, M. E., Sjölander, A., Lichtenstein, P., Larsson, H., & D’Onofrio, B. M. (2017) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2017.16060686 · Cited by 187
Ferdinands, J. M., Rao, S., Dixon, B. E., Mitchell, P. K., DeSilva, M. B., Irving, S. A., Lewis, N., Natarajan, K., Stenehjem, E., Grannis, S. J., et al. (2022) · BMJ · doi:10.1136/bmj-2022-072141 · Cited by 185
Gu, J., Shi, Y., Zhu, Y., Chen, N., Wang, H., Zhang, Z., & Chen, T. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003188 · Cited by 184
“…poisoning by psychotropic agents; poisoning by other medications and drugs; rehabilitation care, fitting of prostheses, and adjustment of devices; and medical examination/evaluation) as negative controls [44], for which no biological or clinical evidence supports an association with air pollution.”
Mamluk, L., Edwards, H. B., Savović, J., Leach, V., Jones, T., Moore, T. H. M., Ijaz, S., Lewis, S. J., Donovan, J. L., Lawlor, D., et al. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015410 · Cited by 176
Català, M., Mercadé-Besora, N., Kolde, R., Trinh, N. T. H., Roel, E., Burn, E., Rathod-Mistry, T., Kostka, K., Man, W. Y., Delmestri, A., et al. (2024) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(23)00414-9 · Cited by 162
Kim, D., Yang, P.-S., You, S. C., Sung, J.-H., Jang, E., Yu, H. T., Kim, T.-H., Pak, H.-N., Lee, M.-H., Lip, G. Y. H., et al. (2021) · BMJ · doi:10.1136/bmj.n991 · Cited by 174
Understanding the assumptions underlying Mendelian randomization
de Leeuw, C., Savage, J., Bucur, I. G., Heskes, T., & Posthuma, D. (2022) · European Journal of Human Genetics · doi:10.1038/s41431-022-01038-5 · Cited by 170
Causal Inference in Developmental Origins of Health and Disease (DOHaD) Research
Gage, S. H., Munafò, M. R., & Davey Smith, G. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033352 · Cited by 173
“If, however, the association seen between the two is due to confounding, then a similar association is likely to be seen in both the analysis of interest and the negative control analysis, in which there is no biologically plausible mechanism for causation (Davey Smith 2008, 2012b; Lipsitch et al. 2010).”
ResultTian, Y., Schuemie, M. J., & Suchard, M. A. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyy120 · Cited by 169
“In addition to simulations under known hazard ratios, we perform negative outcome control experiments using sets of outcomes believed to be unrelated to the compared treatments, thus having a presumed true hazard ratio of 1.(28,29) Negative control outcomes entirely use real-world data, and when properly specified the…”
MethodRelation of Statin Use Prior to Admission to Severity and Recovery Among COVID-19 Inpatients
Daniels, L. B., Sitapati, A. M., Zhang, J., Zou, J., Bui, Q. M., Ren, J., Longhurst, C. A., Criqui, M. H., & Messer, K. (2020) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2020.09.012 · Cited by 169
“We also were able to leverage a large negative control population of all COVID-negative inpatients during the same time period, which enabled us to assess for confounding and bias.(22) In summary, among patients hospitalized for COVID-19, use of statin medication prior to admission was associated with a reduced risk o…”
Examining the Use of Real‐World Evidence in the Regulatory Process
Beaulieu‐Jones, B. K., Finlayson, S. G., Yuan, W., Altman, R. B., Kohane, I. S., Prasad, V., & Yu, K. (2020) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1658 · Cited by 166
You, S. C., Rho, Y., Bikdeli, B., Kim, J., Siapos, A., Weaver, J., Londhe, A., Cho, J., Park, J., Schuemie, M., et al. (2020) · JAMA · doi:10.1001/jama.2020.16167 · Cited by 171
Yao, X., Gersh, B. J., Holmes, D. R., Melduni, R. M., Johnsrud, D. O., Sangaralingham, L. R., Shah, N. D., & Noseworthy, P. A. (2018) · JAMA · doi:10.1001/jama.2018.6024 · Cited by 163
Schuemie, M. J., Hripcsak, G., Ryan, P. B., Madigan, D., & Suchard, M. A. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708282114 · Cited by 163
“This can include bias from multiple sources, such as confounding (5), selection bias, and measurement error (6).”
BackgroundRasmussen, S. G., Ogburn, E. L., McCormack, M., Casey, J. A., Bandeen-Roche, K., Mercer, D. G., & Schwartz, B. S. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.2436 · Cited by 159
Xie, Y., Bowe, B. C., Gibson, A. K., McGill, J. B., Maddukuri, G. S., Yan, Y., & Al‐Aly, Z. (2020) · Diabetes Care · doi:10.2337/dc20-1890 · Cited by 166
Ercümen, A., Gruber, J. S., & Colford, J. M. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1306912 · Cited by 158
“Negative control outcomes (i.e., outcomes that are not expected to be affected by tap water exposure) can be used to assess the magnitude of recall bias when measuring objective outcomes is not feasible (Lipsitch et al. 2010).”
BackgroundEvaluating the Use of Nonrandomized Real‐World Data Analyses for Regulatory Decision Making
Franklin, J. M., Glynn, R. J., Martin, D., & Schneeweiß, S. (2019) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1351 · Cited by 177
Comparative Effectiveness of Generic and Brand-Name Statins on Patient Outcomes: A Cohort Study
Gagne, J. J., Choudhry, N. K., Kesselheim, A. S., Polinski, J. M., Hutchins, D., Matlin, O. S., Brennan, T. A., Avorn, J., & Shrank, W. H. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2942 · Cited by 155
“We used cancer as a negative control outcome (37) to further evaluate whether our findings might be due to unadjusted differences in characteristics between generic and brand-name statin recipients.”
Rozé, J., Cambonie, G., Marchand‐Martin, L., Gournay, V., Durrmeyer, X., Durox, M., Storme, L., Porcher, R., & Ancel, P. (2015) · JAMA · doi:10.1001/jama.2015.6734 · Cited by 152
Kuang, K., Li, L., Geng, Z., Xu, L., Zhang, K., Liao, B., Huang, H., Ding, P., Miao, W., & Jiang, Z. (2020) · Engineering · doi:10.1016/j.eng.2019.08.016 · Cited by 174
“[107,108] provide elegant surveys on the existence of negative controls in observational studies.”
BackgroundCylus, J., Glymour, M. M., & Avendaño, M. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu106 · Cited by 149
Analysis of free text in electronic health records for identification of cancer patient trajectories
Jensen, K., Soguero-Ruíz, C., Mikalsen, K. Ø., Lindsetmo, R., Kouskoumvekaki, I. A., Girolami, M., Skrøvseth, S. O., & Augestad, K. M. (2017) · Scientific Reports · doi:10.1038/srep46226 · Cited by 163
Weinroth, M. D., Belk, A. D., Dean, C. J., Noyes, N. R., Dittoe, D. K., Rothrock, M. J., Ricke, S. C., Myer, P. R., Henniger, M. T., Ramírez, G. A., et al. (2022) · Journal of Animal Science · doi:10.1093/jas/skab346 · Cited by 162
Allinson, J. P., Chaturvedi, N., Wong, A., Shah, I., Donaldson, G. C., Wedzicha, J. A., & Hardy, R. (2023) · The Lancet · doi:10.1016/s0140-6736(23)00131-9 · Cited by 152
“21 If this explanation was correct, we would expect early childhood LRTI to also predict those causes of mortality that are strongly influenced by childhood socioeconomic position.”
Metadata
- Authors
- Lipsitch, M., Tchetgen Tchetgen, E., & Cohen, T.
- Affiliations
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Lipsitch, Marc
- Venue
- Epidemiology
- Published
- 2010-05
- Volume
- 21
- Issue
- 3
- Pages
- 383-388
- ISSN
- 1044-3983 (ISSN-L)
- PMC
- PMC3053408
- OpenAlex
- W2009428399
- Cited by
- 1,583
- References
- 25
- Citations by year
- 2026: 278 · 2025: 215 · 2024: 183 · 2023: 138 · 2022: 153 · 2021: 132 · 2020: 107 · 2019: 75 · 2018: 72 · 2017: 71 · 2016: 56 · 2015: 32 · 2014: 26 · 2013: 25 · 2012: 11
- FWCI
- 11.64
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Funders
- National Institutes of Health (5u01gm076497-03, 1dp2od006663-01)
- National Institute of General Medical Sciences (U54GM088558, U01GM076497, 5U01GM076497)
- National Institute of Environmental Health Sciences (T32ES007142)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- negative control, confounding, spurious association, exposure control, outcome control, observational study validity, recall bias, experimental design, causal inference
- MeSH terms
- Humans, Bias, Causality, Confounding Factors, Epidemiologic, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.