Negative Controls: A Tool for Detecting Confounding and Bias in Observational Studies
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
Noncausal associations between exposures and outcomes are a threat to validity of causal inference in observational studies. Many techniques have been developed for study design and analysis to identify and eliminate such errors. Such problems are not expected to compromise experimental studies, where careful standardization of conditions (for laboratory work) and randomization (for population studies) should, if applied properly, eliminate most such noncausal associations. We argue, however, that a routine precaution taken in the design of biologic laboratory experiments--the use of "negative controls"--is designed to detect both suspected and unsuspected sources of spurious causal inference. In epidemiology, analogous negative controls help to identify and resolve confounding as well as other sources of error, including recall bias or analytic flaws. We distinguish 2 types of negative controls (exposure controls and outcome controls), describe examples of each type from the epidemiologic literature, and identify the conditions for the use of such negative controls to detect confounding. We conclude that negative controls should be more commonly employed in observational studies, and that additional work is needed to specify the conditions under which negative controls will be sensitive detectors of other sources of error in observational studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, in order to evaluate the specificity of the association to ADHD medication, the tenth used selective serotonin reuptake inhibitors (SSRIs) as a negative control medication exposure (27, 30).”
From ADHD Medication and Substance-Related Problems (2017) · cites this paper as [30]
In the citing paper
“The use of ‘negative controls’ for outcomes or exposures can be helpful to assess the likely presence of unmeasured or residual confounding in observational research [51].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [51]
In the citing paper
“In a third approach–negative control studies–we would expect an unbiased negative control study to be null and the ‘real’ study not-null if there is a true causal effect and the assumptions of this approach are met (see Table 1).39,40”
From Triangulation in aetiological epidemiology (2017) · cites this paper as [39]
Citations
Cites
(1979) · Medical Entomology and Zoology · Cited by 11,252
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,285
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · cited 1×
“19 Hill argued that causal inferences were more credible if the exposure (in his example, nickel mining) was associated with only certain types of outcomes (death from lung and nose cancer but not death from other cancers), and if the outcome was associated with one kind of exposure (nickel mining)…”
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844
An introduction to instrumental variables for epidemiologists
Greenland, S. (2000) · International Journal of Epidemiology · doi:10.1093/ije/29.4.722 · Cited by 1,156
Instruments for Causal Inference: An Epidemiologist's Dream?
Hernán, M. A., & Robins, J. M. (2006) · Epidemiology · doi:10.1097/01.ede.0000222409.00878.37 · Cited by 961
Vaccines for preventing influenza in the elderly
Demicheli, V., Jefferson, T., Di Pietrantonj, C., Ferroni, E., Thorning, S., Thomas, R. E., & Rivetti, A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004876.pub4 · Cited by 695
Impact of Influenza Vaccination on Seasonal Mortality in the US Elderly Population
(2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.3.265 · Cited by 543 · cited 1×
“8 To test this hypothesis, Jackson et al 7 reproduced earlier estimates of the protective effect of influenza vaccination, but then repeated the analysis for two sets of negative control outcomes, and showed that the protective effect was observed even in circumstances where the vaccine could not…”
Mortality benefits of influenza vaccination in elderly people: an ongoing controversy
Simonsen, L., Taylor, R. J., Viboud, C., Miller, M. A., & Jackson, L. A. (2007) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(07)70236-0 · Cited by 520 · cited 1×
“However, older age is associated with a less robust immune response to influenza vaccination, and ecological data suggest that the benefits measured in observational studies far exceed the corresponding benefits expected at the population level when influenza vaccination rates have increased among the elderly.(6) Impo…”
Cited by
Items 301–400 of 1,002
Yang, C., Yao, W., Yang, C., Peng, Z., Ou, H., & Kuo, S. (2024) · Journal of Internal Medicine · doi:10.1111/joim.13751 · Cited by 42
“…ratio of 2.94 was close to the estimated E -value of 2.97 and our hypothetical calculation only accounted for residual confounding by HbA1c levels, we further performed a negative control outcome analysis to detect the possibility of residual confounding effects in this observational study [30].”
InfluentialEvaluating the impact of health policies: using a difference-in-differences approach
Saeed, S., Moodie, E. E. M., Strumpf, E., & Klein, M. B. (2019) · International Journal of Public Health · doi:10.1007/s00038-018-1195-2 · Cited by 45
McQuade, E. T. R., Liu, J., Kang, G., Kosek, M. N., Lima, A. Â. M., Bessong, P. O., Samie, A., Haque, R., Mduma, E. R., Shrestha, S. K., et al. (2020) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiaa031 · Cited by 48
“This study design allowed for the novel bias correction based on multiple negative control associations, which generated an empirical null distribution that captured a common distribution of biases [24], but did not require the structure and magnitude of confounding to be identical [47].”
MethodLi, L., Sujan, A. C., Butwicka, A., Chang, Z., Cortese, S., Quinn, P. D., Viktorin, A., Öberg, A. S., D'Onofrio, B. M., & Larsson, H. (2020) · CNS Drugs · doi:10.1007/s40263-020-00728-2 · Cited by 49
Yang, Q., Chang, A., Tong, X., & Merritt, R. K. (2021) · Stroke · doi:10.1161/strokeaha.120.032788 · Cited by 49
Child height, health and human capital: Evidence using genetic markers
von Hinke Kessler Scholder, S., Davey Smith, G., Lawlor, D. A., Propper, C., & Windmeijer, F. (2013) · European Economic Review · doi:10.1016/j.euroecorev.2012.09.009 · Cited by 42
Dave, C. V., Schneeweiss, S., & Patorno, E. (2019) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2019.2813 · Cited by 44
“Oncology and malignant hematology 12 (5)”
Ridenhour, B. J., Campitelli, M. A., Kwong, J. C., Rosella, L. C., Armstrong, B., Mangtani, P., Calzavara, A. J., & Shay, D. K. (2013) · PLoS ONE · doi:10.1371/journal.pone.0076318 · Cited by 44
“0) as a ‘negative control’ outcome for which influenza vaccination is not expected to have an effect [28].”
BackgroundSeals, R. M., Hansen, J., Gredal, O., & Weisskopf, M. G. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kwv169 · Cited by 44
A safety study evaluating non-COVID-19 mortality risk following COVID-19 vaccination
Xu, S., Huang, R., Sy, L. S., Hong, V. X., Glenn, S. C., Ryan, D. S., Morrissette, K. L., Vazquez‐Benitez, G., Glanz, J. M., Klein, N. P., et al. (2022) · Vaccine · doi:10.1016/j.vaccine.2022.12.036 · Cited by 48
“We hypothesize that the negative control outcome, hospitalization for trauma or injury, shares the same potential sources of bias with our primary outcome (death) but cannot plausibly be related to COVID-19 vaccination.[18,32] Trauma or injury hospitalizations were identified with the following ICD-10 codes: S00-T88 f…”
Huitfeldt, A., Hernán, M. Á., Kalager, M., & Robins, J. M. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1234 · Cited by 44
Oswald, W. E., Stewart, A. E. P., Kramer, M. R., Endeshaw, T., Zerihun, M., Melak, B., Sata, E., Gessese, D., Teferi, T., Tadesse, Z., et al. (2017) · Parasites & Vectors · doi:10.1186/s13071-017-2020-0 · Cited by 52
“If these control exposures do not cause AL and TT infection and have a comparable set of confounders as community sanitation usage, then any detected association of these exposures with the outcomes would indicate bias in the main association of interest [31].”
BackgroundMethodInfluentialKernel Methods for Unobserved Confounding: Negative Controls, Proxies, and InstrumentsPreprint
Singh, R. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2012.10315 · Cited by 56
“Subsequent work formalized these concepts [60, 77, 45, 30].”
Borges, M. C., Clayton, G., Freathy, R. M., Felix, J. F., Fernández‐Sanlés, A., Soares, A. G., Kilpi, F., Yang, Q., McEachan, R., Richmond, R. C., et al. (2024) · BMC Medicine · doi:10.1186/s12916-023-03167-0 · Cited by 49
“An extensive description of assumptions and sources of bias for each approach has been reported previously (e.g. [17–21]).”
BackgroundGlanz, J. M., Newcomer, S. R., Jackson, M. L., Omer, S. B., Bednarczyk, R. A., Shoup, J. A., Destefano, F. R., Daley, M. F., Goddard, K., Panneton, M., et al. (2016) · Vaccine · doi:10.1016/j.vaccine.2015.10.082 · Cited by 45
“One feasible approach is to analyze the association between the vaccine schedule and one or more “control” outcomes.(75) In this context, control outcomes are diseases that have no expected causal association with the vaccine schedule.”
BackgroundWeisskopf, M. G., Tchetgen, E. J. T., & Raz, R. (2016) · Epidemiology · doi:10.1097/ede.0000000000000454 · Cited by 43
Andersen, L. W., Holmberg, M. J. J., Granfeldt, A., Løfgren, B., Vellano, K. M., McNally, B., Siegerink, B., Kurth, T., & Donnino, M. W. (2018) · Resuscitation · doi:10.1016/j.resuscitation.2018.02.021 · Cited by 44
Effect of Bisphosphonates on Fracture Outcomes Among Frail Older Adults
Zullo, A. R., Zhang, T., Lee, Y., McConeghy, K. W., Daiello, L. A., Kiel, D. P., Mor, V., & Berry, S. D. (2018) · Journal of the American Geriatrics Society · doi:10.1111/jgs.15725 · Cited by 43
“We also used a negative control outcome of hospitalized heart failure measured using a validated claims-based definition.(36,37) An association between the drugs and heart failure would suggest our primary outcome estimates were biased and that residual confounding may explain the findings.”
MethodPhysical Activity Throughout Adolescence and Peak Hip Strength in Young Adults
Elhakeem, A., Heron, J., Tobias, J. H., & Lawlor, D. A. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.13463 · Cited by 34
Falsification End Points for Observational Studies
Groenwold, R. H. H. (2013) · JAMA · doi:10.1001/jama.2013.3089 · Cited by 43
“To the Editor: A proposed link between Lyme disease and autism has garnered considerable attention.(1,2) Among individuals with autism spectrum disorders, rates of seropositivity for Lyme disease of greater than 20% have been reported.”
BackgroundChange in Neighborhood Disadvantage and Change in Smoking Behaviors in Adults
Halonen, J. I., Pulakka, A., Stenholm, S., Pentti, J., Kawachi, I. I., Mika Kivimaki, B., & Vahtera, J. (2016) · Epidemiology · doi:10.1097/ede.0000000000000530 · Cited by 42
“≥national mean 944 (35) 375 (14) 301 (40) 115 (15) -”
Magnus, M. C., Iliodromiti, S., Lawlor, D. A., Catov, J. M., Nelson, S. M., & Fraser, A. (2017) · Epidemiology · doi:10.1097/ede.0000000000000712 · Cited by 44
“Special occasions only 24,433 (14) 9,075 (7) Never 14,898 (8) 7,654 (6) Prefer not to answer 123 (0) 168 (0) Quintiles of moderate/vigorous physical activity, estimated METS/min/wk, n (%) 1 38,653 (21) 22,923 (17) 2 31,722 (18) 29,306 (22) 3 37,152 (21) 25,879 (19) 4 37,633 (21) 27,064 (20) 5 35,466 (20) 28,087 (21) B…”
Mooney, S. J., Joshi, S., Cerdá, M., Kennedy, G. J., Beard, J., & Rundle, A. (2017) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-16-0827 · Cited by 45
“In this light, our "negative control" finding that no environmental factors were associated with heavy housework after Bonferroni correction provides some, albeit incomplete, evidence against residual confounding (66).”
BackgroundDouble Negative Control Inference in Test-Negative Design Studies of Vaccine Effectiveness
Li, K. Q., Shi, X., Miao, W., & Tchetgen, E. T. (2023) · Journal of the American Statistical Association · doi:10.1080/01621459.2023.2220935 · Cited by 54
“In recent years, negative control variables have emerged as powerful tools to detect, reduce, and potentially correct for unmeasured confounding bias (Lipsitch, Tchetgen Tchetgen, and Cohen 2010; Miao, Geng, and Tchetgen Tchetgen 2018; Shi, Miao, and Tchetgen Tchetgen 2020).”
Mason, K., Alexiou, A., Li, A., & Taylor-Robinson, D. C. (2024) · Social Science & Medicine · doi:10.1016/j.socscimed.2024.116939 · Cited by 41
Lattanner, M. R., McKetta, S. C., Pachankis, J. E., & Hatzenbuehler, M. L. (2024) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-071723-013336 · Cited by 43
Busby, J., McMenamin, Ú., Spence, A., Johnston, B. T., Hughes, C., & Cardwell, C. R. (2018) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.14388 · Cited by 40
Liu, C.-H., Kang, E. Y., Lin, Y., Wu, W., Liu, Z., Kuo, C., Lai, C., & Hwang, Y. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02881-w · Cited by 44
“To detect the possibility of unmeasured confounding factors, we also identified several negative control exposures to ocular diseases, including retinal detachment, uveitis, and blepharitis [17].”
BackgroundHald, S. M., Möller, S., Rodríguez, L. A. G., Salman, R. A., Sharma, M., Christensen, H., Hellfritzsch, M., Pottegård, A., Hallas, J., & Gaist, D. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.8380 · Cited by 46
Bonander, C., Humphreys, D. K., & Esposti, M. D. (2021) · American Journal of Epidemiology · doi:10.1093/aje/kwab211 · Cited by 47
“In Web Appendix 2, we discuss how placebo studies (12), manual restriction of the donor pool (15), and negative control outcomes (53) can be used to assess the robustness of the results in SCM studies.”
MethodInferring Causal Relationships between Risk Factors and Outcomes Using Genetic Variation
Burgess, S., Foley, C. N., & Zuber, V. (2019) · Wiley eBooks · doi:10.1002/9781119487845.ch23 · Cited by 40
Gustavson, K., Torvik, F. A., Smith, G. D., Røysamb, E., & Eilertsen, E. M. (2024) · European Journal of Epidemiology · doi:10.1007/s10654-024-01132-6 · Cited by 43
“Different research designs approach unmeasured confounding in different ways [se examples in 42, 43, 44], and results from sibling control studies should be compared to results from other designs with other sources of bias.”
ResultWelk, B. K., McClure, J. A., Clarke, C. F. M., Vogt, K., & Campbell, J. D. (2019) · European Urology · doi:10.1016/j.eururo.2019.08.031 · Cited by 40
Diabetes Minimally Mediated the Association Between PM2.5 Air Pollution and Kidney Outcomes
Bowe, B. C., Xie, Y., Yan, Y., Xian, H., & Al‐Aly, Z. (2020) · Scientific Reports · doi:10.1038/s41598-020-61115-x · Cited by 42
Xie, Y., Bowe, B., Gibson, A. K., McGill, J. B., Yan, Y., Maddukuri, G., & Al-Aly, Z. (2020) · Diabetes Care · doi:10.2337/dc20-1231 · Cited by 40
“(38) to test the association between empagliflozin and risk of traffic-related injury as a negative outcome control where a priori evidence suggests that an association is not expected.”
BackgroundPapandreou, C., Hernández‐Alonso, P., Bulló, M., Ruiz‐Canela, M., Li, J., Guasch‐Ferré, M., Toledo, E., Clish, C. B., Corella, D., Estruch, R., et al. (2020) · Journal of Nutrition · doi:10.1093/jn/nxaa273 · Cited by 42
“Second, although we carefully adjusted for many potential confounders, residual confounding cannot be ruled out (53).”
BackgroundElser, H. C., Parks, R. M., Moghavem, N., Kiang, M. V., Bozinov, N., Henderson, V. W., Rehkopf, D. H., & Casey, J. A. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003580 · Cited by 43
Guida, F., Tan, V. Y., Corbin, L. J., Smith-Byrne, K., Alcala, K., Langenberg, C., Stewart, I. D., Butterworth, A. S., Surendran, P., Achaintre, D., et al. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003786 · Cited by 40
Redelmeier, D. A., Chan, W. K., & Lu, H. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000369 · Cited by 39
“The purpose of these analyses was to check for the absence of an association where no association would be expected [64].”
BackgroundPrenatal paracetamol exposure and neurodevelopmental outcomes in preschool‐aged children
Trønnes, J. N., Wood, M. E., Lupattelli, A., Ystrøm, E., & Nordeng, H. (2019) · Paediatric and Perinatal Epidemiology · doi:10.1111/ppe.12568 · Cited by 41
Miller, M., Zhang, Y., Prince, L., Swanson, S. A., Wintemute, G. J., Holsinger, E. E., & Studdert, D. M. (2022) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2022.0793 · Cited by 40
Orandi, B. J., Chen, Y., Li, Y., Charyton, D., Lentine, K. L., Lee, B. P., Ali, N., DeMarco, M. P., Weintraub, M. A., Bae, S., et al. (2025) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.0000000750 · Cited by 38
Brew, B. K., Gong, T., Williams, D. M., Larsson, H., & Almqvist, C. (2017) · Scandinavian Journal of Public Health · doi:10.1177/1403494817702324 · Cited by 37
“[4,5] The negative control can be either another outcome or another exposure that ideally shares the same measured and unmeasured confounders as the exposure of interest.[4] Additionally, a control outcome should not be caused by the exposure of interest, and a control exposure should not cause the outcome of interest.”
BackgroundInfluentialAdults with diabetes residing in “food swamps” have higher hospitalization rates
Phillips, A. Z., & Rodríguez, H. P. (2019) · Health Services Research · doi:10.1111/1475-6773.13102 · Cited by 39
Autoimmune and viral risk factors are associated with achalasia: A case‐control study
Gaber, C. E., Cotton, C. C., Eluri, S., Lund, J. L., Farrell, T. M., & Dellon, E. S. (2021) · Neurogastroenterology & Motility · doi:10.1111/nmo.14312 · Cited by 39
State Cannabis Legalization and Psychosis-Related Health Care Utilization
Elser, H. C., Humphreys, K. N., Kiang, M. V., Mehta, S. S., Yoon, J. H., Faustman, W. O., & Matthay, E. C. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.52689 · Cited by 42
Proximal causal inference for complex longitudinal studies
Ying, A., Miao, W., Shi, X., & Tchetgen, E. J. T. (2023) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkad020 · Cited by 68
Young‐Xu, Y., Epstein, L. H., Marconi, V. C., Davey, V. J., Zwain, G., Smith, J., Korves, C., Cunningham, F., Bonomo, R. A., & Ginde, A. A. (2023) · mBio · doi:10.1128/mbio.01024-23 · Cited by 39
Khush, R. S., Arnold, B. F., Srikanth, P., Sudharsanam, S., Ramaswamy, P., Natesan, D., London, A. G., Ramaprabha, P., Rajkumar, P., Balakrishnan, K., et al. (2013) · American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.13-0067 · Cited by 43
“biased by similar confounding or measurement error as diarrhea and could, thus, serve as negative control outcomes.(32) Human subjects research protection.”
BackgroundColiphages and Gastrointestinal Illness in Recreational Waters
Benjamin‐Chung, J., Arnold, B. F., Wade, T. J., Schiff, K., Griffith, J. F., Dufour, A. P., Weisberg, S. B., & Colford, J. M. (2017) · Epidemiology · doi:10.1097/ede.0000000000000681 · Cited by 38
Performing an Informatics Consult: Methods and Challenges
Schuler, A., Callahan, A., Jung, K., & Shah, N. H. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.12.023 · Cited by 37
Wernham, E., Gurney, J., Stanley, J., Ellison-Loschmann, L., & Sarfati, D. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002134 · Cited by 37
“First, we used being SGA as a negative control to assess for residual and unmeasured confounding [13].”
BackgroundMethodLevintow, S. N., Nielson, C. M., Hernandez, R. K., Breskin, A., Pritchard, D., Lash, T. L., Rothman, K. J., Gilbertson, D. T., Muntner, P., Critchlow, C. W., et al. (2023) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.5623 · Cited by 38
“An NCO is a variable that (a) has no plausible mechanism by which it can be caused by the treatment under study and (b) is expected to be affected by the same confounding structure as the treatment contrast for the outcome of interest.(10,11) If condition (a) is met, then an observed non-null association beyond what m…”
BackgroundHuang, T. T. ‐., Wyka, K., Ferris, E., Gardner, J., Evenson, K. R., Tripathi, D., Soto, G. M., Cato, M., Moon, J., Wagner, J., et al. (2016) · BMC Public Health · doi:10.1186/s12889-016-3822-2 · Cited by 39
“Standard demographic information, smoking status [12], flu shot behavior [12], coffee consumption [13] (the latter two will be used as health risk behavior covariates and negative control outcomes [14]) will be collected in the screener or annual surveys in addition to the key outcome measures described below (see Tab…”
MethodWang, X., Zhang, T., Ekheden, I. G., Chang, Z., Hellner, C., Hasselström, J., Jayaram‐Lindström, N., D'Onofrio, B. M., Larsson, H., Mataix‐Cols, D., et al. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104647 · Cited by 42
“, any positive associations of paternal exposure with child outcome would suggest familial confounding) (Lipsitch et al., 2010).”
BackgroundComparative effectiveness of third doses of mRNA-based COVID-19 vaccines in US veterans
Dickerman, B. A., Gerlovin, H., Madenci, A. L., Muñiz, M. J. F., Wise, J. K., Adhikari, N., Ferolito, B. R., Kurgansky, K. E., GAGNON, D. R., Cho, K., et al. (2023) · Nature Microbiology · doi:10.1038/s41564-022-01272-z · Cited by 39
Navarro, C., Lau, C., Buchan, S. A., Burchell, A. N., Nasreen, S., Friedman, L., Okpokoro, E., Austin, P. C., Tan, D. H. S., Gubbay, J. B., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078243 · Cited by 37
“Wesharedstudyresultswithdiversecommunity representativesinterestedinthempoxresponse.”
Khera, R., Schuemie, M. J., Lu, Y., Ostropolets, A., Chen, R., Hripcsak, G., Ryan, P. B., Krumholz, H. M., & Suchard, M. A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-057977 · Cited by 37
Yi, W., Kim, B. H., Kim, M., Kim, J., Im, M., Ryang, S., Kim, E. H., Jeon, Y. K., Kim, S. S., & Kim, I. J. (2022) · Thyroid · doi:10.1089/thy.2021.0634 · Cited by 36
Pharmacoepidemiology: An Overview
Sabaté, M., & Montané, E. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12227033 · Cited by 36
“When using this methodology, the researcher has to have expertise in the subject for the choice of negative controls [48].”
MethodLow-Dose Methotrexate and Serious Adverse Events Among Older Adults With Chronic Kidney Disease
Muanda, F. T., Blake, P. G., Weir, M. A., Ahmadi, F., McArthur, E., Sontrop, J. M., Urquhart, B. L., Kim, R. B., & Garg, A. X. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.45132 · Cited by 37
Denosumab and the Risk of Diabetes in Patients Treated for Osteoporosis
Huang, H., Chuang, A. T.-M., Liao, T., Shao, S., Liu, P. P., Tu, Y., & Lai, E. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.54734 · Cited by 40
Okubo, Y., Hayakawa, I., Sugitate, R., & Nariai, H. (2025) · Paediatric and Perinatal Epidemiology · doi:10.1111/ppe.70071 · Cited by 46
“A negative control method has been used to assess the presence, direction and magnitude of residual bias in this research area [50].”
Assessing the short term impact of air pollution on mortality: a matching approach
Baccini, M., Mattei, A., Mealli, F., Bertazzi, P. A., & Carugno, M. (2017) · Environmental Health · doi:10.1186/s12940-017-0215-7 · Cited by 42
“This analysis on other causes of death, which can be interpreted as a proxy of a negative control analysis [42], showed that the estimated impacts would be negligible if deaths from cardiovascular and respiratory diseases were excluded.”
BackgroundChristiansen, C. F., Thomsen, R. W., Schmidt, M., Pedersen, L. H., & Sørensen, H. T. (2019) · Intensive Care Medicine · doi:10.1007/s00134-019-05648-4 · Cited by 40
Robust Inference on Population Indirect Causal Effects: The Generalized Front Door Criterion
Fulcher, I., Shpitser, I., Marealle, S., & Tchetgen, E. J. T. (2019) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12345 · Cited by 76
Sinnott, S., Douglas, I., Smeeth, L. L., Williamson, E., & Tomlinson, L. A. (2020) · BMJ · doi:10.1136/bmj.m4080 · Cited by 43
Lazzati, A., Epaud, S., Ortala, M., Katsahian, S., & Lanoy, E. (2022) · British Journal of Surgery · doi:10.1093/bjs/znac003 · Cited by 36
Hulme, W. J., Horne, E. M. F., Parker, E. P. K., Keogh, R. H., Williamson, E., Walker, V. M., Palmer, T., Curtis, H. J., Walker, A. J., Mehrkar, A., et al. (2023) · BMJ · doi:10.1136/bmj-2022-072808 · Cited by 39
Association of sulfonylureas with the risk of dementia: A population‐based cohort study
Wu, C., Iskander, C., Wang, C., Xiong, L. Y., Shah, B. R., Edwards, J. D., Kapral, M. K., Herrmann, N., Lanctôt, K. L., Masellis, M., et al. (2023) · Journal of the American Geriatrics Society · doi:10.1111/jgs.18397 · Cited by 38
“Incident cataract surgery was examined as a negative control outcome (falsification outcome), 23 since DPP4 inhibitors and sulfonylureas have no theoretical effect on incident cataract surgery, but the amount of time that it takes to receive cataract surgery may have similar mechanisms of…”
BackgroundHarris, D. A., Hayes, K. N., Zullo, A. R., Mor, V., Chachlani, P., Deng, Y., McCarthy, E. P., Djibo, D. A., McMahill‐Walraven, C. N., & Gravenstein, S. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.26852 · Cited by 38
Performance of prior event rate ratio adjustment method in pharmacoepidemiology: a simulation study
Uddin, M. J., Groenwold, R. H. H., van Staa, T. P., de Boer, A., Belitser, S. V., Hoes, A. W., Roes, K. C. B., & Klungel, O. H. (2014) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.3724 · Cited by 41
The current deconstruction of paradoxes: one sign of the ongoing methodological “revolution”
Porta, M. S., Vineis, P., & Bolúmar, F. (2015) · European Journal of Epidemiology · doi:10.1007/s10654-015-0068-8 · Cited by 38
“confounding, selection bias, and information bias in observational studies in some basic, clinical, and epidemiological settings [58].”
BackgroundWang, Y., Su, B., Xie, J., García‐Rizo, C., & Prieto‐Alhambra, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01853-4 · Cited by 37
A framework for understanding selection bias in real-world healthcare data
Kundu, R., Shi, X., Morrison, J., Barrett, J., & Mukherjee, B. (2024) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1093/jrsssa/qnae039 · Cited by 38
Guideline-Based Physical Activity and Survival Among US Men With Nonmetastatic Prostate Cancer
Dickerman, B. A., Giovannucci, E. L., Pernar, C. H., Mucci, L. A., & Hernán, M. Á. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwy261 · Cited by 37
Identifying Effects of Multiple Treatments in the Presence of Unmeasured Confounding
Miao, W., Hu, W., Ogburn, E. L., & Zhou, X. (2022) · Journal of the American Statistical Association · doi:10.1080/01621459.2021.2023551 · Cited by 62
“studies (Wright, 1928; Goldberger, 1972; Robins, 1994; Baker and Lindeman, 1994; Angrist et al., 1996; Didelez and Sheehan, 2007; Small et al., 2017). Recently, Miao et al. (2018), Shi et al. (2020), Lipsitch et al. (2010), 3 Kuroki and Pearl (2014), Ogburn and VanderWeele (2012), de Luna et al. (2017), Flanders et al…”
BackgroundFranklin, J. M., Platt, R. G., Dreyer, N. A., London, A. J., Simon, G. E., Watanabe, J. H., Horberg, M. A., Hernandez, A. F., & Califf, R. M. (2021) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.2255 · Cited by 38
“Another possibility is assessing the effects of treatments on control outcomes that share a similar confounding mechanism as the outcome of interest but have a known relationship with treatment.(50,51) Replicating the known effect on the control outcome provides some assurance that confounding has been well controlled.”
BackgroundSeo, S. I., Park, C. H., Kim, T. J., Bang, C. S., Kim, J., Lee, K. J., Kim, J., Kim, H. H., You, S. C., & Shin, W. G. (2021) · Cancer Medicine · doi:10.1002/cam4.4514 · Cited by 35
Association Between Regular Laxative Use and Incident Dementia in UK Biobank Participants
Yang, Z., Wei, C., Li, X., Yuan, J.-Q., Gao, X., Li, B., Zhao, Z.-Y., Toh, S., Yu, X., Brayne, C., et al. (2023) · Neurology · doi:10.1212/wnl.0000000000207081 · Cited by 39
“laxative use with dementia would be distorted by residual confounding.(30) Hip fracture was identified using data from hospital inpatient admissions.”
MethodErcümen, A., Arnold, B. F., Naser, A. M., UNICOMB, L. E., Colford, J. M., & Luby, S. P. (2016) · Tropical Medicine & International Health · doi:10.1111/tmi.12803 · Cited by 37
Nguyen, A., Grembi, J. A., Riviere, M., Heitmann, G. B., Hutson, W. D., Athni, T. S., Patil, A., Ercümen, A., Lin, A., Crider, Y. S., et al. (2024) · Environmental Health Perspectives · doi:10.1289/ehp13807 · Cited by 39
“30,31 At enrollment, fi eld sta ff recorded household geocoordinates using a Global Positioning System locating device, allowing us to map outcomes to a speci fi c point in space and time.”
Hillus, D., Le, N. H., Tober‐Lau, P., Fietz, A., Hoffmann, C., Stegherr, R., Huang, L., Baumgarten, A., Voit, F., Bickel, M., et al. (2025) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(25)00018-0 · Cited by 34
Kjerpeseth, L. J., Selmer, R. M., Ariansen, I., Karlstad, Ø., Ellekjær, H., & Skovlund, E. (2019) · PLoS ONE · doi:10.1371/journal.pone.0221500 · Cited by 36
“To test how well we were able to control for confounding, we performed a sensitivity analysis using a presumed “neutral” outcome, hospitalization or death from pneumonia, as a negative control [34] (S7 Table).”
MethodMartin, E. T., Cheng, C. K., Petrie, J. G., Alyanak, E., Gaglani, M., Middleton, D. B., Ghamande, S. A., Silveira, F. P., Murthy, K., Zimmerman, R. K., et al. (2020) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiaa685 · Cited by 35
Easey, K., & Sharp, G. C. (2021) · Reproductive Health · doi:10.1186/s12978-021-01266-w · Cited by 36
Heikkilä, P., NISKANEN, L. K., But, A., Sorsa, T. A., & Haukka, J. K. (2022) · Frontiers in Oral Health · doi:10.3389/froh.2022.956072 · Cited by 34
Gatto, N. M., Vititoe, S. E., Rubinstein, E. R., Reynolds, R. F., & Campbell, U. B. (2023) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.2883 · Cited by 34
Young‐Xu, Y., Snider, J. T., van Aalst, R., Mahmud, S. M., Thommes, E. W., Lee, J. K. H., Greenberg, D. P., & Chit, A. (2019) · Vaccine · doi:10.1016/j.vaccine.2019.01.063 · Cited by 36
“This is especially so when valid negative controls are lacking and when the confounders are difficult to quantify and measure [14].”
BackgroundProton pump inhibitors and risk of periampullary cancers—A nested case–control study
Chien, L., Huang, Y., Shao, Y. J., Chang, C., Chuang, M., Chiou, H., & Yen, Y. (2015) · International Journal of Cancer · doi:10.1002/ijc.29896 · Cited by 37
“This is also a tool to detect confounding variables in observational studies.(23) Lung cancer was selected as its demographic characteristic is very similar to periampullary cancers.”
MethodUse of proton pump inhibitors and risk of ischemic events in the general population
Casula, M., Scotti, L., Galimberti, F., Mozzanica, F., Tragni, E., Corrao, G., & Catapano, A. L. (2018) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2018.08.035 · Cited by 33
“Of note, the studies investigating the association between PPIs and CV risk have paid particular attention to the use of strategies, both in the design and in the analysis, that minimized these biases (35), such as the adjustment for cardiovascular therapies, use of the propensity score, the new-users design (36), the…”
MethodLiu, L. Y., Bush, W. S., Koyutürk, M., & Karakurt, G. (2020) · BMC Women s Health · doi:10.1186/s12905-020-01104-4 · Cited by 37
Visontay, R., Sunderland, M., Slade, T. N., Wilson, J., & Mewton, L. R. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01486-5 · Cited by 35
“Regarding alternative designs, twin studies and other family-based designs control automatically for shared confounders, as do negative controls [18].”
BackgroundHtoo, P. T., Buse, J. B., Cavender, M. A., Wang, T., Pate, V., Edwards, J. K., & Stürmer, T. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.121.022376 · Cited by 34
Muanda, F. T., Sood, M. M., Weir, M. A., Sontrop, J. M., Ahmadi, F., Yoo, E. C., Kim, R. B., Silverman, M., Knoll, G. A., & Garg, A. X. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.24892 · Cited by 38
Shi, X., Li, K. Q., & Mukherjee, B. (2022) · American Journal of Epidemiology · doi:10.1093/aje/kwac203 · Cited by 44
Ang, M. Y., Takeuchi, F., & Kato, N. (2023) · Journal of Human Genetics · doi:10.1038/s10038-023-01189-3 · Cited by 36
“Negative control and benchmarking analysis are crucial for experimental reliability [31].”
Keele, L. J., Lorch, S. A., Passarella, M. A., Small, D. S., & Titiunik, R. (2017) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0731-905320170000038007 · Cited by 46
Negative Control Outcomes and the Analysis of Standardized Mortality Ratios
Richardson, D. B., Keil, A. P., Tchetgen, E. T., & Cooper, G. S. (2015) · Epidemiology · doi:10.1097/ede.0000000000000353 · Cited by 39
A research agenda for understanding how social inequality is linked to brain structure and function
Hatzenbuehler, M. L., McLaughlin, K. A., Weissman, D. G., & Cikara, M. (2024) · Nature Human Behaviour · doi:10.1038/s41562-023-01774-8 · Cited by 30
“A second strategy for addressing plausible alternative explanations is the strategic selection of control groups (also known as negative control analyses) 102 in which researchers examine whether there is an association in a group where it would not be expected to occur.”
Metadata
- Authors
- Lipsitch, M., Tchetgen Tchetgen, E., & Cohen, T.
- Affiliations
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Lipsitch, Marc
- Venue
- Epidemiology
- Published
- 2010-05
- Volume
- 21
- Issue
- 3
- Pages
- 383-388
- ISSN
- 1044-3983 (ISSN-L)
- PMC
- PMC3053408
- OpenAlex
- W2009428399
- Cited by
- 1,583
- References
- 25
- Citations by year
- 2026: 278 · 2025: 215 · 2024: 183 · 2023: 138 · 2022: 153 · 2021: 132 · 2020: 107 · 2019: 75 · 2018: 72 · 2017: 71 · 2016: 56 · 2015: 32 · 2014: 26 · 2013: 25 · 2012: 11
- FWCI
- 11.64
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Funders
- National Institutes of Health (5u01gm076497-03, 1dp2od006663-01)
- National Institute of General Medical Sciences (U54GM088558, U01GM076497, 5U01GM076497)
- National Institute of Environmental Health Sciences (T32ES007142)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- negative control, confounding, spurious association, exposure control, outcome control, observational study validity, recall bias, experimental design, causal inference
- MeSH terms
- Humans, Bias, Causality, Confounding Factors, Epidemiologic, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.