Negative Controls: A Tool for Detecting Confounding and Bias in Observational Studies
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
Noncausal associations between exposures and outcomes are a threat to validity of causal inference in observational studies. Many techniques have been developed for study design and analysis to identify and eliminate such errors. Such problems are not expected to compromise experimental studies, where careful standardization of conditions (for laboratory work) and randomization (for population studies) should, if applied properly, eliminate most such noncausal associations. We argue, however, that a routine precaution taken in the design of biologic laboratory experiments--the use of "negative controls"--is designed to detect both suspected and unsuspected sources of spurious causal inference. In epidemiology, analogous negative controls help to identify and resolve confounding as well as other sources of error, including recall bias or analytic flaws. We distinguish 2 types of negative controls (exposure controls and outcome controls), describe examples of each type from the epidemiologic literature, and identify the conditions for the use of such negative controls to detect confounding. We conclude that negative controls should be more commonly employed in observational studies, and that additional work is needed to specify the conditions under which negative controls will be sensitive detectors of other sources of error in observational studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, in order to evaluate the specificity of the association to ADHD medication, the tenth used selective serotonin reuptake inhibitors (SSRIs) as a negative control medication exposure (27, 30).”
From ADHD Medication and Substance-Related Problems (2017) · cites this paper as [30]
In the citing paper
“The use of ‘negative controls’ for outcomes or exposures can be helpful to assess the likely presence of unmeasured or residual confounding in observational research [51].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [51]
In the citing paper
“In a third approach–negative control studies–we would expect an unbiased negative control study to be null and the ‘real’ study not-null if there is a true causal effect and the assumptions of this approach are met (see Table 1).39,40”
From Triangulation in aetiological epidemiology (2017) · cites this paper as [39]
Citations
Cites
(1979) · Medical Entomology and Zoology · Cited by 11,252
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,285
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · cited 1×
“19 Hill argued that causal inferences were more credible if the exposure (in his example, nickel mining) was associated with only certain types of outcomes (death from lung and nose cancer but not death from other cancers), and if the outcome was associated with one kind of exposure (nickel mining)…”
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844
An introduction to instrumental variables for epidemiologists
Greenland, S. (2000) · International Journal of Epidemiology · doi:10.1093/ije/29.4.722 · Cited by 1,156
Instruments for Causal Inference: An Epidemiologist's Dream?
Hernán, M. A., & Robins, J. M. (2006) · Epidemiology · doi:10.1097/01.ede.0000222409.00878.37 · Cited by 961
Vaccines for preventing influenza in the elderly
Demicheli, V., Jefferson, T., Di Pietrantonj, C., Ferroni, E., Thorning, S., Thomas, R. E., & Rivetti, A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004876.pub4 · Cited by 695
Impact of Influenza Vaccination on Seasonal Mortality in the US Elderly Population
(2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.3.265 · Cited by 543 · cited 1×
“8 To test this hypothesis, Jackson et al 7 reproduced earlier estimates of the protective effect of influenza vaccination, but then repeated the analysis for two sets of negative control outcomes, and showed that the protective effect was observed even in circumstances where the vaccine could not…”
Mortality benefits of influenza vaccination in elderly people: an ongoing controversy
Simonsen, L., Taylor, R. J., Viboud, C., Miller, M. A., & Jackson, L. A. (2007) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(07)70236-0 · Cited by 520 · cited 1×
“However, older age is associated with a less robust immune response to influenza vaccination, and ecological data suggest that the benefits measured in observational studies far exceed the corresponding benefits expected at the population level when influenza vaccination rates have increased among the elderly.(6) Impo…”
Cited by
Items 201–300 of 1,002
Control Outcomes and Exposures for Improving Internal Validity of Nonrandomized Studies
Dusetzina, S. B., Brookhart, M. A., & Maciejewski, M. L. (2015) · Health Services Research · doi:10.1111/1475-6773.12279 · Cited by 78
Danaei, G., García Rodríguez, L. A., Fernandez Cantero, O., & Hernán, M. A. (2013) · Diabetes Care · doi:10.2337/dc12-1756 · Cited by 73
“We also examined the effect of statins on peptic ulcer as a negative outcome control (11) because there is no plausible mechanism that statins would affect risk of peptic ulcer, and all of the observed association between statin use and incident peptic ulcer should be due to bias.”
BackgroundMacCabe, J. H., Sariaslan, A., Almqvist, C., Lichtenstein, P., Larsson, H., & Kyaga, S. (2018) · The British Journal of Psychiatry · doi:10.1192/bjp.2018.23 · Cited by 81
“14 We selected a negative exposure (studying for a law degree) and a negative outcome (diabetes, ICD-8/-9: 250; ICD-10: E10-14) on the basis that they had no known or suspected aetiological associations, but had similar prevalences, to the exposures and outcomes of interest.”
Kutcher, S., Brophy, J. M., Banack, H. R., Kaufman, J. S., & Samuel, M. (2021) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.05.012 · Cited by 87
Ryan, P. B., Schuemie, M. J., Gruber, S., Zorych, I., & Madigan, D. (2013) · Drug Safety · doi:10.1007/s40264-013-0099-6 · Cited by 76
“In this regard, the comparator group is intended to serve as a negative control [47].”
BackgroundBias Analysis for Uncontrolled Confounding in the Health Sciences
Arah, O. A. (2015) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032315-021644 · Cited by 80
Glucagon-like peptide 1 receptor agonists and the risk of incident diabetic retinopathy
Douros, A., Filion, K. B., Yin, H., Yu, O. H., Etminan, M., Udell, J. A., & Azoulay, L. (2018) · Diabetes Care · doi:10.2337/dc17-2280 · Cited by 70
“Exposures To further assess the validity of our findings, we conducted two additional analyses using negative and positive control exposures (24).”
Longitudinal depression or anxiety in mothers and offspring asthma: a Swedish population-based study
Brew, B. K., Lundholm, C., Viktorin, A., Lichtenstein, P., Larsson, H., & Almqvist, C. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyx208 · Cited by 70
“Paternal distress analysis is especially useful for in utero and preconception comparisons, when the paternal exposure is not expected to have a direct association with the outcome.(16) An association between paternal distress and the outcome could imply confounding in the association between maternal distress and off…”
Chiu, Y., Chavarro, J. E., Dickerman, B. A., Manson, J. E., Mukamal, K. J., Rexrode, K. M., Rimm, E. B., & Hernán, M. Á. (2021) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqab100 · Cited by 79
GLP-1RA Use and Thyroid Cancer Risk
Brito, J. P., Herrin, J. S., SWARNA, K. S., Ospina, N. S., Montori, V. M., Toro-Tobón, D., Umpierrez, G. E., Galindo, R. J., Deng, Y., Mickelson, M. M., et al. (2025) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2024.4852 · Cited by 76
Effects of Radon and UV Exposure on Skin Cancer Mortality in Switzerland
Vienneau, D., de Hoogh, K., Hauri, D. D., Vicedo‐Cabrera, A. M., Schindler, C., Huss, A., Röösli, M., & for the SNC Study Group (2017) · Environmental Health Perspectives · doi:10.1289/ehp825 · Cited by 70
“We therefore included glioma (ICD10: C71, definitive primary cause of death) as a negative control outcome (Lipsitch et al. 2010) because it is associated with SEP (Khanolkar et al. 2016).”
BackgroundRan, J., Qiu, H., Sun, S., & Tian, L. (2018) · Environment International · doi:10.1016/j.envint.2018.04.049 · Cited by 73
Shin, A., Koo, B. K., Lee, J. Y., & Kang, E. H. (2024) · BMJ · doi:10.1136/bmj-2024-079475 · Cited by 69
“23 28”
Pharmacoepidemiology for nephrologists (part 2): potential biases and how to overcome them
Fu, E. L., van Diepen, M., Xu, Y., Trevisan, M., Dekker, F. W., Zoccali, C., Jager, K. J., & Carrero, J. J. (2020) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa242 · Cited by 75
“For certain research questions ‘negative control outcomes’ can be used to provide guidance about the presence and magnitude of unmeasured confounding in observational studies [45].”
BackgroundDesai, R. J., Wang, S. V., Sreedhara, S. K., Zabotka, L. E., Khosrow‐Khavar, F., Nelson, J. C., Shi, X., Toh, S., Wyss, R., Patorno, E., et al. (2024) · BMJ · doi:10.1136/bmj-2023-076460 · Cited by 73
“54 55 Inability to replicate the known effect sizes in these analyses could alert investigators to the presence of bias.”
BackgroundHanson, L. L. M., Pentti, J., Nordentoft, M., Xu, T., Rugulies, R., Madsen, I. E. H., Conway, P. M., Westerlund, H., Vahtera, J., Ervasti, J. M., et al. (2023) · The Lancet Public Health · doi:10.1016/s2468-2667(23)00096-8 · Cited by 79
Busby, J., Mills, K., Zhang, S.-D., Liberante, F., & Cardwell, C. (2018) · Breast Cancer Research · doi:10.1186/s13058-017-0928-0 · Cited by 66
“Similarly, we conducted negative control analyses for tricyclic antidepressants and venlafaxine to identify confounding [48], as they have a much weaker effect on prolactin levels [49] and do not interact with tamoxifen metabolism [50, 51] but have similar indications to SSRIs.”
Brynge, M., Sjöqvist, H., Gardner, R. M., Lee, B. K., Dalman, C., & Karlsson, H. (2022) · The Lancet Psychiatry · doi:10.1016/s2215-0366(22)00264-4 · Cited by 67
Ambati, J., Magagnoli, J., Leung, H., Wang, S.-B., Andrews, C. A., Fu, D., Pandey, A., Sahu, S., Narendran, S., Hirahara, S., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-18528-z · Cited by 66
Liew, Z., Kioumourtzoglou, M., Roberts, A. L., O’Reilly, É. J., Ascherio, A., & Weisskopf, M. G. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwy288 · Cited by 69
Chen, S., Chan, Y., Lin, C., Wu, V. C., Cheng, Y., Chen, D., Chang, S., Hung, K., Chu, P., & Chou, A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.0469 · Cited by 70
Arnold, B. F., Schiff, K., Griffith, J. F., Gruber, J. S., Yau, V. M., Wright, C. C., Wade, T. J., Burns, S., Hayes, J. M., McGee, C. D., et al. (2013) · Epidemiology · doi:10.1097/01.ede.0000434431.06765.4a · Cited by 66
Prenatal exposure to outdoor air pollution and child behavioral problems at school age in Japan
Yorifuji, T., Kashima, S., Diez, M. H., Kado, Y., Sanada, S., & Doi, H. (2016) · Environment International · doi:10.1016/j.envint.2016.11.016 · Cited by 68
Fukuda, S., Varshney, A., Fowler, B. J., Wang, S.-B., Narendran, S., Ambati, K., Yasuma, T., Magagnoli, J., Leung, H., Hirahara, S., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2022751118 · Cited by 65
Zhou, Z., Ofori‐Asenso, R., Curtis, A. J., Breslin, M. C., Wolfe, R. S., McNeil, J. J., Murray, A. M., Ernst, M. E., Reid, C. M., Lockery, J. E., et al. (2020) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.05.016 · Cited by 69
“We next checked the validity of the IPTW with a negative control (26).”
MethodXie, Y., Bowe, B. C., & Al‐Aly, Z. (2023) · BMJ · doi:10.1136/bmj-2022-073312 · Cited by 67
“Nirmatrelvir was associated with a reduced risk of”
Addressing unmeasured confounding in comparative observational research
Zhang, X., Faries, D. E., Li, H., Stamey, J. D., & Imbens, G. W. (2018) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4394 · Cited by 74
Abrahami, D., Douros, A., Yin, H., Yu, O. H., Faillie, J.-L., Montastruc, F., Platt, R. W., Bouganim, N., & Azoulay, L. (2018) · BMJ · doi:10.1136/bmj.k4880 · Cited by 63
Konings, B., Villatoro, L. G., den Eynde, J. V., Barahona, G. A., Burns, R., McKnight, M., Hui, K. Y., Yenokyan, G., Tack, J. F., & Pasricha, P. J. (2023) · Gut · doi:10.1136/gutjnl-2023-329685 · Cited by 67
“26 A broader implementation of empirical approaches to evaluate and correct for the presence of systematic error in observational studies is needed.”
BackgroundAntidepressants and Fracture Risk in Older Adults: A Comparative Safety Analysis
Gagne, J. J., Patrick, A. R., Mogun, H., & Solomon, D. H. (2011) · Clinical Pharmacology & Therapeutics · doi:10.1038/clpt.2011.54 · Cited by 62
Kim, S., Kim, H., & Lee, J.-T. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16101866 · Cited by 66
“We also selected lung cancer as the negative control outcome for greenness to detect confounding and other biases [31], and the lack of association with exposure to green space lends confidence to the validity of our findings.”
ResultZhang, L., Zhu, N., Sjölander, A., Nourredine, M., Li, L., Garcia‐Argibay, M., Kuja‐Halkola, R., Brikell, I., Lichtenstein, P., D'Onofrio, B. M., et al. (2025) · BMJ · doi:10.1136/bmj-2024-083658 · Cited by 71
The formal approach to quantitative causal inference in epidemiology: misguided or misrepresented?
Daniel, R., Stavola, B. L. D., & Vansteelandt, S. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyw227 · Cited by 70
Long-term exposure to fine particulate matter and dementia incidence: A cohort study in Hong Kong
Ran, J., Schooling, C. M., Han, L., Sun, S., Zhao, S., Zhang, X., Chan, K.-P., Guo, F., Lee, R. S.-Y., Qiu, Y., et al. (2021) · Environmental Pollution · doi:10.1016/j.envpol.2020.116303 · Cited by 59
“1) as a negative control outcome to detect possible bias that may lead to spurious causal inference (Lipsitch et al., 2010).”
BackgroundVashisht, R., Jung, K., Schuler, A., Banda, J. M., Park, R. W., Jin, S., Li, L., Dudley, J. T., Johnson, K. W., Shervey, M. M., et al. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.1755 · Cited by 61
Abel, K. M., Carr, M., Ashcroft, D. M., Chalder, T., Chew-Graham, C. A., Hope, H. F., Kapur, N., McManus, S., Steeg, S., Webb, R. T., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.34803 · Cited by 63
Acute Effects of Ambient Air Pollution on Asthma Emergency Department Visits in Ten U.S. States
Bi, J., D’Souza, R. R., Moss, S. E., Senthilkumar, N., Russell, A. G., Scovronick, N., Chang, H. H., & Ebelt, S. (2023) · Environmental Health Perspectives · doi:10.1289/ehp11661 · Cited by 61
“46,47 In our case, the negative outcome control was tomor-row ’ s (lag − 1) air pollution concentrations as an indicator in the single-day lag modeling framework.”
Mercadé‐Besora, N., Li, X., Kolde, R., Trinh, N. T., Sanchez-Santos, M. T., Man, W. Y., Roel, E., Reyes, C., Delmestri, A., Nordeng, H., et al. (2024) · Heart · doi:10.1136/heartjnl-2023-323483 · Cited by 66
Recent cannabis use and myocardial infarction in young adults: a cross-sectional study
Ladha, K. S., Mistry, N., Wijeysundera, D. N., Clarke, H. A., Verma, S., Hare, G. M. T., & Mazer, C. D. (2021) · Canadian Medical Association Journal · doi:10.1503/cmaj.202392 · Cited by 68
Colbourne, L., & Harrison, P. J. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01615-6 · Cited by 61
“outcomes for which there are no known or predicted links to the brain penetrability of CCBs) to help assess residual confounding [52, 53].”
BackgroundSystemic hormonal contraception and risk of venous thromboembolism
Heikinheimo, O., Toffol, E., Partonen, T., But, A., Latvala, A., & Haukka, J. (2022) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.14384 · Cited by 58
The burden and dynamics of hospital-acquired SARS-CoV-2 in England
Cooper, B. S., Evans, S., Jafari, Y., Pham, T. M., Mo, Y., Lim, C., Pritchard, M. G., Pople, D., Hall, V. J., Stimson, J. A., et al. (2023) · Nature · doi:10.1038/s41586-023-06634-z · Cited by 61
Physical activity and longevity: how to move closer to causal inference
Wade, K. H., Richmond, R. C., & Davey Smith, G. (2018) · British Journal of Sports Medicine · doi:10.1136/bjsports-2017-098995 · Cited by 57
Reese, H., Routray, P., Torondel, B., Sinharoy, S. S., Mishra, S., Freeman, M. C., Chang, H. H., & Clasen, T. (2019) · International Journal of Epidemiology · doi:10.1093/ije/dyz157 · Cited by 57
“Prevalence of bruising or scrapes (combined) was collected in round 3 (February–June 2016) as a negative control to allow qualitative assessment of differential reporting bias for self-reported outcomes.(25) We used direct observation to assess water, santitation and hygiene (WaSH) infrastructure characteristics.”
MethodOn the Nature of Informative Presence Bias in Analyses of Electronic Health Records
McGee, G., Haneuse, S., Coull, B. A., Weisskopf, M. G., & Rotem, R. S. (2021) · Epidemiology · doi:10.1097/ede.0000000000001432 · Cited by 63
Zang, C., Zhang, Y., Xu, J., Bian, J., Morozyuk, D., Schenck, E. J., Khullar, D., Nordvig, A. S., Shenkman, E. A., Rothman, R. L., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-37653-z · Cited by 62
Burgess, S., & Cronjé, H. T. (2024) · eGastroenterology · doi:10.1136/egastro-2023-100042 · Cited by 65
“We note that both positive and negative controls (including control outcomes, control exposures, control populations and control time periods) can be used in many ways, 49 including in the selection of genetic variants and the validation of results based on the selected variants.”
MethodTielemans, S. M. A. J., de Melker, H. E., Hahné, S. J. M., Boef, A. G. C., van der Klis, F. R. M., Sanders, E. A. M., van der Sande, M. A. B., & Knol, M. J. (2017) · BMJ · doi:10.1136/bmj.j3862 · Cited by 58
Bowe, B. C., Gibson, A. K., Xie, Y., Yan, Y., van Donkelaar, A., Martin, R. V., & Al‐Aly, Z. (2021) · Environmental Health Perspectives · doi:10.1289/ehp7944 · Cited by 61
“Because there is no prior evidence or biologic plausibility for an association between atmospheric sodium and weight or BMI (Lipsitch et al. 2010), associations with ambient sodium concentrations would suggest potential bias in associations with PM2:5 related to the use of air monitoring station…”
BackgroundIng, C. H., Ma, X., Sun, M., Lu, Y., Wall, M. M., Olfson, M., & Li, G. (2020) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000004619 · Cited by 58
Szmulewicz, A., Bateman, B. T., Levin, R., & Huybrechts, K. F. (2021) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2020.20071038 · Cited by 59
Cusimano, M. C., Chiu, M., Ferguson, S. E., Moineddin, R., Aktar, S., Liu, N., & Baxter, N. N. (2021) · BMJ · doi:10.1136/bmj-2021-067528 · Cited by 65
Lewnard, J. A., Bruxvoort, K. J., Fischer, H. J., Hong, V. X., Grant, L. R., Jódar, L., Cane, A. D., Gessner, B. D., & Tartof, S. Y. (2021) · Clinical Infectious Diseases · doi:10.1093/cid/ciab1051 · Cited by 56
Flanders, W. D., Strickland, M. J., & Klein, M. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx013 · Cited by 59
“Ability of the indicator to detect confounding was justified using causal considerations, much like those for negative exposure controls (11) and Granger causality (12).”
InfluentialSarkar, C., Webster, C. J., & Gallacher, J. (2018) · The Lancet Planetary Health · doi:10.1016/s2542-5196(18)30208-0 · Cited by 58
Comparison of rates of dementia among older adult recipients of two, one, or no vaccinations
Wiemken, T. L., Salas, J., Morley, J. E., Hoft, D. F., Jacobs, C., & Scherrer, J. F. (2022) · Journal of the American Geriatrics Society · doi:10.1111/jgs.17606 · Cited by 54
“We also employed a negative outcome control by modeling the association between vaccination and incident back pain in patients without back pain at index.(26)”
MethodLophatananon, A., Carr, M., McMillan, B., Dobson, C. B., Itzhaki, R. F., Parisi, R., Ashcroft, D. M., & Muir, K. R. (2023) · BMC Public Health · doi:10.1186/s12889-023-16768-4 · Cited by 58
“The analysis of influenza vaccination was carried out as a “comparator group” which aimed to help identify and resolve whether there was possible residual confounding [34].”
MethodSujan, A. C., Öberg, A. S., Quinn, P. D., & D'Onofrio, B. M. (2019) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13004 · Cited by 52
Outdoor light at night, overweight, and obesity in school-aged children and adolescents
Lin, L., Zeng, X., Deb, B., Tabet, M., Xu, S.-L., Wu, Q.-Z., Zhou, Y., Ma, H., Chen, D.-H., Chen, G., et al. (2022) · Environmental Pollution · doi:10.1016/j.envpol.2022.119306 · Cited by 60
Strickland, M. J., Klein, M., Flanders, W. D., Chang, H. H., Mulholland, J. A., Tolbert, P. E., & Darrow, L. A. (2014) · Epidemiology · doi:10.1097/ede.0000000000000170 · Cited by 61
Ioannou, G. N., Locke, E. R., Green, P. K., & Berry, K. (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101326 · Cited by 58
Structural stigma and all-cause mortality among sexual minorities: Differences by sexual behavior?
Hatzenbuehler, M. L., Rutherford, C. G., McKetta, S. C., Prins, S. J., & Keyes, K. M. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.112463 · Cited by 61
“, those with only opposite-sex partners) who lived in communities with higher levels of structural stigma, indicating evidence of a negative control exposure (Lipsitch et al., 2010).”
BackgroundIndirect protection of children from SARS-CoV-2 infection through parental vaccination
Hayek, S., Shaham, G., Ben‐Shlomo, Y., Kepten, E., Dagan, N., Nevo, D., Lipsitch, M., Reis, B. Y., Balicer, R. D., & Barda, N. (2022) · Science · doi:10.1126/science.abm3087 · Cited by 53
Self-citationEvidence triangulation in health research
Gutierrez, S., Glymour, M. M., & Smith, G. D. (2025) · European Journal of Epidemiology · doi:10.1007/s10654-024-01194-6 · Cited by 65
“Negative controls can detect the presence of residual confounding, recall bias, and other analytic flaws, including reverse causation and selection bias [8, 16, 28].”
A comparison of Bayesian and Monte Carlo sensitivity analysis for unmeasured confounding
McCandless, L. C., & Gustafson, P. (2017) · Statistics in Medicine · doi:10.1002/sim.7298 · Cited by 61
“The preceding discussion is related to the literature on negative controls for causal inference [25].”
BackgroundRichardson, D. B., Laurier, D., Schubauer-Berigan, M. K., Tchetgen, E. T., & Cole, S. R. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu211 · Cited by 51
Self-citationMacKay, E. N., Dalman, C., Karlsson, H., & Gardner, R. M. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.4257 · Cited by 51
Principles of Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND)
Schuemie, M. J., Ryan, P. B., Pratt, N., Chen, R., You, S. C., Krumholz, H. M., Madigan, D., Hripcsak, G., & Suchard, M. A. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa103 · Cited by 53
Associations in Medical Research Can Be Misleading: A Clinician's Guide to Causal Inference
Karamitros, G., Grant, M. P., & Lamaris, G. A. (2025) · Journal of Surgical Research · doi:10.1016/j.jss.2025.03.043 · Cited by 62
Walsh, L. K., Donelle, J., Dodds, L. A., Hawken, S., Wilson, K. R., Benchimol, E. I., Chakraborty, P. K., Guttmann, A., Kwong, J. C., MacDonald, N. E., et al. (2019) · BMJ · doi:10.1136/bmj.l4151 · Cited by 53
Lip, G. Y. H., Skjøth, F., Nielsen, P. B., Kjældgaard, J. N., & Larsen, T. B. (2017) · JAMA Cardiology · doi:10.1001/jamacardio.2017.1883 · Cited by 55
“To evaluate the potential for residual confounding, falsification analysis was performed by applying the propensity–weighted cohort in the analyses on (falsification) end points that a priori should be expected not to be associated with the effects of treatment.(14) For this study, we considered pneumonia, hip fractur…”
MethodAssessing the impact of unmeasured confounders for credible and reliable real‐world evidence
Zhang, X., Stamey, J. D., & Mathur, M. B. (2020) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.5117 · Cited by 56
Yau, V. M., Schiff, K., Arnold, B. F., Griffith, J. F., Gruber, J. S., Wright, C. C., Wade, T. J., Burns, S., Hayes, J. M., McGee, C. D., et al. (2014) · Water Research · doi:10.1016/j.watres.2014.03.050 · Cited by 50
Ruckart, P. Z., Bove, F., & Maslia, M. L. (2013) · Environmental Health · doi:10.1186/1476-069x-12-104 · Cited by 50
“is no potential uncontrolled confounding or selection bias that would bias the results away from the null [32].”
BackgroundMethodThe limitations of observation studies for decision making regarding drugs efficacy and safety
Gueyffier, F., & Cucherat, M. (2018) · Therapies · doi:10.1016/j.therap.2018.11.001 · Cited by 56
Proposition 8 and Homophobic Bullying in California
Hatzenbuehler, M. L., Shen, Y., Vandewater, E. A., & Russell, S. T. (2019) · PEDIATRICS · doi:10.1542/peds.2018-2116 · Cited by 53
Design considerations in an active medical product safety monitoring system.
Gagne, J. J., Fireman, B., Ryan, P. B., Maclure, M., Gerhard, T., Toh, S., Rassen, J. A., Nelson, J. C., & Schneeweiss, S. (2012) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.2316 · Cited by 49
Mason, K., Alexiou, A., Bennett, D. L., Summerbell, C., Barr, B., & Taylor-Robinson, D. C. (2021) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2020-216064 · Cited by 57
Ioannou, G. N., Bohnert, A. S. B., O’Hare, A. M., Boyko, E. J., Maciejewski, M. L., Smith, V. A., Bowling, C. B., Viglianti, E. M., Iwashyna, T. J., Hynes, D. M., et al. (2022) · Annals of Internal Medicine · doi:10.7326/m22-1856 · Cited by 55
“We compared as a negative outcome control (34) the incidence of SARS-CoV-2 infection within 10 days after time zero in both groups, which should not be affected by vaccination, to verify there was no uncontrolled residual confounding after matching or any ascertainment bias.”
ResultQuasi-experimental evaluation of a nationwide diabetes prevention programme
Lemp, J. M., Bommer, C., Xie, M., Michalik, F., Jani, A. R., Davies, J. I., Bärnighausen, T., Vollmer, S., & Geldsetzer, P. (2023) · Nature · doi:10.1038/s41586-023-06756-4 · Cited by 52
“Following this, we performed a set of extra robustness checks following the approach by others to further investigate if there are any other inherent sources of bias or confounding 34 .”
BackgroundMethodValproate Use During Spermatogenesis and Risk to Offspring
Christensen, J., Trabjerg, B. B., & Dreier, J. W. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.14709 · Cited by 51
Maternal Smoking in Pregnancy and Offspring Depression: a cross cohort and negative control study
Taylor, A. E., Carslake, D., de Mola, C. L., Rosenqvist, M. A., Nilsen, T. I. L., Bjørngaard, J. H., Horta, B. L., Pearson, R. M., Rai, D., Galanti, M. R., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-11836-3 · Cited by 55
Lloyd, E. C., Haase, A. M., Foster, C. E., & Verplanken, B. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.05.010 · Cited by 42
Moon, A. M., Jiang, Y., Rogal, S. S., Tapper, E. B., Lieber, S. R., & Barritt, A. S. (2020) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.15639 · Cited by 49
Cheung, K. S., Chan, E. W., Seto, W., Wong, I. C. K., & Leung, W. K. (2020) · Hypertension · doi:10.1161/hypertensionaha.120.15317 · Cited by 50
“We used CCBs (calcium channel blockers; dihydropyridine and nondihydropyridine) and diuretics (loop diuretics, thiazide diuretics, aldosterone antagonists, and potassium-sparing diuretics) as negative control exposures, as these 2 classes of anti-hypertensives were not reported to be associated with CRC development.(4…”
BackgroundMethodXie, Y., Bowe, B., Gibson, A. K., McGill, J. B., Maddukuri, G., & Al-Aly, Z. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2488 · Cited by 48
Zürcher, K., Zwahlen, M., Ballif, M., Rieder, H. L., Egger, M., & Fenner, L. (2016) · PLoS ONE · doi:10.1371/journal.pone.0162575 · Cited by 52
Post–Modern Epidemiology: When Methods Meet Matter
Davey Smith, G. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwz064 · Cited by 46
“18–19), is the basis of the nowlauded “negative controls” (79, 80) will not be lost on those familiarwith earlier uses—from interrogatingmidcentury occupational exposures (discussed by Greenwood in 1948 (81)) through the many subsequent applications (71).”
BackgroundContrast-associated acute kidney injury is a myth: Yes
Ehrmann, S., Aronson, D., & Hinson, J. S. (2017) · Intensive Care Medicine · doi:10.1007/s00134-017-4950-6 · Cited by 48
Has reducing fine particulate matter and ozone caused reduced mortality rates in the United States?
Cox, L. A., & Popken, D. A. (2014) · Annals of Epidemiology · doi:10.1016/j.annepidem.2014.11.006 · Cited by 50
Maternal smoking during pregnancy and autism: using causal inference methods in a birth cohort study
Caramaschi, D., Taylor, A. E., Richmond, R. C., Havdahl, K. A., Golding, J., Relton, C. L., Munafò, M. R., Davey Smith, G., & Rai, D. (2018) · Translational Psychiatry · doi:10.1038/s41398-018-0313-5 · Cited by 43
Zhou, Y., Sinnathamby, V., Yu, Y., Sikora, L., Johnson, C. Y., Mossey, P. A., & Little, J. (2020) · Birth Defects Research · doi:10.1002/bdr2.1827 · Cited by 57
Xie, Y., Bowe, B., Gibson, A. K., McGill, J. B., Maddukuri, G., & Al‐Aly, Z. (2021) · Journal of the American Heart Association · doi:10.1161/jaha.120.020237 · Cited by 45
Marcum, Z. A., Cohen, J. B., Zhang, C., Derington, C. G., Greene, T. H., Ghazi, L., Herrick, J. S., King, J. B., Cheung, A. K., Bryan, N., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.45319 · Cited by 46
“84) Protocol-defined MCI or death 256 (26) 226 (34) 0.”
A systematic review on the effects of nanomaterials on gut microbiota
Utembe, W., Tlotleng, N., & Kamng’ona, A. W. (2022) · Current Research in Microbial Sciences · doi:10.1016/j.crmicr.2022.100118 · Cited by 57
“The use of negative controls would detect both suspected and unsuspected sources of spurious causal inferences (Lipsitch et al., 2010).”
BackgroundDoes the Current Definition of Contrast‐induced Acute Kidney Injury Reflect a True Clinical Entity?
Sinert, R. H., Brandler, E. S., Subramanian, R. A., & Miller, A. C. (2012) · Academic Emergency Medicine · doi:10.1111/acem.12011 · Cited by 46
Effects of Fluoroquinolones on Outcomes of Patients With Aortic Dissection or Aneurysm
Chen, S., Chan, Y., Wu, V. C., Cheng, Y., Chen, D., Lin, C., Hung, K., Chang, S., Chu, P., & Chou, A. (2021) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2021.02.047 · Cited by 46
Antioxidant supplementation, redox deficiencies and exercise performance: A falsification design
Margaritelis, N. V., Paschalis, V., Theodorou, A. A., Kyparos, A., & Nikolaidis, M. G. (2020) · Free Radical Biology and Medicine · doi:10.1016/j.freeradbiomed.2020.06.029 · Cited by 45
Arambepola, R., Yang, Y., Hutchinson, K., Mwansa, F. D., Doherty, J. A., Bwalya, F., Ndubani, P., Musukwa, G., Moss, W. J., Wesolowski, A., et al. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2021-007479 · Cited by 53
Short term exposure to air pollution and mortality in the US: a double negative control analysis
Liu, R., Wei, Y., Qiu, X., Kosheleva, A., & Schwartz, J. D. (2022) · Environmental Health · doi:10.1186/s12940-022-00886-4 · Cited by 49
Metadata
- Authors
- Lipsitch, M., Tchetgen Tchetgen, E., & Cohen, T.
- Affiliations
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Lipsitch, Marc
- Venue
- Epidemiology
- Published
- 2010-05
- Volume
- 21
- Issue
- 3
- Pages
- 383-388
- ISSN
- 1044-3983 (ISSN-L)
- PMC
- PMC3053408
- OpenAlex
- W2009428399
- Cited by
- 1,583
- References
- 25
- Citations by year
- 2026: 278 · 2025: 215 · 2024: 183 · 2023: 138 · 2022: 153 · 2021: 132 · 2020: 107 · 2019: 75 · 2018: 72 · 2017: 71 · 2016: 56 · 2015: 32 · 2014: 26 · 2013: 25 · 2012: 11
- FWCI
- 11.64
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Funders
- National Institutes of Health (5u01gm076497-03, 1dp2od006663-01)
- National Institute of General Medical Sciences (U54GM088558, U01GM076497, 5U01GM076497)
- National Institute of Environmental Health Sciences (T32ES007142)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- negative control, confounding, spurious association, exposure control, outcome control, observational study validity, recall bias, experimental design, causal inference
- MeSH terms
- Humans, Bias, Causality, Confounding Factors, Epidemiologic, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.