Negative Controls: A Tool for Detecting Confounding and Bias in Observational Studies
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
Noncausal associations between exposures and outcomes are a threat to validity of causal inference in observational studies. Many techniques have been developed for study design and analysis to identify and eliminate such errors. Such problems are not expected to compromise experimental studies, where careful standardization of conditions (for laboratory work) and randomization (for population studies) should, if applied properly, eliminate most such noncausal associations. We argue, however, that a routine precaution taken in the design of biologic laboratory experiments--the use of "negative controls"--is designed to detect both suspected and unsuspected sources of spurious causal inference. In epidemiology, analogous negative controls help to identify and resolve confounding as well as other sources of error, including recall bias or analytic flaws. We distinguish 2 types of negative controls (exposure controls and outcome controls), describe examples of each type from the epidemiologic literature, and identify the conditions for the use of such negative controls to detect confounding. We conclude that negative controls should be more commonly employed in observational studies, and that additional work is needed to specify the conditions under which negative controls will be sensitive detectors of other sources of error in observational studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, in order to evaluate the specificity of the association to ADHD medication, the tenth used selective serotonin reuptake inhibitors (SSRIs) as a negative control medication exposure (27, 30).”
From ADHD Medication and Substance-Related Problems (2017) · cites this paper as [30]
In the citing paper
“The use of ‘negative controls’ for outcomes or exposures can be helpful to assess the likely presence of unmeasured or residual confounding in observational research [51].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [51]
In the citing paper
“In a third approach–negative control studies–we would expect an unbiased negative control study to be null and the ‘real’ study not-null if there is a true causal effect and the assumptions of this approach are met (see Table 1).39,40”
From Triangulation in aetiological epidemiology (2017) · cites this paper as [39]
Citations
Cites
(1979) · Medical Entomology and Zoology · Cited by 11,252
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,285
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · cited 1×
“19 Hill argued that causal inferences were more credible if the exposure (in his example, nickel mining) was associated with only certain types of outcomes (death from lung and nose cancer but not death from other cancers), and if the outcome was associated with one kind of exposure (nickel mining)…”
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844
An introduction to instrumental variables for epidemiologists
Greenland, S. (2000) · International Journal of Epidemiology · doi:10.1093/ije/29.4.722 · Cited by 1,156
Instruments for Causal Inference: An Epidemiologist's Dream?
Hernán, M. A., & Robins, J. M. (2006) · Epidemiology · doi:10.1097/01.ede.0000222409.00878.37 · Cited by 961
Vaccines for preventing influenza in the elderly
Demicheli, V., Jefferson, T., Di Pietrantonj, C., Ferroni, E., Thorning, S., Thomas, R. E., & Rivetti, A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004876.pub4 · Cited by 695
Impact of Influenza Vaccination on Seasonal Mortality in the US Elderly Population
(2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.3.265 · Cited by 543 · cited 1×
“8 To test this hypothesis, Jackson et al 7 reproduced earlier estimates of the protective effect of influenza vaccination, but then repeated the analysis for two sets of negative control outcomes, and showed that the protective effect was observed even in circumstances where the vaccine could not…”
Mortality benefits of influenza vaccination in elderly people: an ongoing controversy
Simonsen, L., Taylor, R. J., Viboud, C., Miller, M. A., & Jackson, L. A. (2007) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(07)70236-0 · Cited by 520 · cited 1×
“However, older age is associated with a less robust immune response to influenza vaccination, and ecological data suggest that the benefits measured in observational studies far exceed the corresponding benefits expected at the population level when influenza vaccination rates have increased among the elderly.(6) Impo…”
Cited by
Items 101–200 of 1,002
Gardner, R. M., Lee, B. K., Magnusson, C., Rai, D., Frisell, T., Karlsson, H., Idring, S., & Dalman, C. (2015) · International Journal of Epidemiology · doi:10.1093/ije/dyv081 · Cited by 141
“We used paternal BMI data collected at the time of conscription to the Swedish military to determine whether maternal BMI associations with ASD were independent of other familial factors.(7,8) Follow-up data on paternal BMI proximal to the birth of the index child were not available.”
MethodUse of administrative data in healthcare research
Mazzali, C., & Duca, P. (2015) · Internal and Emergency Medicine · doi:10.1007/s11739-015-1213-9 · Cited by 141
“Negative controls are used in biology to rule out possible non-causal interpretations of results, but they can also be used in other observational studies, as discussed in [20].”
Long-term gastrointestinal outcomes of COVID-19
Xu, E., Xie, Y., & Al‐Aly, Z. (2023) · Nature Communications · doi:10.1038/s41467-023-36223-7 · Cited by 156
Smoking in Pregnancy and Child ADHD
Gustavson, K., Ystrom, E., Stoltenberg, C., Susser, E., Surén, P., Magnus, P., Knudsen, G. P., Smith, G. D., Langley, K., Rutter, M., et al. (2017) · Pediatrics · doi:10.1542/peds.2016-2509 · Cited by 125
“The negative control approach compares the association of interest with another association that is affected by much of the same confounding factors, but that is not suspected to be causal 13,14 .”
Wu, C.-Y., Iskander, C., Wang, C., Xiong, L. Y., Shah, B. R., Edwards, J. D., Kapral, M. K., Herrmann, N., Lanctôt, K. L., Masellis, M., et al. (2023) · Diabetes Care · doi:10.2337/dc22-1705 · Cited by 139
Adjusting for unmeasured confounding in nonrandomized longitudinal studies: a methodological review
Streeter, A., Lin, N. X., Crathorne, L., Haasova, M., Hyde, C., Melzer, D. R., & Henley, W. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.04.022 · Cited by 159
Approaches for drawing causal inferences from epidemiological birth cohorts: A review
Richmond, R. C., Al-Amin, A., Davey Smith, G., & Relton, C. L. (2014) · Early Human Development · doi:10.1016/j.earlhumdev.2014.08.023 · Cited by 134
“A negative control situation is one that cannot involve the hypothesised causal mechanism, but which is likely to involve the same sources of bias or confounding as in the original association [18,35,36].”
BackgroundHripcsak, G., Suchard, M. A., Shea, S., Chen, R., You, S. C., Pratt, N., Madigan, D., Krumholz, H. M., Ryan, P. B., & Schuemie, M. J. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2019.7454 · Cited by 139
Air Pollution and Autism Spectrum Disorders: Causal or Confounded?
Weisskopf, M. G., Kioumourtzoglou, M.-A., & Roberts, A. L. (2015) · Current Environmental Health Reports · doi:10.1007/s40572-015-0073-9 · Cited by 127
Association between air pollution and type 2 diabetes: an updated review of the literature
Li, Y., Xu, L., Shan, Z., Teng, W., & Han, C. (2019) · Therapeutic Advances in Endocrinology and Metabolism · doi:10.1177/2042018819897046 · Cited by 146
Arnold, B. F., Ercumen, A., Benjamin-Chung, J., & Colford, J. M. (2016) · Epidemiology · doi:10.1097/ede.0000000000000504 · Cited by 130
Mok, P. L. H., Carr, M., Guthrie, B., Morales, D. R., Sheikh, A., Elliott, R. A., Camacho, E. M., van Staa, T. P., Avery, A. J., & Ashcroft, D. M. (2024) · BMJ · doi:10.1136/bmj-2023-076268 · Cited by 152
“53 These outcomes were chosen because evidence of an association with antipsychotic use is lacking from the literature.”
BackgroundHyams, C., Marlow, R., Maseko, Z., King, J., Ward, L., Fox, K., Heath, R., Tuner, A., Friedrich, Z., Morrison, L., et al. (2021) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(21)00330-3 · Cited by 131
Mason, K. E., Pearce, N., & Cummins, S. (2018) · The Lancet Public Health · doi:10.1016/s2468-2667(17)30212-8 · Cited by 128
How to detect and reduce potential sources of biases in studies of SARS-CoV-2 and COVID-19
Accorsi, E. K., Qiu, X., Rumpler, E., Kennedy-Shaffer, L., Kahn, R., Joshi, K., Goldstein, E., Stensrud, M. J., Niehus, R., Cevik, M., et al. (2021) · European Journal of Epidemiology · doi:10.1007/s10654-021-00727-7 · Cited by 124
“Moreover, negative control outcomes [34] (e.”
Self-citationMortality caused by tropical cyclones in the United States
Young, R. M., & Hsiang, S. (2024) · Nature · doi:10.1038/s41586-024-07945-5 · Cited by 159
“Thus, these tests allow us to examine whether, in each case, the remaining structure generates artefacts in the model that would produce a spurious result, also known as a negative exposure control 70 .”
Lipsitch, M., Jha, A., & Simonsen, L. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyw124 · Cited by 145
“Moreover, under certain assumptions the direction of bias should be the same for the outcome of interest (D) as for the negative control (D’): in this case, the measured VE against death should be biased upwards.(22) The assumptions underlying the valid use of a negativecontrol outcome are: (i) that the exposure does…”
BackgroundResultSelf-citationTriangulating Evidence through the Inclusion of Genetically Informed Designs
Munafò, M. R., Higgins, J. P. T., & Smith, G. D. (2021) · Cold Spring Harbor Perspectives in Medicine · doi:10.1101/cshperspect.a040659 · Cited by 125
Pleiotropy-robust Mendelian randomization.
van Kippersluis, H., & Rietveld, C. A. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyx002 · Cited by 125
Promises and pitfalls of electronic health record analysis
Farmer, R. E., Mathur, R., Bhaskaran, K., Eastwood, S. V., Chaturvedi, N., & Smeeth, L. L. (2017) · Diabetologia · doi:10.1007/s00125-017-4518-6 · Cited by 136
“If such options are not available, a negative control can be an informative way of investigating the impact of unmeasured confounding [45].”
BackgroundDouros, A., Dell’Aniello, S., Yu, O. H. Y., Filion, K. B., Azoulay, L., & Suissa, S. (2018) · BMJ · doi:10.1136/bmj.k2693 · Cited by 135
Colford, J. M., Schiff, K., Griffith, J. F., Yau, V., Arnold, B. F., Wright, C. C., Gruber, J. S., Wade, T. J., Burns, S., Hayes, J. M., et al. (2012) · Water Research · doi:10.1016/j.watres.2012.01.033 · Cited by 123
Abrahami, D., Douros, A., Yin, H., Yu, O. H. Y., Renoux, C., Bitton, A., & Azoulay, L. (2018) · BMJ · doi:10.1136/bmj.k872 · Cited by 119
Sarvet, A. L., Wall, M. M., Keyes, K. M., Olfson, M., Cerdá, M., & Hasin, D. S. (2018) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2018.01.009 · Cited by 128
Early life exposures: impact on asthma and allergic disease
Kozyrskyj, A. L., Bahreinian, S., & Azad, M. B. (2011) · Current Opinion in Allergy & Clinical Immunology · doi:10.1097/aci.0b013e328349b166 · Cited by 117
Intraseason Waning of Influenza Vaccine Effectiveness
Ray, G. T., Lewis, N., Klein, N. P., Daley, M. F., Wang, S. V., Kulldorff, M., & Fireman, B. (2019) · Clinical Infectious Diseases · doi:10.1093/cid/ciy770 · Cited by 118
“A negative-control outcome should be one that is subject to the same source of bias as the outcome of interest but is not affected by the exposure of interest [21, 27].”
BackgroundThe Control Outcome Calibration Approach for Causal Inference With Unobserved Confounding
Tchetgen Tchetgen, E. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwt303 · Cited by 119
Franklin, J. M., Pawar, A., Martin, D., Glynn, R. J., Levenson, M., Temple, R., & Schneeweiss, S. (2020) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1633 · Cited by 119
Casas, M., Chatzi, L., Carsin, A.-E., Amiano, P., Guxens, M., Kogevinas, M., Koutra, K., Lertxundi, N., Murcia, M., Rebagliato, M., et al. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dyt002 · Cited by 115
DNA methylation as a marker for prenatal smoke exposure in adults
Richmond, R. C., Suderman, M., Langdon, R., Relton, C. L., & Davey Smith, G. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyy091 · Cited by 118
Butler, A. M., Brown, D. S., Durkin, M. J., Sahrmann, J. M., Nickel, K. B., O’Neil, C. A., Olsen, M. A. A., Hyun, D. Y., Zetts, R. M., & Newland, J. G. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.14153 · Cited by 124
Arnold, B. F., & Ercümen, A. (2016) · JAMA · doi:10.1001/jama.2016.17700 · Cited by 118
In utero exposure to selective serotonin reuptake inhibitors and risk for autism spectrum disorder
Gidaya, N. B., Lee, B. K., Burstyn, I., Yudell, M., Mortensen, E. L., & Newschaffer, C. J. (2014) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-014-2128-4 · Cited by 115
“Negative controls are used to indicate that potential confounding may exist, if a result with a non-causal explanation is observed (Lipsitch et al. 2010).”
BackgroundPrice, M. A., Weisz, J. R., McKetta, S., Hollinsaid, N. L., Lattanner, M. R., Reid, A. E., & Hatzenbuehler, M. L. (2022) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2021.07.808 · Cited by 115
“, depression) for each RCT, consistent with best practices.(34) However, most studies included multiple ESs, and thus violated the assumption of independent ESs.”
ResultSanderson, E., Richardson, T. G., Hemani, G., & Davey Smith, G. (2021) · International Journal of Epidemiology · doi:10.1093/ije/dyaa288 · Cited by 115
Ercumen, A., Naser, A. M., Unicomb, L., Arnold, B. F., Colford Jr., J. M., & Luby, S. P. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121907 · Cited by 106
“In addition to diarrhea, the field team recorded caregiver-reported prevalence of skin rashes and ear infections to serve as negative control outcomes [37]; these were symptoms that could not plausibly be affected by the drinking water interventions and were used to detect potential differential reporting bias associa…”
BackgroundLawlor, D. A. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dys209 · Cited by 114
Perspective: Big Data and Machine Learning Could Help Advance Nutritional Epidemiology
Morgenstern, J., Rosella, L. C., Costa, A. P., de Souza, R. J., & Anderson, L. N. (2020) · Advances in Nutrition · doi:10.1093/advances/nmaa183 · Cited by 127
Semiparametric Proximal Causal Inference
Cui, Y., Pu, H., Shi, X., Miao, W., & Tchetgen, E. T. (2023) · Journal of the American Statistical Association · doi:10.1080/01621459.2023.2191817 · Cited by 197
“…meth-ods that leverage certain types of proxies known as negative control variables to mitigate confounding bias in analysis of observational data (Lipsitch, Tchetgen, and Cohen 2010; Kuroki and Pearl 2014; Miao, Geng, and Tchetgen Tchetgen 2018; Shi et al. 2019; Shi, Miao, and Tchetgen Tchetgen…”
BackgroundErcumen, A., Arnold, B. F., Kumpel, E., Burt, Z., Ray, I., Nelson, K., & Colford, J. M. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001892 · Cited by 112
“We specified caregiver-reported cough/cold (combined) and scrapes/bruises (combined) in children aged<5 y as negative control outcomes [20].”
BackgroundRiedel, C., Schönberger, K., Yang, S., Koshy, G., Chen, Y., Gopinath, B., Ziebarth, S., & von Kries, R. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu150 · Cited by 123
“The main objective of our meta-analyses is not to quantify the independent effects of maternal and paternal/ household smoking on the risk of overweight or obesity in the offspring but to compare differences of their effect estimates as a negative control approach with paternal smoking acting as a negative control.(42…”
BackgroundTransparent reporting of observational studies emulating a target trial: the TARGET Statement
Cashin, A. G., Hansford, H. J., Hernán, M. Á., Swanson, S. A., Lee, H., Jones, M. D., Dahabreh, I. J., Dickerman, B. A., Egger, M., García‐Albéniz, X., et al. (2025) · BMJ · doi:10.1136/bmj-2025-087179 · Cited by 123
“Sensitivity analyses may include outcome or population controls 31 and quantitative bias analysis.”
Tustin, A. W., Hirsch, A. G., Rasmussen, S. G., Casey, J. A., Bandeen‐Roche, K., & Schwartz, B. S. (2016) · Environmental Health Perspectives · doi:10.1289/ehp281 · Cited by 108
Casey, J. A., Karasek, D., Ogburn, E. L., Goin, D. E., Dang, K., Braveman, P. A., & Morello-Frosch, R. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwy110 · Cited by 98
“Finally, we implemented a negative exposure control (46) by randomly selecting 8 new oil and coal power plants in California that had not retired during our study period, assigning them the retirement dates of the original 8 retired power plants, and repeating our main analyses (WebAppendix 2).”
MethodOu, S.-M., Shih, C.-J., Chao, P.-W., Chu, H., Kuo, S.-C., Lee, Y.-J., Wang, S.-J., Yang, C.-Y., Lin, C.-C., Chen, T.-J., et al. (2015) · Annals of Internal Medicine · doi:10.7326/m15-0308 · Cited by 101
Shi, X., Miao, W., Nelson, J. C., & Tchetgen, E. J. T. (2020) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12361 · Cited by 151
“Negative controls have a long-standing tradition in laboratory sciences and epidemiology to rule out non-causal explanations of empirical findings (Rosenbaum, 1989; Weiss, 2002; Lipsitch et al., 2010; Glass, 2014).”
BackgroundXie, Y., Bowe, B., Xian, H., Loux, T., McGill, J. B., & Al-Aly, Z. (2023) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(23)00171-7 · Cited by 104
Sarkar, C., Webster, C. J., & Gallacher, J. (2018) · International Journal of Hygiene and Environmental Health · doi:10.1016/j.ijheh.2018.01.009 · Cited by 107
Theoretical Framework for Retrospective Studies of the Effectiveness of SARS-CoV-2 Vaccines
Lewnard, J. A., Patel, M. M., Jewell, N. P., Verani, J. R., Kobayashi, M., Tenforde, M. W., Dean, N. E., Cowling, B. J., & Lopman, B. A. (2021) · Epidemiology · doi:10.1097/ede.0000000000001366 · Cited by 103
Bowe, B. C., Xie, Y., Gibson, A. K., Cai, M., van Donkelaar, A., Martin, R. V., Burnett, R. T., & Al‐Aly, Z. (2021) · Environment International · doi:10.1016/j.envint.2021.106564 · Cited by 108
“We tested ambient sodium concentration as a negative exposure control where there is no prior evidence nor biologic plausibility of an association between atmospheric sodium and hospitalization (Lipsitch et al., 2010).”
Theoretical framework for retrospective studies of the effectiveness of SARS-CoV-2 vaccines
(2021) · doi:10.1101/2021.01.21.21250258 · Cited by 103
Clarifying questions about “risk factors”: predictors versus explanation
Schooling, C. M., & Jones, H. E. (2018) · Emerging Themes in Epidemiology · doi:10.1186/s12982-018-0080-z · Cited by 111
Cooke, S. J., Birnie‐Gauvin, K., Lennox, R. J., Taylor, J. J., Rytwinski, T., Rummer, J. L., Franklin, C. E., Bennett, J., & Haddaway, N. (2017) · Conservation Physiology · doi:10.1093/conphys/cox043 · Cited by 115
“This is also known as non-causal association, and can pose a threat to the validity of a study (Lipsitch et al. , 2010).”
BackgroundSmaller Hippocampal Volume Among Black and Latinx Youth Living in High-Stigma Contexts
Hatzenbuehler, M. L., Weissman, D. G., McKetta, S. C., Lattanner, M. R., Ford, J. V., Barch, D. M., & McLaughlin, K. A. (2021) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2021.08.017 · Cited by 104
Chen, H., Burnett, R. T., Copes, R., Kwong, J. C., Villeneuve, P. J., Goldberg, M. S., Brook, R. D., van Donkelaar, A., Jerrett, M., Martin, R. V., et al. (2016) · Environmental Health Perspectives · doi:10.1289/ehp185 · Cited by 97
Miano, T. A., Hennessy, S., Yang, W. P., Dunn, T. G., Weisman, A. R., Oniyide, O., Agyekum, R. S., Turner, A. P., Ittner, C. A. G., Anderson, B. J., et al. (2022) · Intensive Care Medicine · doi:10.1007/s00134-022-06811-0 · Cited by 100
Modelling estimates of the burden of respiratory syncytial virus infection in children in the UK
Taylor, S., Taylor, R. J., Lustig, R. L., Schuck‐Paim, C., Haguinet, F., Webb, D. J., Logie, J. W., Matias, G., & Fleming, D. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-009337 · Cited by 106
“Negative outcome controls may be useful to detect confounding.(20) The expected”
BackgroundCausation and causal inference
Hoggatt, K. J., Greenland, S., & VanderWeele, T. J. (2015) · Oxford Textbook of Global Public Health · doi:10.1093/med/9780199661756.003.0117 · Cited by 95
Proton Pump Inhibitor Use and Risk of Serious Infections in Young Children
Lassalle, M., Zureik, M., & Dray‐Spira, R. (2023) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2023.2900 · Cited by 104
A confounding bridge approach for double negative control inference on causal effects
Miao, W., Shi, X., Li, Y., & Tchetgen, E. J. T. (2024) · Statistical Theory and Related Fields · doi:10.1080/24754269.2024.2390748 · Cited by 125
“A special case is the nondifferential assumption of Lipsitch et al. (2010) and Tchetgen Tchetgen (2014), which further requires W ⊥⊥ Y | U and does not allow for extra confounders of W –”
InfluentialMethods to control for unmeasured confounding in pharmacoepidemiology: an overview
Uddin, M. J., Groenwold, R. H. H., Ali, M. S., de Boer, A., Roes, K. C. B., Chowdhury, M. A. B., & Klungel, O. H. (2016) · International Journal of Clinical Pharmacy · doi:10.1007/s11096-016-0299-0 · Cited by 104
“If the NC is empirically associated with the exposure (or outcome) after adjustment for measured confounders, the observed association may be the result of unmeasured confounding [44, 47, 48].”
Seo, S. I., Park, C. H., You, S. C., Kim, J., Lee, K. J., Kim, J., Kim, Y., Yoo, J. J., Seo, W., Lee, H., et al. (2021) · Gut · doi:10.1136/gutjnl-2020-323845 · Cited by 103
A Method for Detection of Residual Confounding in Time-series and Other Observational Studies
Flanders, W. D., Klein, M., Darrow, L. A., Strickland, M. J., Sarnat, S. E., Sarnat, J. A., Waller, L. A., Winquist, A., & Tolbert, P. E. (2011) · Epidemiology · doi:10.1097/ede.0b013e3181fdcabe · Cited by 91
Ivers, L. C., Hilaire, I. J., Teng, J. E., Almazor, C. P., Jerome, J.-G., Ternier, R., Boncy, J., Buteau, J., Murray, M. B., Harris, J. B., et al. (2015) · The Lancet Global Health · doi:10.1016/s2214-109x(14)70368-7 · Cited by 95
Xian, Y., Xu, H., O’Brien, E. C., Shah, S., Thomas, L., Pencina, M. J., Fonarow, G. C., Olson, D. M., Schwamm, L. H., Bhatt, D. L., et al. (2019) · JAMA Neurology · doi:10.1001/jamaneurol.2019.2099 · Cited by 92
Processing of meats and cardiovascular risk: time to focus on preservatives
Micha, R., Michas, G., Lajous, M., & Mozaffarian, D. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-136 · Cited by 93
“A second method to evaluate potential confounding is use of a ‘negative control’, that is, a health outcome on which the risk factor of interest would have little plausible effect [11].”
MethodZhang, X., Wong, G. L., Yip, T. C., Tse, Y., Liang, L. Y., Hui, V. W., Lin, H., Li, G., Lai, J. C., Chan, H. L., et al. (2022) · Hepatology · doi:10.1002/hep.32294 · Cited by 93
de Kraker, M. E. A., Abbas, M., Huttner, B. D., & Harbarth, S. (2017) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2017.05.019 · Cited by 96
Thorrington, D., Andrews, N. J., Stowe, J., Miller, E., & van Hoek, A. J. (2018) · BMC Medicine · doi:10.1186/s12916-018-1004-z · Cited by 96
“To account for biases arising from potential secular trends in admission practice over the study period, the IRRs of each disease endpoint were compared to the IRRs of five control conditions that should not be affected by changes in the introduction of the PCV programme [11].”
MethodTrevisan, M., Fu, E. L., Xu, Y., Savarese, G., Dekker, F. W., Lund, L. H., Clase, C. M., Sjölander, A., & Carrero, J. J. (2021) · European Journal of Heart Failure · doi:10.1002/ejhf.2287 · Cited by 99
Elhakeem, A., Hughes, R. A., Tilling, K. M., Cousminer, D. L., Jackowski, S. A., Cole, T. J., Kwong, A. S. F., Li, Z., Grant, S. F. A., Baxter-Jones, A. D. G., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01542-8 · Cited by 104
“Useful strategies for checking if residual confounding influences results include negative control variables and comparing cohorts with different confounding structures [38, 67-69].”
BackgroundZhang, M., Mueller, N. T., Wang, H., Hong, X., Appel, L. J., & Wang, X. (2018) · Hypertension · doi:10.1161/hypertensionaha.117.10944 · Cited by 91
Lin, S.-Y., Ju, S.-W., Lin, C., Hsu, W., Lin, C., Ting, I., & Kao, C. (2020) · Environmental Pollution · doi:10.1016/j.envpol.2020.114154 · Cited by 93
De Giorgi, R., Koychev, I., Adler, A. I., Cowen, P. J., Harmer, C. J., Harrison, P. J., & Taquet, M. (2024) · eClinicalMedicine · doi:10.1016/j.eclinm.2024.102726 · Cited by 78
“…with speci fi c codes in Appendix, pp 15 To assess for possible unmeasured confounding, we also estimated the risk of 15 negative control outcomes 25 (NCOs, i.e., outcomes such as dog bite or wrist/hand fracture that are not expected to be affected by differences between choices of antidiabetic…”
Matias, G., Taylor, R. J., Haguinet, F., Schuck‐Paim, C., Lustig, R. L., & Fleming, D. (2016) · BMC Public Health · doi:10.1186/s12889-016-3128-4 · Cited by 98
“However, because we used a control outcome (hospitalisation for accidents) to detect confounding [23], the expected absence of attributions for the negative control provides support for the modelling methodology employed.”
MethodSanderson, E. C. M., Macdonald-Wallis, C. M., & Smith, G. D. (2017) · International Journal of Epidemiology · doi:10.1093/ije/dyx213 · Cited by 97
“The results obtained from the negative control exposure study should instead contribute to a triangulation of evidence from a range of sources which are subject to different sources of potential bias, and feed into the overall result.(1,2,11) Key Messages”
BackgroundMethodMatthay, E. C., Hagan, E., Gottlieb, L. M., Tan, M. L., Vlahov, D., Adler, N. E., & Glymour, M. M. (2019) · SSM - Population Health · doi:10.1016/j.ssmph.2019.100526 · Cited by 91
Negative Control Exposures in Epidemiologic Studies
Smith, G. D. (2012) · Epidemiology · doi:10.1097/ede.0b013e318245912c · Cited by 88
Raz, R., Levine, H., Pinto, O., Broday, D. M., Yuval, & Weisskopf, M. G. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwx294 · Cited by 88
Sofer, T., Richardson, D. B., Colicino, E., Schwartz, J. D., & Tchetgen, E. J. T. (2016) · Statistical Science · doi:10.1214/16-sts558 · Cited by 137
Bedson, J., Chen, Y., Ashworth, J., Hayward, R. A., Dunn, K. M., & Jordan, K. P. (2019) · European Journal of Pain · doi:10.1002/ejp.1357 · Cited by 91
“We also included “control” outcomes (Arnold et al., 2016; Lipsitch et al., 2010) which hypothetically should not be associated with long‐term opioid prescribing to detect potential bias.”
BackgroundKioumourtzoglou, M.-A., Coull, B. A., O’Reilly, É. J., Ascherio, A., & Weisskopf, M. G. (2018) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2018.0727 · Cited by 79
A causal roadmap for generating high-quality real-world evidence
Dang, L. E., Gruber, S., Lee, H., Dahabreh, I. J., Stuart, E. A., Williamson, B. D., Wyss, R., Diaz, I. M., Ghosh, D., Kıcıman, E., et al. (2023) · Journal of Clinical and Translational Science · doi:10.1017/cts.2023.635 · Cited by 114
Izurieta, H. S., Wernecke, M., Kelman, J. A., Wong, S., Forshee, R. A., Pratt, D., Lu, Y., Sun, Q., Jankosky, C., Krause, P. R., et al. (2016) · Clinical Infectious Diseases · doi:10.1093/cid/ciw854 · Cited by 83
“The balance of covariates between the matched cohorts was assessed using the standardized mean difference [20].”
MethodSteenland, K., Schubauer-Berigan, M. K., Vermeulen, R., Lunn, R. M., Straif, K., Zahm, S., Stewart, P., Arroyave, W. D., Mehta, S. S., & Pearce, N. (2020) · Environmental Health Perspectives · doi:10.1289/ehp6980 · Cited by 69
“, cross-context comparisons); b) usually does not assess the likely direction and magnitude of the bias and impact on effect estimates; c) may not account for important secondary evidence such as exposure or outcome negative controls (Lipsitch et al. 2010); and d) may downgrade a specific study based on one flaw while…”
BackgroundBosch, N. A., Teja, B. J., Law, A. C., Pang, B., Jafarzadeh, S. R., & Walkey, A. J. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.0258 · Cited by 91
Yorifuji, T., Kashima, S., Diez, M. H., Kado, Y., Sanada, S., & Doi, H. (2015) · Epidemiology · doi:10.1097/ede.0000000000000361 · Cited by 86
Acute Gastroenteritis and Recreational Water: Highest Burden Among Young US Children
Arnold, B. F., Wade, T. J., Benjamin‐Chung, J., Schiff, K., Griffith, J. F., Dufour, A. P., Weisberg, S. B., & Colford, J. M. (2016) · American Journal of Public Health · doi:10.2105/ajph.2016.303279 · Cited by 84
Shao, S., Chang, K., Hung, M., Yang, N.-I., Chan, Y., Chen, H., Yang, Y. K., & Lai, E. C. (2019) · Cardiovascular Diabetology · doi:10.1186/s12933-019-0919-9 · Cited by 81
Waddington, H. S., Masset, E., Bick, S., & Cairncross, S. (2023) · PLoS Medicine · doi:10.1371/journal.pmed.1004215 · Cited by 91
Imamura, F., Lemaître, R. N., King, I. B., Song, X., Steffen, L. M., Folsom, A. R., Siscovick, D. S., & Mozaffarian, D. (2013) · Circulation · doi:10.1161/circulationaha.112.001197 · Cited by 78
“To determine whether LCMUFA-CHF associations were myocardium specific, we analyzed incident stroke as a prespecified negativecontrol outcome.(21) We selected stroke because it shares many risk factors with CHF (eg, hypertension) but should be unaffected by any causal processes specific to cardiac steatosis, ie, the hy…”
Estimating the population-level impact of vaccines using synthetic controls
Bruhn, C., Hetterich, S., Schuck‐Paim, C., Kürüm, E., Taylor, R. J., Lustig, R. L., Shapiro, E. D., Warren, J. L., Simonsen, L., & Weinberger, D. M. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1612833114 · Cited by 87
“Ideally, diseases used for comparison would share the same set of potential biases and causal factors as the outcome of interest but would not be influenced by the vaccine (15, 16).”
BackgroundDurrmeyer, X., Marchand‐Martin, L., Porcher, R., Gascoin-Lachambre, G., Rozé, J., Storme, L., Favrais, G., Ancel, P., & Cambonie, G. (2017) · Archives of Disease in Childhood Fetal & Neonatal · doi:10.1136/archdischild-2016-312104 · Cited by 77
Association of Baclofen With Encephalopathy in Patients With Chronic Kidney Disease
Muanda, F. T., Weir, M. A., Bathini, L., Blake, P. G., Chauvin, K., Dixon, S. N., McArthur, E., Sontrop, J. M., MOIST, L. M., & Garg, A. X. (2019) · JAMA · doi:10.1001/jama.2019.17725 · Cited by 81
Gandhi, S., Shariff, S. Z., Al‐Jaishi, A. A., Reiss, J. P., Mamdani, M., Hackam, D. G., Li, L., McArthur, E., Weir, M. A., & Garg, A. X. (2016) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2016.08.020 · Cited by 83
Toulis, K. A., Willis, B. H., Marshall, T., Kumarendran, B., Gokhale, K., Ghosh, S., Thomas, G. N., Cheng, K. K., Narendran, P., Hanif, W., et al. (2017) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2016-3446 · Cited by 77
“(19) and was implemented in a similar study design (20).”
Acute Illness Among Surfers After Exposure to Seawater in Dry- and Wet-Weather Conditions
Arnold, B. F., Schiff, K., Ercümen, A., Benjamin‐Chung, J., Steele, J. A., Griffith, J. F., Steinberg, S. J., Smith, P., McGee, C. D., Wilson, R. W., et al. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx019 · Cited by 76
“In Web Appendices 3–6 we describe additional analyses, including conversion of estimates to the absolute risk scale, sensitivity analyses, and negative control exposure analyses (25, 26).”
MethodHolt, A., Blanche, P., Zareini, B., Rajan, D., El‐Sheikh, M., Schjerning, A.-M., Schou, M., Torp-Pedersen, C. T., McGettigan, P., Gislason, G., et al. (2020) · European Heart Journal · doi:10.1093/eurheartj/ehaa1058 · Cited by 78
The Avon Longitudinal Study of Parents and children ALSPAC G0 Partners: A cohort profile
Northstone, K., Ben‐Shlomo, Y., Teyhan, A., Hill, A., Groom, A., Mummé, M., Timpson, N. J., & Golding, J. (2023) · Wellcome Open Research · doi:10.12688/wellcomeopenres.18782.1 · Cited by 85
“, 2019), this is an example of negative control analysis (Lipsitch et al., 2010).”
BackgroundQuantifying the impact of unmeasured confounding in observational studies with the E value
Gaster, T., Eggertsen, C. M., Støvring, H., Ehrenstein, V., & Petersen, I. (2023) · BMJ Medicine · doi:10.1136/bmjmed-2022-000366 · Cited by 87
Metadata
- Authors
- Lipsitch, M., Tchetgen Tchetgen, E., & Cohen, T.
- Affiliations
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Lipsitch, Marc
- Venue
- Epidemiology
- Published
- 2010-05
- Volume
- 21
- Issue
- 3
- Pages
- 383-388
- ISSN
- 1044-3983 (ISSN-L)
- PMC
- PMC3053408
- OpenAlex
- W2009428399
- Cited by
- 1,583
- References
- 25
- Citations by year
- 2026: 278 · 2025: 215 · 2024: 183 · 2023: 138 · 2022: 153 · 2021: 132 · 2020: 107 · 2019: 75 · 2018: 72 · 2017: 71 · 2016: 56 · 2015: 32 · 2014: 26 · 2013: 25 · 2012: 11
- FWCI
- 11.64
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Funders
- National Institutes of Health (5u01gm076497-03, 1dp2od006663-01)
- National Institute of General Medical Sciences (U54GM088558, U01GM076497, 5U01GM076497)
- National Institute of Environmental Health Sciences (T32ES007142)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- negative control, confounding, spurious association, exposure control, outcome control, observational study validity, recall bias, experimental design, causal inference
- MeSH terms
- Humans, Bias, Causality, Confounding Factors, Epidemiologic, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.