Hurtful words: quantifying biases in clinical contextual word embeddings
The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
In this work, we examine the extent to which embeddings may encode marginalized populations differently, and how this may lead to a perpetuation of biases and worsened performance on clinical tasks. We pretrain deep embedding models (BERT) on medical notes from the MIMIC-III hospital dataset, and quantify potential disparities using two approaches. First, we identify dangerous latent relationships that are captured by the contextual word embeddings using a fill-in-the-blank method with text from real clinical notes and a log probability bias score quantification. Second, we evaluate performance gaps across different definitions of fairness on over 50 downstream clinical prediction tasks that include detection of acute and chronic conditions. We find that classifiers trained from BERT representations exhibit statistically significant differences in performance, often favoring the majority group with regards to gender, language, ethnicity, and insurance status. Finally, we explore shortcomings of using adversarial debiasing to obfuscate subgroup information in contextual word embeddings, and recommend best practices for such deep embedding models in clinical settings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Given that NLP-based techniques have shown biases against under-represented subpopulations in both medical 32 and non-medical 33 domains, the automatic labeler could potentially be a large source of bias.”
From Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations (2021) · cites this paper as [32]
In the citing paper
“Following best practice 16,32 for FPR, FNR and FDR estimation, we select a single threshold for all groups, which maximizes the F1 score.”
From Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations (2021) · cites this paper as [32]
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Le Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Mitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387
Ensuring fairness in machine learning to advance health equity.
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1990 · Cited by 1,341
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · cited 2×
“While these techniques have high predictive ability, they can capture relationships between words that reflect biases, e.g., along gender [9] or ethnic identity [21].”
Background(2017) · PubMed · Cited by 961
Multitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958 · cited 3×
“For all tasks, we use the same held-out test set defined in previous work [27].”
MethodInfluentialInherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 2×
“” the model is, we must turn to a definition of fairness from the literature. There are many technical definitions of fairness, and specific arguments for or against using each in healthcare settings [26, 35, 51]. In this work, we examine the demographic parity, equality of opportunity for the positive class, and equa…”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Fiscella, K., Franks, P., Doescher, M. P., & Saver, B. G. (2002) · Medical Care · doi:10.1097/00005650-200201000-00007 · Cited by 906
Jones, A. E., Trzeciak, S., & Kline, J. A. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e31819def97 · Cited by 850
AI can be sexist and racist — it’s time to make it fair
Zou, J., & Schiebinger, L. (2018) · Nature · doi:10.1038/d41586-018-05707-8 · Cited by 765 · cited 1×
“…exist within both traditional non-contextual word embeddings [9, 73] and contextual word embeddings [36, 62, 72], biases in pretrained contextual embedding models can additionally encode historical biases in the training corpora, class imbalance in datasets, and data quality differences [53, 74].”
BackgroundSahlgren, M. (2008) · DiVA (Digitala Vetenskapliga Arkivet) · Cited by 752
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618
Lum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611
The Wilcoxon signed rank test for paired comparisons of clustered data
Rosner, B., Glynn, R. J., & Lee, M. T. (2006) · Biometrics · doi:10.1111/j.1541-0420.2005.00389.x · Cited by 557
Meghani, S. H., Byun, E., & Gallagher, R. M. (2012) · Pain Medicine · doi:10.1111/j.1526-4637.2011.01310.x · Cited by 528
Racial disparities in psychotic disorder diagnosis: a review of empirical literature
Schwartz, R. C. (2014) · World Journal of Psychiatry · doi:10.5498/wjp.v4.i4.133 · Cited by 461 · cited 1×
“Importantly, as algorithms enter healthcare systems, it is important not to exacerbate the treatment disparities amongst existing subgroups [23, 43, 57].”
BackgroundGonen, H., & Goldberg, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1061 · Cited by 454
Out-Of-Pocket Medical Spending For Care Of Chronic Conditions
Hwang, W., Weller, W., Ireys, H., & Anderson, G. (2001) · Health Affairs · doi:10.1377/hlthaff.20.6.267 · Cited by 440
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · cited 1×
“There are documented effects of bias propagated by real-world machine learning systems, from gender biases in job classification [15], to decreased prediction accuracies on dark-skinned females due to sample imbalances [11], and to overprescribing policing resources to historically over-policed neighbourhoods [41].”
BackgroundBias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · arXiv · doi:10.48550/arxiv.1901.09451 · Cited by 337
Enhancing clinical concept extraction with contextual embeddings
Si, Y., Wang, J., Xu, H., & Roberts, K. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz096 · Cited by 322
Gender Bias in Contextualized Word Embeddings
Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., & Chang, K.-W. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1064 · Cited by 319
Johnson, A. E. W., Kramer, A. A., & Clifford, G. D. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e31828a24fe · Cited by 297
Framing the challenges of artificial intelligence in medicine
Yu, K.-H., & Kohane, I. S. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008551 · Cited by 233
Why is my classifier discriminatory?
Chen, I., Johansson, F. D., & Sontag, D. (2018) · arXiv · doi:10.48550/arxiv.1805.12002 · Cited by 185 · cited 1×
“on healthcare data may exhibit such biases [12, 13, 69], and it is important to take measures that minimize the impact of bias on predictive systems [48, 65].”
BackgroundSciBERT: Pretrained Contextualized Embeddings for Scientific Text
Beltagy, I., Cohan, A., & Lo, K. (2019) · Cited by 173
Learning Adversarially Fair and Transferable Representations
Madras, D., Creager, E., Pitassi, T., & Zemel, R. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06309 · Cited by 171
Multiculturalism, Chronic Illness, and Disability
Groce, N. E., & Zola, I. K. (1993) · Pediatrics · doi:10.1542/peds.91.5.1048 · Cited by 155
Racial variation in the use of do-not-resuscitate orders
Shepardson, L. B., Gordon, H. S., Ibrahim, S. A., Harper, D. L., & Rosenthal, G. E. (1999) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.1999.00275.x · Cited by 128
Understanding the Origins of Bias in Word Embeddings
Brunet, M.-E., Alkalay-Houlihan, C., Anderson, A., & Zemel, R. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03611 · Cited by 99
Dimensions of Institutional Racism in Psychiatry
SABSHIN, M., DIESENHAUS, H., & WILKERSON, R. (1970) · American Journal of Psychiatry · doi:10.1176/ajp.127.6.787 · Cited by 95
Opportunities in Machine Learning for Healthcare.Preprint
Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L., & Ranganath, R. (2018) · arXiv (Cornell University) · Cited by 59
Self-citationCreating Fair Models of Atherosclerotic Cardiovascular Disease Risk
Pfohl, S., Marafino, B., Coulet, A., Rodriguez, F., Palaniappan, L., & Shah, N. H. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314278 · Cited by 52
Barriers to Prompt Presentation to Emergency Departments in Colorado after Onset of Stroke Symptoms
Trent, S. A., Morse, E. A., Ginde, A. A., Havranek, E. P., & Haukoos, J. S. (2019) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2018.10.38731 · Cited by 6
Research works printed but not linked: 31
- 2020, Artificial Intelligence in Medicine. https://www.ibm.com/watson-health/learn/artificial-intelligence-medicine
- 2020, Drug development gets big data analytics boost. https://www.novartis.com/stories/discovery/drug-development-gets-big-data-analytics-boost
- Alsentzer Emily, 2019, McDermott
- Arnold Emily A, 2014, Triply cursed: racism, homophobia and HIV-related stigma are barriers to regular HIV testing, treatment adherence and disclosure among young Black gay men. Culture, health & sexuality 16, 6
- Basta Christine, 2019, Evaluating the underlying gender bias in contextualized word embeddings. arXiv preprint arXiv:1904.08783 · arXiv
- Beutel Alex, 2017, Chi
- Tolga Bolukbasi Kai-Wei Chang James Y Zou Venkatesh Saligrama and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems. 4349--4357. Tolga Bolukbasi Kai-Wei Chang James Y Zou Venkatesh Saligrama and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems . 4349--4357.
- Buolamwini Joy, 2018, Conference on fairness, accountability and transparency. 77--91
- Chen Irene Y, 2019, Can AI Help Reduce Disparities in General Medical and Mental Health Care? AMA journal of ethics 21, 2
- Yiming Cui Wanxiang Che Ting Liu Bing Qin Ziqing Yang Shijin Wang and Guoping Hu. 2019. Pre-Training with Whole Word Masking for Chinese BERT. (2019). [arxiv]1906.08101 http://arxiv.org/abs/1906.08101 Yiming Cui Wanxiang Che Ting Liu Bing Qin Ziqing Yang Shijin Wang and Guoping Hu. 2019. Pre-Training with Whole Word Masking for Chinese BERT. (2019). [arxiv]1906.08101 http://arxiv.org/abs/1906.08101 · arXiv
- DuBard C Annette, 2006, Effect of language on heart attack and stroke awareness among US Hispanics. American journal of preventive medicine 30, 3
- Edwards Harrison, 2015, Censoring representations with an adversary. arXiv preprint arXiv:1511.05897 · arXiv
- Elazar Yanai, 2018, Adversarial removal of demographic attributes from text data. arXiv preprint arXiv:1808.06640 · arXiv
- Glance Laurent G, 2013, Trends in racial disparities for injured patients admitted to trauma centers. Health services research 48, 5
- Moritz Hardt Eric Price Nati Srebro etal 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323. Moritz Hardt Eric Price Nati Srebro et al. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems . 3315--3323.
- Hashimoto Tatsunori, 2018, Fairness Without Demographics in Repeated Loss Minimization. In International Conference on Machine Learning. 1934--1943
- Johnson Alistair EW, 2016, Leo Anthony Celi, and Roger G Mark
- Kurita Keita, 2019, Measuring Bias in Contextualized Word Representations. arXiv preprint arXiv:1906.07337 · arXiv
- LeCun Yann, 2015, Deep learning. nature 521, 7553
- Lee Jinhyuk, 2019, Chan Ho So, and Jaewoo Kang
- Liu Yinhan, 2019, RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 · arXiv
- Milner Kerry A, 2004, Gender and age differences in chief complaints of acute myocardial infarction (Worcester Heart Attack Study). The American journal of cardiology 93, 5
- Nelson Gregory S, 2019, Bias in artificial intelligence. North Carolina medical journal 80, 4
- Radford Alec, 2019, Language models are unsupervised multitask learners. OpenAI Blog 1, 8
- Suresh Harini, 2017, Machine Learning for Healthcare Conference. 322--337
- Yi Chern Tan and L. Elisa Celis. 2019. Assessing Social and Intersectional Biases in Contextualized Word Representations. [arxiv]cs.CL/1911.01485 Yi Chern Tan and L. Elisa Celis. 2019. Assessing Social and Intersectional Biases in Contextualized Word Representations. [arxiv]cs.CL/1911.01485
- Topol Eric J, 2019, High-performance medicine: the convergence of human and artificial intelligence. Nature medicine 25, 1
- Vayena Effy, 2018, Machine learning in medicine: Addressing ethical challenges. PLoS medicine 15, 11
- Wang Tianlu, 2018, Adversarial Removal of Gender from Deep Image Representations. arXiv preprint arXiv:1811.08489 · arXiv
- Zemel Rich, 2013, International Conference on Machine Learning. 325--333
- Zhao Jieyu, 2018, Learning gender-neutral word embeddings. arXiv preprint arXiv:1809.01496 · arXiv
Cited by
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724
Self-citation
Metadata
- Authors
- Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M.
- Affiliations
- University of Toronto (CA)
- Massachusetts Institute of Technology (US)
- Venue
- Proceedings of the ACM Conference on Health, Inference, and Learning
- Publisher
- ACM
- Published
- 2020-04-02
- Pages
- 110-120
- OpenAlex
- W3011762034
- Cited by
- 122
- References
- 44
- Citations by year
- 2026: 8 · 2025: 19 · 2024: 24 · 2023: 22 · 2022: 17 · 2021: 25 · 2020: 7
- FWCI
- 6.68
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic bias, contextual word embeddings, BERT, clinical notes, MIMIC-III, marginalized communities, fairness, clinical prediction, performance gap, gender, language, ethnicity, insurance status, adversarial debiasing, clinical language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.