Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.

Hurtful words: quantifying biases in clinical contextual word embeddings

Zhang, Haoran, Lu, Amy X., Abdalla, Mohamed, McDermott, Matthew, Ghassemi, Marzyeh · 2020 · Proceedings of the ACM Conference on Health, Inference, and Learning

Abstract

In this work, we examine the extent to which embeddings may encode marginalized populations differently, and how this may lead to a perpetuation of biases and worsened performance on clinical tasks. We pretrain deep embedding models (BERT) on medical notes from the MIMIC-III hospital dataset, and quantify potential disparities using two approaches. First, we identify dangerous latent relationships that are captured by the contextual word embeddings using a fill-in-the-blank method with text from real clinical notes and a log probability bias score quantification. Second, we evaluate performance gaps across different definitions of fairness on over 50 downstream clinical prediction tasks that include detection of acute and chronic conditions. We find that classifiers trained from BERT representations exhibit statistically significant differences in performance, often favoring the majority group with regards to gender, language, ethnicity, and insurance status. Finally, we explore shortcomings of using adversarial debiasing to obfuscate subgroup information in contextual word embeddings, and recommend best practices for such deep embedding models in clinical settings.

Overview

Integrity signals

Registry

Crossref · 10.1145/3368555.3384448

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 41 references

Research works printed but not linked: 33

  1. 2020, Artificial Intelligence in Medicine. https://www.ibm.com/watson-health/learn/artificial-intelligence-medicine
  2. 2020, Drug development gets big data analytics boost. https://www.novartis.com/stories/discovery/drug-development-gets-big-data-analytics-boost
  3. Alsentzer Emily, 2019, McDermott
  4. Arnold Emily A, 2014, Triply cursed: racism, homophobia and HIV-related stigma are barriers to regular HIV testing, treatment adherence and disclosure among young Black gay men. Culture, health & sexuality 16, 6
  5. Basta Christine, 2019, Evaluating the underlying gender bias in contextualized word embeddings. arXiv preprint arXiv:1904.08783 · arXiv
  6. Beutel Alex, 2017, Chi
  7. Tolga Bolukbasi Kai-Wei Chang James Y Zou Venkatesh Saligrama and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems. 4349--4357. Tolga Bolukbasi Kai-Wei Chang James Y Zou Venkatesh Saligrama and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems . 4349--4357.
  8. Buolamwini Joy, 2018, Conference on fairness, accountability and transparency. 77--91
  9. Chen Irene Y, 2019, Can AI Help Reduce Disparities in General Medical and Mental Health Care? AMA journal of ethics 21, 2
  10. Yiming Cui Wanxiang Che Ting Liu Bing Qin Ziqing Yang Shijin Wang and Guoping Hu. 2019. Pre-Training with Whole Word Masking for Chinese BERT. (2019). [arxiv]1906.08101 http://arxiv.org/abs/1906.08101 Yiming Cui Wanxiang Che Ting Liu Bing Qin Ziqing Yang Shijin Wang and Guoping Hu. 2019. Pre-Training with Whole Word Masking for Chinese BERT. (2019). [arxiv]1906.08101 http://arxiv.org/abs/1906.08101 · arXiv
  11. DuBard C Annette, 2006, Effect of language on heart attack and stroke awareness among US Hispanics. American journal of preventive medicine 30, 3
  12. Edwards Harrison, 2015, Censoring representations with an adversary. arXiv preprint arXiv:1511.05897 · arXiv
  13. Elazar Yanai, 2018, Adversarial removal of demographic attributes from text data. arXiv preprint arXiv:1808.06640 · arXiv
  14. Glance Laurent G, 2013, Trends in racial disparities for injured patients admitted to trauma centers. Health services research 48, 5
  15. Moritz Hardt Eric Price Nati Srebro etal 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323. Moritz Hardt Eric Price Nati Srebro et al. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems . 3315--3323.
  16. Hashimoto Tatsunori, 2018, Fairness Without Demographics in Repeated Loss Minimization. In International Conference on Machine Learning. 1934--1943
  17. Hwang Wenke, 2001, Out-of-pocket medical spending for care of chronic conditions. Health affairs 20, 6
  18. Johnson Alistair EW, 2016, Leo Anthony Celi, and Roger G Mark
  19. Kurita Keita, 2019, Measuring Bias in Contextualized Word Representations. arXiv preprint arXiv:1906.07337 · arXiv
  20. LeCun Yann, 2015, Deep learning. nature 521, 7553
  21. Lee Jinhyuk, 2019, Chan Ho So, and Jaewoo Kang
  22. Liu Yinhan, 2019, RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 · arXiv
  23. Mehrabi Ninareh, 2019, A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 · arXiv
  24. Milner Kerry A, 2004, Gender and age differences in chief complaints of acute myocardial infarction (Worcester Heart Attack Study). The American journal of cardiology 93, 5
  25. Nelson Gregory S, 2019, Bias in artificial intelligence. North Carolina medical journal 80, 4
  26. Radford Alec, 2019, Language models are unsupervised multitask learners. OpenAI Blog 1, 8
  27. Suresh Harini, 2017, Machine Learning for Healthcare Conference. 322--337
  28. Yi Chern Tan and L. Elisa Celis. 2019. Assessing Social and Intersectional Biases in Contextualized Word Representations. [arxiv]cs.CL/1911.01485 Yi Chern Tan and L. Elisa Celis. 2019. Assessing Social and Intersectional Biases in Contextualized Word Representations. [arxiv]cs.CL/1911.01485
  29. Topol Eric J, 2019, High-performance medicine: the convergence of human and artificial intelligence. Nature medicine 25, 1
  30. Vayena Effy, 2018, Machine learning in medicine: Addressing ethical challenges. PLoS medicine 15, 11
  31. Wang Tianlu, 2018, Adversarial Removal of Gender from Deep Image Representations. arXiv preprint arXiv:1811.08489 · arXiv
  32. Zemel Rich, 2013, International Conference on Machine Learning. 325--333
  33. Zhao Jieyu, 2018, Learning gender-neutral word embeddings. arXiv preprint arXiv:1809.01496 · arXiv

Cited by

See all 122

Metadata

Authors
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M.
Affiliations
  • University of Toronto (CA)
  • Massachusetts Institute of Technology (US)
Venue
Proceedings of the ACM Conference on Health, Inference, and Learning
Publisher
ACM
Published
2020-04-02
Pages
110-120
OpenAlex
W3011762034
Cited by
122
References
41
Citations by year
2026: 8 · 2025: 19 · 2024: 24 · 2023: 22 · 2022: 17 · 2021: 25 · 2020: 7
FWCI
6.68
Type
conference paper
Language
en
Open access
closed
Locations
ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
algorithmic bias, contextual word embeddings, BERT, clinical notes, MIMIC-III, marginalized communities, fairness, clinical prediction, performance gap, gender, language, ethnicity, insurance status, adversarial debiasing, clinical language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Hurtful words: quantifying biases in clinical contextual word embeddings. https://app.cochat.ai/papers/10.1145/3368555.3384448
The paper
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020). Hurtful words: quantifying biases in clinical contextual word embeddings. Proceedings of the ACM Conference on Health, Inference, and Learning. https://doi.org/10.1145/3368555.3384448

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.