Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators---such as first names and pronouns---in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are "scrubbed," and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Under the premise of profession-induced gender bias, De-Arteaga et al (2019) introduced the TPR-GAP metric, which assesses the difference (GAP) in the true posi -tive rate (TPR) between the two genders for each occupation.”
From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as De-Arteaga et al (2019)
In the citing paper
“Given that NLP-based techniques have shown biases against under-represented subpopulations in both medical 32 and non-medical 33 domains, the automatic labeler could potentially be a large source of bias.”
From Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations (2021) · cites this paper as [33]
In the citing paper
“There is extensive work on detecting and mitigating bias in NLP models [12,14,15,22,36,98].”
From Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey) (2024) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · cited 1×
“The WE representation encodes the ith biography as a vector xWE i , obtained by averaging the fastText word embeddings [9, 31] for the word types present in that biography.”
MethodHierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1174 · Cited by 4,712
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“y computing the TPR gender gap— i.e., the difference in TPRs between genders—for each occupation. This notion of bias is closely related to the equality of opportunity fairness metric of Hardt et al. [25]. We choose to focus on TPR gender gaps because they enable us to study the ways in which gender imbalances may be…”
ResultWord embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · cited 1×
“Recent work has studied the ways in which stereotypes and other human biases may be reflected in semantic representations such as word embeddings [10, 13, 21].”
BackgroundGender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 1×
“Natural language processing researchers have also studied gender bias in coreference resolution [34, 44], showing that systems perform better when linking a gender pronoun to an occupation in which that gender is overrepresented than to an occupation in which it is underrepresented.”
BackgroundMeasuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · cited 1×
“Finally, machine learning methods for identifying toxic comments exhibit disproportionately high false positive rates for words like gay and homosexual [17].”
BackgroundAutomatically Categorizing Written Texts by Author Gender
Koppel, M. (2002) · Literary and Linguistic Computing · doi:10.1093/llc/17.4.401 · Cited by 623 · cited 1×
“Computational linguistics researchers have explored the use of lexical and syntactic features to infer authors’ genders [15, 30].”
BackgroundFA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 1×
“Natural language processing researchers have also studied gender bias in coreference resolution [34, 44], showing that systems perform better when linking a gender pronoun to an occupation in which that gender is overrepresented than to an occupation in which it is underrepresented.”
BackgroundWomen in Economics: Moving Up or Falling Off the Academic Career Ladder?
Ginther, D. K., & Kahn, S. (2004) · Journal of Economic Perspectives · doi:10.1257/0895330042162386 · Cited by 429
Measuring Fairness in Ranked Outputs
Yang, K., & Stoyanovich, J. (2017) · Proceedings of the 29th International Conference on Scientific and Statistical Database Management · doi:10.1145/3085504.3085526 · Cited by 378
Gender and Dialect Bias in YouTube’s Automatic Captions
Tatman, R. (2017) · Proceedings of the First ACL Workshop on Ethics in Natural Language Processing · doi:10.18653/v1/w17-1606 · Cited by 338
Equity of Attention: Amortizing Individual Fairness in Rankings
Biega, A. J., Gummadi, K. P., & Weikum, G. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210063 · Cited by 337
Who Takes the Floor and Why: Gender, Power, and Volubility in Organizations
Brescoll, V. L. (2011) · Administrative Science Quarterly · doi:10.1177/0001839212439994 · Cited by 302
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., & Goldberg, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04207 · Cited by 283
Kleinberg, J., Ludwig, J., Mullainathan, S., & Rambachan, A. (2018) · AEA Papers and Proceedings · doi:10.1257/pandp.20181018 · Cited by 278
Author gender identification from text
Cheng, N., Chandramouli, R., & Subbalakshmi, K. P. (2011) · Digital Investigation · doi:10.1016/j.diin.2011.04.002 · Cited by 223
Blodgett, S. L., & O'Connor, B. (2017) · arXiv · doi:10.48550/arxiv.1707.00061 · Cited by 153
Discrimination and Privacy in the Information Society: Data Mining and Profiling in Large Databases
(2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3 · Cited by 100
Do Women and Men in Congress Cultivate Different Images? Evidence from Congressional Web Sites
Niven, D., & Zilber, J. (2001) · Political Communication · doi:10.1080/10584600152647100 · Cited by 93
Dwork, C., & Ilvento, C. (2019) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2019.33 · Cited by 91 · cited 1×
“Recently, Dwork and Ilvento [19] showed that fairness does not hold under composition, meaning that if two classifiers are individually fair according to some fairness metric, then the sequential use of these classifiers will not necessarily be fair according the same metric.”
BackgroundImplementing Anti-discrimination Policies in Statistical Profiling Models
Pope, D. G., & Sydnor, J. R. (2011) · American Economic Journal: Economic Policy · doi:10.1257/pol.3.3.206 · Cited by 87
Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
Zhao, J., Wang, T., Yatskar, M., Ordóñez, V., & Chang, K.-W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.06876 · Cited by 85
Altenburger, K., De, R., Frazier, K., Avteniev, N., & Hamilton, J. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14929 · Cited by 35 · cited 1×
“[2] analyzed self-promotion in LinkedIn, finding that women are more modest than men in expressing accomplishments and are less likely to use free-form fields.”
BackgroundEquity of Attention: Amortizing Individual Fairness in RankingsPreprint
Biega, A. J., Gummadi, K. P., & Weikum, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01788 · Cited by 26
Research works printed but not linked: 16
- [3] Ian Ayres. 2002. Outcome tests of racial disparities in police practices. Justice research and Policy 4, 1-2 (2002), 131-142. -
- [5] Marianne Bertrand and Esther Duflo. 2017. Field Experiments on Discrimination. In Handbook of Economic Field Experiments . Vol. 1. Elsevier, 309-393. -
- [10] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems . 4349-4357. -
- [12] Toon Calders and Indr˙ e Žliobait˙ e. 2013. Why unbiased computational processes can lead to discriminative decision procedures. In Discrimination and privacy in the information society . Springer, 43-57. -
- [14] L Elisa Celis, Damian Straszak, and Nisheeth K Vishnoi. 2018. Ranking with fairness constraints. In Proceedings of the International Colloquium on Automata, Languages, and Programming . -
- [20] Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Mark DM Leiserson. 2018. Decoupled classifiers for group-fair and efficient machine learning. In Conference on Fairness, Accountability and Transparency . 119-133. -
- [24] Claudia Goldin and Cecilia Rouse. 2000. Orchestrating impartiality: The impact of" blind" auditions on female musicians. American economic review 90, 4 (2000), 715-741. -
- [26] Deborah Hellman. 2018. Indirect Discrimination and the Duty to Avoid Compounding Injustice. Foundations of Indirect Discrimination Law, Forthcoming (2018). -
- [27] Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. 2017. Avoiding discrimination through causal reasoning. In Advances in Neural Information Processing Systems . 656-666. -
- [28] Pauline T Kim. 2016. Data-driven discrimination at work. Wm. & Mary L. Rev. 58 (2016), 857. -
- [31] Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in Pre-Training Distributed Word Representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018) . -
- [35] Heather Sarsons. 2015. Gender differences in recognition for group work. Harvard University Working Paper (2015). -
- [36] Heather Sarsons. 2017. Interpreting signals in the labor market: evidence from medical referrals. Job Market Paper (2017). -
- [37] David G Smith, Judith E Rosenstein, Margaret C Nikolov, and Darby A Chaney. 2018. The Power of Language: Gender, Status, and Agency in Performance Evaluations. Sex Roles (2018), 1-13. -
- [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems . 5998-6008. -
- [43] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning . 325-333. -
Cited by
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928
“Given that NLP-based techniques have shown biases against under-represented subpopulations in both medical 32 and non-medical 33 domains, the automatic labeler could potentially be a large source of bias.”
BackgroundAn Attentive Survey of Attention Models
Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3465055 · Cited by 689
Racial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765
A Framework of Potential Sources of Harm Throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 576
Metadata
- Authors
- De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T.
- Affiliations
- Carnegie Mellon University (US)
- University of Massachusetts Lowell (US)
- Microsoft (United States) (US)
- LinkedIn (United States) (US)
- ORCID
- Venue
- Proceedings of the Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2019-01-29
- Pages
- 120-128
- OpenAlex
- W2909212904
- Cited by
- 337
- References
- 18
- Citations by year
- 2026: 34 · 2025: 41 · 2024: 36 · 2023: 71 · 2022: 67 · 2021: 52 · 2020: 25 · 2019: 11
- FWCI
- 40.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- gender bias, occupational classification, gender indicators, given names, pronouns, algorithmic fairness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.