Exploring gender biases in ML and AI academic research through systematic literature review
Open accessThe DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Automated systems that implement Machine learning (ML) and Artificial Intelligence (AI) algorithms present promising solutions to a variety of technological and non-technological issues. Although, industry leaders are rapidly adopting these systems for anything from marketing to national defense operations, these systems are not without flaws. Recently, many of these systems are found to inherit and propagate gender and racial biases that disadvantages the minority population. In this paper, we analyze academic publications in the area of gender biases in ML and AI algorithms thus outlining different themes, mitigation and detection methods explored through research in this topic. Through a detailed analysis of N = 120 papers, we map the current research landscape on gender specific biases present in ML and AI assisted automated systems. We further point out the aspects of ML/AI gender biases research that are less explored and require more attention. Mainly we focus on the lack of user studies and inclusivity in this field of study. We also shed some light into the gender bias issue as experienced by the algorithm designers. In conclusion, in this paper we provide a holistic view of the breadth of studies conducted in the field of exploring, detecting and mitigating gender biases in ML and AI systems and, a future direction for the studies to take in order to provide a fair and accessible ML and AI systems to all users.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“For example, some papers look into different specific incidences of gender bias in justice systems, medical robots, and self-driving cars (Howard and Borenstein, 2018; Amershi et al., 2019; Brandao, 2019).”
BackgroundLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 2×
“Through their paper, authors present the solution of fixing the gender-bias problem from outside-in, i.e., proactively understanding the context and data rather than fixing the algorithm after training (Blodgett et al., 2020).”
BackgroundBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 2×
“By understanding the type and characteristics of the dataset, ML model creators and users will be able to gauge the prediction quality of the ML model and its appropriate application (Bender and Friedman, 2018).”
BackgroundLambrecht, A., & Tucker, C. (2019) · Management Science · doi:10.1287/mnsc.2018.3093 · Cited by 854 · cited 1×
“The jobs shown to women were discriminatory in nature as they were shown far fewer STEM ads than men (Lambrecht and Tucker, 2019).”
BackgroundMitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · cited 1×
“For example, Caton et al. review different biases detected in automated systems based on sensitive attributes like gender, marital status, race, and disabilities (Caton and Haas, 2020).”
BackgroundMitigating Gender Bias in Natural Language Processing: Literature Review
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1159 · Cited by 492 · cited 3×
“Here, NLP operations refers to functions like embeddings, coreference resolution, dialogue generation, machine translation, text parsing, sentiment analysis, hate speech detection, and so on (Sun et al., 2019; Blodgett et al., 2020).”
BackgroundGender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 1×
“In their paper Rudinger et al., the authors present the system with Winograd style sentence-pairs that use profession and differ only in pronouns.”
Assessing gender bias in machine translation: a case study with Google Translate
Prates, M. O. R., Avelar, P. H., & Lamb, L. C. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04144-6 · Cited by 410
Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate
Prates, M. O. R., Avelar, P. H. C., & Lamb, L. (2018) · arXiv · doi:10.48550/arxiv.1809.02208 · Cited by 409
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384
WinoGrande: an adversarial winograd schema challenge at scale
Sakaguchi, K., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · Communications of the ACM · doi:10.1145/3474381 · Cited by 356 · cited 1×
“have proposed Winograd based questionnaire framework which can be used to test the presence of gender bias in coreference resolution systems and word association algorithms respectively (Sakaguchi et al., 2021).”
MethodBias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337
The Ugly Truth About Ourselves and Our Robot Creations: The Problem of Bias and Social Inequity
Howard, A., & Borenstein, J. (2018) · Science and Engineering Ethics · doi:10.1007/s11948-017-9975-2 · Cited by 337
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · arXiv · doi:10.48550/arxiv.1901.09451 · Cited by 337
Bias detectives: the researchers striving to make algorithms fair
Courtland, R. (2018) · Nature · doi:10.1038/d41586-018-05469-3 · Cited by 205 · cited 1×
“Baeza-Yates and Courtland also emphasize the idea of understanding the context of model implementation and the data used for model training to prevent pre-existing/data biases (Courtland, 2018; Baeza-Yates, 2020).”
MethodGuo, W., & Caliskan, A. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462536 · Cited by 180 · cited 5×
“Chen et al., look across nine human languages, including English, Spanish, Arabic, German, French, and Urdu, and find gender bias in gendered nouns for profession words (Chen et al., 2021) in Word Embeddings.”
BackgroundInfluentialFairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171
Designing and Evaluating mHealth Interventions for Vulnerable Populations: A Systematic Review
Stowell, E., Lyson, M. C., Saksono, H., Wurth, R. C., Jimison, H., Pavel, M., & Parker, A. G. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173589 · Cited by 169 · cited 2×
“Their research focused onmHealth intervention for vulnerable population, and phishing and authentication respectively (Stowell et al., 2018; Das et al., 2019).”
BackgroundToward situated interventions for algorithmic equity: lessons from the field
Katell, M., Young, M., Dailey, D., Herman, B., Guetler, V., Tam, A., Bintz, C., Raz, D., & Krafft, P. M. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372874 · Cited by 153 · cited 1×
“Surprisingly, we found nine papers that expanded on the legal and ethical implication of gender biases in automated systems (Katell et al., 2020; Fleisher, 2021).”
BackgroundAndrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445888 · Cited by 111
Gender bias in sentiment analysis
Thelwall, M. (2018) · Online Information Review · doi:10.1108/oir-05-2017-0139 · Cited by 78 · cited 1×
“…resolution, the majority of detection methods discussed in these papers target NLP tasks like sentiment analysis (Kiritchenko and Mohammad, 2018; Thelwall, 2018; Sarraf et al., 2021), information retrieval (Rekabsaz and Schedl, 2020), Word Embeddings (Leavy, 2018; Guo and Caliskan, 2021) and a…”
Singh, V. K., Chayko, M., Inamdar, R., & Floegel, D. (2020) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24335 · Cited by 69 · cited 1×
“also highlights the issue of lack of women representation in occupations images in various digital platforms (Singh et al., 2020).”
BackgroundDo neural ranking models intensify gender bias?
Rekabsaz, N., & Schedl, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401280 · Cited by 64 · cited 1×
“…these papers target NLP tasks like sentiment analysis (Kiritchenko and Mohammad, 2018; Thelwall, 2018; Sarraf et al., 2021), information retrieval (Rekabsaz and Schedl, 2020), Word Embeddings (Leavy, 2018; Guo and Caliskan, 2021) and a combination of language processing tasks (Hitti et al., 2019;…”
BackgroundO'Reilly-Shah, V. N., Gentry, K. R., Walters, A. M., Zivot, J., Anderson, C. T., & Tighe, P. J. (2020) · British Journal of Anaesthesia · doi:10.1016/j.bja.2020.07.040 · Cited by 62
Glymour, B., & Herington, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287573 · Cited by 50 · cited 1×
“Glymour and Herington also measure the severity of these biases and lay out the implication of these biases (Glymour and Herington, 2019).”
BackgroundAge and gender bias in pedestrian detection algorithms
Brandao, M. (2019) · arXiv · doi:10.48550/arxiv.1906.10490 · Cited by 32 · cited 1×
“For example, some papers look into different specific incidences of gender bias in justice systems, medical robots, and self-driving cars (Howard and Borenstein, 2018; Amershi et al., 2019; Brandao, 2019).”
BackgroundUser Bias in Beyond-Accuracy Measurement of Recommendation Algorithms
Wang, N., & Chen, L. (2021) · Fifteenth ACM Conference on Recommender Systems · doi:10.1145/3460231.3474244 · Cited by 22 · cited 6×
“The paper by Wang et al. also conducts a case study into career recommender systems and implementation of debiasing technique (Wang et al., 2021).”
BackgroundInfluentialEnd-To-End Bias Mitigation: Removing Gender Bias in Deep Learning
Feldman, T., & Peake, A. (2021) · arXiv · doi:10.48550/arxiv.2104.02532 · Cited by 19
Cited by
Bias and ethics of AI systems applied in auditing - A systematic review
Murikah, W., Nthenge, J. K., & Musyoka, F. M. (2024) · Scientific African · doi:10.1016/j.sciaf.2024.e02281 · Cited by 148
Alkhaaldi, S. M. I., Kassab, C. H., Dimassi, Z., Oyoun Alsoud, L., Al Fahim, M., Al Hageh, C., & Ibrahim, H. (2023) · JMIR Medical Education · doi:10.2196/51302 · Cited by 132
“Prior studies have shown that AI can perpetuate racial and gender biases and stereotypes [40].”
BackgroundFairness-Aware Machine Learning Engineering: How Far Are We?
Ferrara, C., Sellitto, G., Ferrucci, F., Palomba, F., & De Lucia, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10402-y · Cited by 60
Barriers to AI adoption for women in higher education: a systematic review of the Asian context
Kalim, U., Kanwar, A., Sha, J., & Huang, R. (2025) · Smart Learning Environments · doi:10.1186/s40561-025-00390-5 · Cited by 22
Metadata
- Authors
- Shrestha, S., & Das, S.
- Venue
- Frontiers in Artificial Intelligence
- Publisher
- Frontiers Media SA
- Published
- 2022-10-11
- Volume
- 5
- Pages
- 976838-976838
- ISSN
- 2624-8212 (ISSN-L)
- OpenAlex
- W4304183812
- Cited by
- 95
- References
- 34
- FWCI
- 3.82
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.