AI can be sexist and racist — it’s time to make it fair
Open accessAbstract
Computer scientists must identify sources of bias, de-bias training data and develop artificial-intelligence algorithms that are robust to skews in the data, argue James Zou and Londa Schiebinger. Computer scientists must identify sources of bias, de-bias training data and develop artificial-intelligence algorithms that are robust to skews in the data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Zou and Schiebinger identified gender biases in AI algorithms.
“Similarly, Zou and Schiebinger [51] identified gender biases in AI algorithms, underscoring the intersection of technology with broader social justice issues.”
From The Role of AI in Nursing Education and Practice: Umbrella Review (2025) · cites this paper as Zou and Schiebinger [51]
AI systems may be systematically biased toward different outputs, which may lead to different or even unfair treatment, on the basis of extant features.
“It has been shown that AI systems may be systematically biased toward different outputs, which may lead to different or even unfair treatment, on the basis of extant features53,57–59.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [58]
Criticism around large models has been growing, especially ethical concerns regarding societal biases, carbon footprint, misuse and misinterpretation, privatization of AI research, and research opportunity costs.
“criticism around large models has been growing lately [9], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [9, 120], carbon footprint [9, 99], misuse and misinterpretation [9], privatization of AI research [99], and even research opportunity costs [49].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [120]
Abstractions in chatbot platforms can hide underlying information about machine learning models, AI decision-making, and latent bias in training data that can translate into social biases.
“abstractions can also hide underlying information about machine learning models, AI decision-making, as well as latent bias in the training data (e.g., [101]) that can translate into social biases (e.g., [120]).”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [120]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · see citing works
Self-citationShankar, S., Halpern, Y., Breck, E. J., Atwood, J. R., Wilson, J., & Sculley, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08536 · Cited by 175 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V. & Kalai, A. Adv. Neural Inf. Proc. Syst. 2016, 4349-4357 (2016).
- Wagner, C., Garcia, D., Jadidi, M. & Strohmaier, M. Proc. 9th Int. AAAI Conf. Web Soc. Media 454-463 (2015).
- Buolamwini, J. & Gebru, T. Proc. Mach. Learn. Res. 81, 1-15 (2018).
- Gebru, T. et al. Preprint at https://arxiv.org/ abs/1803.09010 (2018). · arXiv
- Dwork, C., Hardt, M., Pitassi, T., Reingold, O. & Zemel, R. Proc. 3rd Innov. Theor. Comp. Sci. Conf. 2012, 214-226 (2012).
- Zemel, R., Wu, Y., Swersky, K., Pitassi, T. & Dwork, C. Proc. 30th Int. Conf. Int. Conf. Mach. Learn. 28, III-325-III-333 (2013).
- Garg, N., Schiebinger, L., Jurafsky, D. & Zou, J. Proc. Natl Acad. Sci. USA 115, E3635-E3644 (2018).
Cited by
Sorted by citations
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · see citing works
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · see citing works
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1034-x · Cited by 1,421 · see citing works
Consumers and artificial intelligence: An experiential perspective
Puntoni, S., Reczek, R. W., Giesler, M., & Botti, S. (2021) · Journal of Marketing · doi:10.1177/0022242920953847 · Cited by 1,193 · see citing works
Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians
Asan, O., Bayrak, A. E., & Choudhury, A. (2020) · Journal of Medical Internet Research · doi:10.2196/15154 · Cited by 825 · see citing works
Liu, X., Rivera, S. C., Moher, D., Calvert, M. J., & Denniston, A. K. (2020) · BMJ · doi:10.1136/bmj.m3164 · Cited by 702 · see citing works
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685 · see citing works
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., Ashrafian, H., Beam, A. L., Chan, A.-W., Collins, G. S., Deeks, A. D. J., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30218-1 · Cited by 555 · see citing works
Responsible Artificial Intelligence Governance: A Review and Research Framework
Papagiannidis, E., Mikalef, P., & Conboy, K. (2025) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2024.101885 · Cited by 565 · see citing works
Kim, D. W., Jang, H. Y., Kim, K. W., Shin, Y., & Park, S. H. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0025 · Cited by 459 · see citing works
Can AI Help Reduce Disparities in General Medical and Mental Health Care?
(2019) · AMA Journal of Ethics · doi:10.1001/amajethics.2019.167 · Cited by 401 · see citing works
Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging
Arun, N., Gaw, N., Singh, P., Chang, K., Aggarwal, M., Chen, B., Hoebel, K., Gupta, S., Patel, J., Gidwani, M., et al. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021200267 · Cited by 342 · see citing works
Chi, O. H., Denton, G., & Gursoy, D. (2020) · Journal of Hospitality Marketing & Management · doi:10.1080/19368623.2020.1721394 · Cited by 334 · see citing works
Use of Digital Mental Health for Marginalized and Underserved Populations
Schueller, S. M., Hunter, J. F., Figueroa, C., & Aguilera, A. (2019) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-019-00181-z · Cited by 268 · see citing works
Deep learning for early warning signals of tipping points
Bury, T. M., Sujith, R. I., Pavithran, I., Scheffer, M., Lenton, T. M., Anand, M., & Bauch, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2106140118 · Cited by 284 · see citing works
Illuminating the dark spaces of healthcare with ambient intelligence
Haque, A., Milstein, A., & Fei-Fei, L. (2020) · Nature · doi:10.1038/s41586-020-2669-y · Cited by 274 · see citing works
Future directions for chatbot research: An interdisciplinary research agenda
Følstad, A., Araujo, T., Law, E. L.-C., Brandtzaeg, P. B., Papadopoulos, S., Reis, L., Baez, M., Laban, G., McAllister, P., Ischen, C., et al. (2021) · Computing · doi:10.1007/s00607-021-01016-7 · Cited by 247 · see citing works
Treating health disparities with artificial intelligence
Chen, I. Y., Joshi, S., & Ghassemi, M. (2020) · Nature Medicine · doi:10.1038/s41591-019-0649-2 · Cited by 235 · see citing works
Digital twins for predictive oncology will be a paradigm shift for precision cancer care
Hernandez-Boussard, T., Macklin, P., Greenspan, E. J., Gryshuk, A. L., Stahlberg, E., Syeda-Mahmood, T., & Shmulevich, I. (2021) · Nature Medicine · doi:10.1038/s41591-021-01558-5 · Cited by 219 · see citing works
Yang, J., Soltan, A. A. S., & Clifton, D. A. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00614-9 · Cited by 181 · see citing works
Algorithmic realism: expanding the boundaries of algorithmic thought
Green, B., & Viljoen, S. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372840 · Cited by 161 · see citing works
Algorithmic fairness in computational medicine
Xu, J., Xiao, Y., Wang, W. H., Ning, Y., Shenkman, E. A., Bian, J., & Wang, F. (2022) · eBioMedicine · doi:10.1016/j.ebiom.2022.104250 · Cited by 170 · see citing works
The Role of AI in Nursing Education and Practice: Umbrella Review
El Arab, R. A., Al Moosa, O. A., Abuadas, F. H., & Somerville, J. (2025) · Journal of Medical Internet Research · doi:10.2196/69881 · Cited by 160 · see citing works
“Similarly, Zou and Schiebinger [51] identified gender biases in AI algorithms, underscoring the intersection of technology with broader social justice issues.”
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · see citing works
Threats by artificial intelligence to human health and human existence
Federspiel, F., Mitchell, R., Asokan, A., Umana, C., & McCoy, D. (2023) · BMJ Global Health · doi:10.1136/bmjgh-2022-010435 · Cited by 156 · see citing works
Putting the data before the algorithm in big data addressing personalized healthcare
Cahan, E. M., Hernandez-Boussard, T., Thadaney-Israni, S., & Rubin, D. L. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0157-2 · Cited by 149 · see citing works
Artificial intelligence, bureaucratic form, and discretion in public service
Bullock, J., Young, M. M., & Wang, Y.-F. (2020) · Information Polity · doi:10.3233/ip-200223 · Cited by 139 · see citing works
Diversity, Equity, and Inclusion in Artificial Intelligence: An Evaluation of Guidelines
Cachat-Rosset, G., & Klarsfeld, A. (2023) · Applied Artificial Intelligence · doi:10.1080/08839514.2023.2176618 · Cited by 138 · see citing works
Why addressing digital inequality should be a priority
Imran, A. (2023) · THE ELECTRONIC JOURNAL OF INFORMATION SYSTEMS IN DEVELOPING COUNTRIES · doi:10.1002/isd2.12255 · Cited by 124 · see citing works
Hurtful words: quantifying biases in clinical contextual word embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384448 · Cited by 122 · see citing works
Steps to avoid overuse and misuse of machine learning in clinical research
Volovici, V., Syn, N. L., Ercole, A., Zhao, J. J., & Liu, N. (2022) · Nature Medicine · doi:10.1038/s41591-022-01961-6 · Cited by 110 · see citing works
Responsible AI for conservation
Wearn, O. R., Freeman, R., & Jacoby, D. M. P. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0022-7 · Cited by 112 · see citing works
Algorithmic inclusion: Shaping the predictive algorithms of artificial intelligence in hiring
Kelan, E. K. (2024) · Human Resource Management Journal · doi:10.1111/1748-8583.12511 · Cited by 106 · see citing works
DeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98 · see citing works
Peeking into a black box, the fairness and generalizability of a MIMIC-III benchmarking model.
Röösli, E., Bozkurt, S., & Hernandez-Boussard, T. (2022) · Scientific Data · doi:10.1038/s41597-021-01110-7 · Cited by 73 · see citing works
AI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71 · see citing works
One AI Does Not Fit All: A Cluster Analysis of the Laypeople’s Perception of AI Roles
Kim, T., Molina, M. D., Rheu, M. (., Zhan, E. S., & Peng, W. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581340 · Cited by 68 · see citing works
Biased data, biased AI: deep networks predict the acquisition site of TCGA images
Dehkharghanian, T., Bidgoli, A. A., Riasatian, A., Mazaheri, P., Campbell, C. J. V., Pantanowitz, L., Tizhoosh, H. R., & Rahnamayan, S. (2023) · Diagnostic Pathology · doi:10.1186/s13000-023-01355-3 · Cited by 67 · see citing works
A Survey of Machine Unlearning
Nguyen, T. T., Huynh, T. T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., & Nguyen, Q. V. H. (2022) · arXiv · doi:10.48550/arxiv.2209.02299 · Cited by 193 · see citing works
“more likely to consider African-American offenders to have higher risk scores than Caucasians, even though ethnicity information is not part of the input [181].”
Truth machines: synthesizing veracity in AI language models
Munn, L., Magee, L., & Arora, V. (2024) · AI & SOCIETY · doi:10.1007/s00146-023-01756-4 · Cited by 65 · see citing works
The ethical adoption of artificial intelligence in radiology
Mudgal, K. S., & Das, N. (2020) · BJR|Open · doi:10.1259/bjro.20190020 · Cited by 52 · see citing works
Evaluating artificial intelligence applications in clinical settings
Nsoesie, E. O. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.2658 · Cited by 48 · see citing works
Algorithmic fairness of machine learning models for Alzheimer disease progression
Yuan, C., Linn, K. A., & Hubbard, R. A. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.42203 · Cited by 49 · see citing works
The impact of ChatGPT and LLMs on medical imaging stakeholders: Perspectives and use cases
Yang, J., Li, H. B., & Wei, D. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100007 · Cited by 46 · see citing works
Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · arXiv · doi:10.48550/arxiv.2003.11515 · Cited by 37 · see citing works
Customer service with AI-powered human-robot collaboration (HRC): A literature review
Leocádio, D., Guedes, L., Oliveira, J., Reis, J., & Melão, N. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.01.120 · Cited by 37 · see citing works
Claudy, M. C., Aquino, K., & Graso, M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.898027 · Cited by 34 · see citing works
Kang, J., Thompson, R. F., Aneja, S., Lehman, C., Trister, A., Zou, J., Obcemea, C., & El Naqa, I. (2021) · Practical Radiation Oncology · doi:10.1016/j.prro.2020.06.001 · Cited by 30 · see citing works
Self-citationRating AI systems for bias to promote trustable applications
Srivastava, B., & Rossi, F. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2935966 · Cited by 22 · see citing works
Inspecting the Geographical Representativeness of Images from Text-to-Image Models
Basu, A., Babu, R. V., & Pruthi, D. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00474 · Cited by 21 · see citing works
Racial Bias Can Confuse AI for Genomic Studies
Dai, B., Xu, Z., Li, H., Wang, B., Cai, J., & Liu, X. (2022) · Oncologie · doi:10.32604/oncologie.2022.020259 · Cited by 18 · see citing works
The ethics of artificial intelligence in psychotherapy
Manríquez Roa, T., Biller-Andorno, N., & Trachsel, M. (2021) · Oxford Handbook of Psychotherapy Ethics · doi:10.1093/oxfordhb/9780198817338.013.91 · Cited by 19 · see citing works
Measuring and Mitigating Gender Bias in Legal Contextualized Language Models
Bozdag, M., Sevim, N., & Koç, A. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3628602 · Cited by 15 · see citing works
Frechman, E., Jaeger, B. C., Kowalkowski, M., Williamson, J. D., Lenoir, K. M., Palakshappa, J. A., Wells, B. J., Callahan, K. E., Pajewski, N. M., & Gabbard, J. L. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf062 · Cited by 3 · see citing works
- Authors
- Zou, J., & Schiebinger, L.
- Affiliations
- Stanford University (US)
- Corresponding author
- Zou, James
- Schiebinger, Londa
- Venue
- Nature
- Published
- 2018-07-18
- Volume
- 559
- Issue
- 7714
- Pages
- 324-326
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2883147591
- Cited by
- 765
- References
- 4
- Citations by year
- 2026: 60 · 2025: 108 · 2024: 138 · 2023: 108 · 2022: 93 · 2021: 112 · 2020: 94 · 2019: 46 · 2018: 5
- FWCI
- 127.66
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Nature (publishedVersion, other-oa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- algorithmic bias, sexism, racism, training data bias, bias mitigation, algorithmic discrimination, skewed data
- MeSH terms
- Machine Learning, Race Factors, Artificial Intelligence, Female, Humans, Male, Natural Language Processing, Sex Factors, Social Justice, Neural Networks, Computer, Racism, Sexism, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
