Accounting for diversity in AI for medicine
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
In healthcare, gender and sex considerations are crucial because they affect individuals' health and disease differences. Yet, most algorithms deployed in the healthcare context do not consider these aspects and do not account for bias detection. Missing these dimensions in algorithms used in medicine is a huge point of concern, as neglecting these aspects will inevitably produce far from optimal results and generate errors that may lead to misdiagnosis and potential discrimination. This paper explores how current algorithmic-based systems may reinforce gender biases and affect marginalized communities in healthcare-related applications. To do so, we bring together notions and reflections from computer science, queer media studies, and legal insights to better understand the magnitude of failing to consider gender and sex difference in the use of algorithms for medical purposes. Our goal is to illustrate the potential impact that algorithmic bias may have on inadvertent discriminatory, safety, and privacy-related concerns for patients in increasingly automated medicine. This is necessary because by rushing the deployment of AI technologies that do not account for diversity, we risk having an even more unsafe and inadequate healthcare delivery. By promoting the account for privacy, safety, diversity, and inclusion in algorithmic developments with health-related outcomes, we ultimately aim to inform the Artificial Intelligence (AI) global governance landscape and practice on the importance of integrating gender and sex considerations in the development of algorithms to avoid exacerbating existing or new prejudices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Private traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
Haenlein, M., & Kaplan, A. (2019) · California Management Review · doi:10.1177/0008125619864925 · Cited by 2,655
Sex and gender: modifiers of health, disease, and medicine
Mauvais-Jarvis, F., Bairey Merz, N., Barnes, P. J., Brinton, R. D., Carrero, J.-J., DeMeo, D. L., De Vries, G. J., Epperson, C. N., Govindan, R., Klein, S. L., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)31561-0 · Cited by 2,001
Privacy in the age of medical big data
Price, W. N., & Cohen, I. G. (2019) · Nature Medicine · doi:10.1038/s41591-018-0272-7 · Cited by 1,522
Artificial intelligence in surgery: promises and perils
Hashimoto, D. A., Rosman, G., Rus, D., & Meireles, O. R. (2018) · Annals of Surgery · doi:10.1097/sla.0000000000002693 · Cited by 1,468
Overview of artificial intelligence in medicine
Amisha, Malik, P., Pathania, M., & Rathaur, V. K. (2019) · Journal of Family Medicine and Primary Care · doi:10.4103/jfmpc.jfmpc_440_19 · Cited by 1,106
Privacy Preserving Data Mining
Lindell, Y., & Pinkas, B. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-44598-6_3 · Cited by 1,077
Inkster, B., Sarda, S., & Subramanian, V. (2018) · doi:10.2196/preprints.12106 · Cited by 1,063
Lee, D., & Yoon, S. N. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18010271 · Cited by 1,017
Zhang, J., Gajjala, S., Agrawal, P., Tison, G. H., Hallock, L. A., Beussink-Nelson, L., Lassen, M. H., Fan, E., Aras, M. A., Jordan, C., et al. (2018) · Circulation · doi:10.1161/circulationaha.118.034338 · Cited by 1,006
Yu, K.-H., Zhang, C., Berry, G. J., Altman, R. B., Ré, C., Rubin, D. L., & Snyder, M. (2016) · Nature Communications · doi:10.1038/ncomms12474 · Cited by 972
The impact of artificial intelligence in medicine on the future role of the physician
Ahuja, A. S. (2019) · PeerJ · doi:10.7717/peerj.7702 · Cited by 789
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685
Singh, H., Meyer, A. N. D., & Thomas, E. J. (2014) · BMJ Quality & Safety · doi:10.1136/bmjqs-2013-002627 · Cited by 677
Towards transparency by design for artificial intelligence
Felzmann, H., Fosch-Villaronga, E., Lutz, C., & Tamò-Larrieux, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00276-4 · Cited by 588
Self-citationSex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553
A Right to Reasonable Inferences: Re-Thinking Data Protection Law in the Age of Big Data and AI
Wachter, S., & Mittelstadt, B. (2018) · doi:10.31228/osf.io/mu2kf · Cited by 463
The Intuitive Appeal of Explainable Machines
Selbst, A. D., & Barocas, S. (2018) · doi:10.2139/ssrn.3126971 · Cited by 392
Definitions of Gender and Sex: The Subtleties of Meaning
Pryzgoda, J., & Chrisler, J. C. (2000) · Sex Roles · doi:10.1023/a:1007123617636 · Cited by 268
Gender-related variables for health research
Nielsen, M. W., Stefanick, M. L., Peragine, D., Neilands, T. B., Ioannidis, J. P. A., Pilote, L., Prochaska, J. J., Cullen, M. R., Einstein, G., Klinge, I., et al. (2021) · Biology of Sex Differences · doi:10.1186/s13293-021-00366-3 · Cited by 238
Ethics of healthcare robotics: Towards responsible research and innovation
Stahl, B. C., & Coeckelbergh, M. (2016) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2016.08.018 · Cited by 235
The Role of Informatics in Promoting Patient-Centered Care
Snyder, C. F., Wu, A. W., Miller, R. S., Jensen, R. E., Bantug, E. T., & Wolff, A. C. (2011) · The Cancer Journal · doi:10.1097/ppo.0b013e318225ff89 · Cited by 166
Artificial intelligence for breast cancer screening: Opportunity or hype?
Houssami, N., Lee, C. I., Buist, D. S. M., & Tao, D. (2017) · The Breast · doi:10.1016/j.breast.2017.09.003 · Cited by 106
Discrimination and Privacy in the Information Society: Data Mining and Profiling in Large Databases
(2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3 · Cited by 100
Self-citationDiversity and Inclusion Metrics in Subset Selection
Mitchell, M., Baker, D., Moorosi, N., Denton, R., Hutchinson, B., Hanna, A., Gebru, T., & Morgenstern, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375832 · Cited by 87
La Fors, K., Custers, B., & Keymolen, E. (2019) · Ethics and Information Technology · doi:10.1007/s10676-019-09503-4 · Cited by 50
Self-citation
Cited by
Camilleri, M. A. (2024) · Expert Systems · doi:10.1111/exsy.13406 · Cited by 331
Abbas, A., Rehman, M. S., & Rehman, S. S. (2024) · Cureus · doi:10.7759/cureus.55991 · Cited by 58
“Expanding the diversity of medical subjects on which LLMs are trained is also vital, as studies have shown that current algorithmic-based systems may reinforce gender biases and affect marginalized communities in healthcare-related applications [24].”
BackgroundLu, Y., & Zhang, J. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2024.104205 · Cited by 44
Next-generation endoscopy in inflammatory bowel disease
Zammarchi, I., Santacroce, G., & Iacucci, M. (2023) · Diagnostics · doi:10.3390/diagnostics13152547 · Cited by 28
“Moreover, the lack of diversity in the training data can lead to biased and suboptimal AI algorithms, potentially compromising their accuracy in real-world clinical settings [103].”
BackgroundTowards experimental standardization for AI governance in the EU
Prifti, K., & Fosch-Villaronga, E. (2024) · Computer Law & Security Review · doi:10.1016/j.clsr.2024.105959 · Cited by 12
Self-citation
Metadata
- Authors
- Fosch-Villaronga, E., Drukarch, H., Khanna, P., Verhoef, T., & Custers, B.
- Venue
- Computer Law & Security Review
- Publisher
- Elsevier BV
- Published
- 2022-11
- Volume
- 47
- Pages
- 105735
- ISSN
- 2212-473X (ISSN-L)
- OpenAlex
- W4292656665
- Cited by
- 50
- References
- 35
- FWCI
- 12.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.