Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) systems for computer-aided diagnosis and image-based screening are being adopted worldwide by medical institutions. In such a context, generating fair and unbiased classifiers becomes of paramount importance. The research community of medical image computing is making great efforts in developing more accurate algorithms to assist medical doctors in the difficult task of disease diagnosis. However, little attention is paid to the way databases are collected and how this may influence the performance of AI systems. Our study sheds light on the importance of gender balance in medical imaging datasets used to train AI systems for computer-assisted diagnosis. We provide empirical evidence supported by a large-scale study, based on three deep neural network architectures and two well-known publicly available X-ray image datasets used to diagnose various thoracic diseases under different gender imbalance conditions. We found a consistent decrease in performance for underrepresented genders when a minimum balance is not fulfilled. This raises the alarm for national agencies in charge of regulating and approving computer-assisted diagnosis systems, which should include explicit gender balance and diversity recommendations. We also establish an open problem for the academic medical image computing community which needs to be addressed by novel algorithms endowed with robustness to gender imbalance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The choice of these subgroups is motivated by the clear history, in both traditional medicine and AI algorithms, of bias for subgroups on these axes 6,8,10,11 .”
From Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations (2021) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, if a dataset predominantly consists of images from a particular racial or ethnic group, the AI model trained on this dataset may exhibit reduced accuracy when applied to individuals from other groups.”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [24]
Experimental — accuracy not yet measuredIn the citing paper
“Similar issues arise with gender, age, and socio-economic status, where AI systems may perform better for the demographic groups that are overrepresented in the training data (Figure 4).24”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“Additionally, we evaluated two other CNN architectures, the ResNet (33) and the Inception-v3 (34), confirming that our observations generalize for different neural models.”
MethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“A model pretrained on ImageNet (32) was used to initialize the network weights.”
MethodDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · cited 4×
“( B and C ) AUC achieved for two exemplar diseases under a gradient of gender imbalance ratios, from 0% of female images in training data to 100%, with increments of 25%.”
MethodInfluentialProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · cited 1×
“AI systems cannot only augment the information provided by such images with useful annotations (1, 2), but they are also starting to take autonomous decisions by performing computer-aided diagnosis (CAD) (3, 4).”
BackgroundDermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 9×
“Given a frontal X-ray image, the CAD system predicts the presence or absence of the 14 thoracic diseases.”
BackgroundInfluentialWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 4×
“We use the NIH Chest-XRay14 dataset (16, 19), which includes 112,120 chest X-ray images from 30,805 patients, labeled with 14 common thorax diseases (including hernia, pneumonia, fibrosis, emphysema, edema, cardiomegaly, pleural thickening, consolidation, mass, pneumothorax, nodule, atelectasis, effusion, and infiltra…”
MethodInfluential(2018) · Cited by 3,510
Survey on deep learning with class imbalance
Johnson, J. M., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0192-5 · Cited by 3,032 · cited 1×
“It is worth mentioning that most of the existing work dealing with imbalanced data in the context of deep learning focuses on cases where it is related to the target classes (14, 15).”
BackgroundClinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639 · cited 9×
“Given a frontal X-ray image, the CAD system predicts the presence or absence of the 14 thoracic diseases.”
BackgroundInfluentialLearning from class-imbalanced data: Review of methods and applications
Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H., & Bing, G. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.12.035 · Cited by 2,377
Deep visual domain adaptation: A survey
Wang, M., & Deng, W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.083 · Cited by 2,209
Sex and Gender Equity in Research: rationale for the SAGER guidelines and recommended use
Heidari, S., Babor, T. F., De Castro, P., Tort, S., & Curno, M. (2016) · Research Integrity and Peer Review · doi:10.1186/s41073-016-0007-6 · Cited by 2,184 · cited 1×
“However, given that some anatomical attributes are reflected in X-ray images, the term sex could be more accurate, according to the Sex and Gender Equity in Research guidelines (29).”
MethodNo evidence that carotenoid pigments boost either immune or antioxidant defenses in a songbird
Koch, R. E., Kavazis, A. N., Hasselquist, D., Hood, W. R., Zhang, Y., Toomey, M. B., & Hill, G. E. (2018) · Nature Communications · doi:10.1038/s41467-018-02974-x · Cited by 1,924
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · cited 1×
“This tendency of AI systems to learn biased models, which reproduce social stereotypes and underperform in minority groups, is especially dangerous in the context of healthcare (11, 12).”
BackgroundMIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · cited 2×
“, the recent MIMIC-CXR (24) x-ray dataset or the Retinal Fundus Glaucoma Challenge (REFUGE) database of ophtalmological images (25), just to name a few].”
BackgroundMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266 · cited 1×
“This tendency of AI systems to learn biased models, which reproduce social stereotypes and underperform in minority groups, is especially dangerous in the context of healthcare (11, 12).”
BackgroundCheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219 · cited 4×
“We use the NIH Chest-XRay14 dataset (16, 19), which includes 112,120 chest X-ray images from 30,805 patients, labeled with 14 common thorax diseases (including hernia, pneumonia, fibrosis, emphysema, edema, cardiomegaly, pleural thickening, consolidation, mass, pneumothorax, nodule, atelectasis, effusion, and infiltra…”
MethodInfluentialOrlando, J. I., Fu, H., Barbosa Breda, J., van Keer, K., Bathula, D. R., Diaz-Pinto, A., Fang, R., Heng, P.-A., Kim, J., Lee, J., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101570 · Cited by 855 · cited 1×
“, the recent MIMIC-CXR (24) x-ray dataset or the Retinal Fundus Glaucoma Challenge (REFUGE) database of ophtalmological images (25), just to name a few].”
BackgroundAI can be sexist and racist — it’s time to make it fair
Zou, J., & Schiebinger, L. (2018) · Nature · doi:10.1038/d41586-018-05707-8 · Cited by 765
Enhancing studies of the connectome in autism using the autism brain imaging data exchange II
Di Martino, A., O’Connor, D., Chen, B., Alaerts, K., Anderson, J. S., Assaf, M., Balsters, J. H., Baxter, L., Beggiato, A., Bernaerts, S., et al. (2017) · Scientific Data · doi:10.1038/sdata.2017.10 · Cited by 710 · cited 1×
“, Autism Brain Imaging Data Exchange (ABIDE) I (28)].”
BackgroundDeep neural network improves fracture detection by clinicians
Lindsey, R., Daluiski, A., Chopra, S., Lachapelle, A., Mozer, M., Sicular, S., Hanel, D., Gardner, M., Gupta, A., Hotchkiss, R., et al. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806905115 · Cited by 640 · cited 1×
“AI systems cannot only augment the information provided by such images with useful annotations (1, 2), but they are also starting to take autonomous decisions by performing computer-aided diagnosis (CAD) (3, 4).”
BackgroundCheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.07031 · Cited by 392 · cited 2×
“The uncertainty labels included in CheXpert were interpreted as negative following the U-Zeros approach discussed in the original paper (30).”
MethodWhy rankings of biomedical image analysis competitions should be interpreted with care
Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A. P., Carass, A., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-07619-7 · Cited by 380 · cited 1×
“The same tendency is observed in many of the datasets included in a recent analysis of 150 databases from grand challenges on biomedical image analysis (26), which provides recommendations for database and challenge design, where there is no explicit mention of the importance of sex/gender demographic information.”
BackgroundComputer-aided detection in chest radiography based on artificial intelligence: a survey
Qin, C., Yao, D., Shi, Y., & Song, Z. (2018) · BioMedical Engineering OnLine · doi:10.1186/s12938-018-0544-y · Cited by 377 · cited 1×
“A special type of deep neural network, known as CNNs, was used to implement the CAD system (19, 20).”
MethodWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2019) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-030-13969-8_18 · Cited by 371
Evaluating Gender Bias in Machine Translation
Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1164 · Cited by 292
Schiebinger, L. L., & Schraudner, M. (2011) · Interdisciplinary Science Reviews · doi:10.1179/030801811x13013181961518 · Cited by 175
On Evaluating Artificial Intelligence Systems for Medical Diagnosis
Chandrasekaran, B. (1983) · AI Magazine · doi:10.1609/aimag.v4i2.397 · Cited by 31 · cited 1×
“Although the interest in performing fair and unbiased evaluations of AI medical systems has existed since the 1980s (5), the ethical aspects of AI have gained relevance in the last few years.”
BackgroundEven artificial intelligence can acquire biases against race and gender
Hutson, M. (2017) · Science · doi:10.1126/science.aal1053 · Cited by 23
Song, H.-J., Yoon, B.-H., Youn, Y.-S., Park, C.-Y., Kim, J.-D., & Kim, Y.-S. (2018) · BioMedical Engineering OnLine · doi:10.1186/s12938-018-0583-4 · Cited by 10
Cited by
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
“AI can make healthcare more accessible to under-served groups, but it also risks reinforcing existing inequities, because AI models can perpetuate biases lurking in the data 112 .”
BackgroundSeyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928
Machine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
Ethical Machine Learning in Healthcare
Chen, I. Y., Pierson, E., Rose, S., Joshi, S., Ferryman, K., & Ghassemi, M. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092820-114757 · Cited by 645
Self-supervised learning in medicine and healthcare
Krishnan, R., Rajpurkar, P., & Topol, E. J. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00914-1 · Cited by 605
“In addition, gender-biased datasets yield models that perform better on the majority class 59 .”
Background
Metadata
- Authors
- Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E.
- Affiliations
- Consejo Nacional de Investigaciones Científicas y Técnicas (AR)
- Corresponding author
- Ferrante, Enzo
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2020-05-26
- Volume
- 117
- Issue
- 23
- Pages
- 12592-12594
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC7293650
- OpenAlex
- W3030030520
- Cited by
- 685
- References
- 36
- Citations by year
- 2026: 117 · 2025: 149 · 2024: 125 · 2023: 110 · 2022: 101 · 2021: 71 · 2020: 10
- FWCI
- 21.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-sa)
- Funders
- AXA Research Fund (2017-AXA-PDOC1-097)
- MINCyT | Agencia Nacional de Promoción Científica y Tecnológica (PICT 2017-4596, PICT-2018-03907, PICT 2014-2627, PICT-2018-3384)
- Universidad Nacional del Litoral (CAID-PIC-50220140100084L, 2016-082)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Sex and Gender in Healthcare
- Keywords
- gender imbalance, medical imaging datasets, algorithmic fairness, computer-aided diagnosis, X-ray images, thoracic disease diagnosis, deep neural network, underrepresented groups, dataset collection
- MeSH terms
- Deep Learning, Female, Humans, Male, Radiographic Image Interpretation, Computer-Assisted, Reference Standards, Sex Factors, Radiography, Thoracic, Bias, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.