Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.

Generative models improve fairness of medical classifiers under distribution shifts

Open access

Published version of a preprint

Abstract

Domain generalization is a ubiquitous challenge for machine learning in healthcare. Model performance in real-world conditions might be lower than expected because of discrepancies between the data encountered during deployment and development. Underrepresentation of some groups or conditions during model development is a common cause of this phenomenon. This challenge is often not readily addressed by targeted data acquisition and ‘labeling’ by expert clinicians, which can be prohibitively expensive or practically impossible because of the rarity of conditions or the available clinical expertise. We hypothesize that advances in generative artificial intelligence can help mitigate this unmet need in a steerable fashion, enriching our training dataset with synthetic examples that address shortfalls of underrepresented conditions or subgroups. We show that diffusion models can automatically learn realistic augmentations from data in a label-efficient manner. We demonstrate that learned augmentations make models more robust and statistically fair in-distribution and out of distribution. To evaluate the generality of our approach, we studied three distinct medical imaging contexts of varying difficulty: (1) histopathology, (2) chest X-ray and (3) dermatology images. Complementing real samples with synthetic ones improved the robustness of models in all three medical tasks and increased fairness by improving the accuracy of clinical diagnosis within underrepresented groups, especially out of distribution.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41591-024-02838-6

Cites

83 of 83 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Generated samples and method overview. a, Samples generated by our conditional diffusion model for the different imaging modalities. b, Method overview. In the proposed approach, we first trained a diffusion model on both labeled and unlabeled data (if available). In a general setting, unlabeled data may consist of in-distribution or OOD data (for example, from an unseen hospital) for which expert labels are not available. Subsequently, we sampled synthetic images from the diffusion model according to particular specifications (for example, an image of a female individual with pulmonary edema). Finally, we trained a downstream diagnostic model on a combination of the real labeled images and the synthetic images sampled from the diffusion model. The dotted outlines represent synthetic data, while the dashed outlines represent unlabeled data.
Fig. 1 Generated samples and method overview. a, Samples generated by our conditional diffusion model for the different imaging modalities. b, Method overview. In the proposed approach, we first trained a diffusion model on both labeled and unlabeled data (if available). In a general setting, unlabeled data may consist of in-distribution or OOD data (for example, from an unseen hospital) for which expert labels are not available. Subsequently, we sampled synthetic images from the diffusion model according to particular specifications (for example, an image of a female individual with pulmonary edema). Finally, we trained a downstream diagnostic model on a combination of the real labeled images and the synthetic images sampled from the diffusion model. The dotted outlines represent synthetic data, while the dashed outlines represent unlabeled data.CC BY · PubMed Central
Fig. 2 Results on histopathology dataset. a, In-distribution fairness gap (in percentage) between the best and worst performing hospital versus overall prediction accuracy for the presence of breast cancer metastases in histopathology images on n = 33,560 samples. In the less skewed setting, we included 1,000 labeled samples from hospitals 3 and 4, while in the more skewed setting we included only 100 samples. b, OOD distribution results on n = 85,054 samples. Prediction accuracy (x axis) on the validation and test hospitals when training the generative model on all in-distribution labeled examples is shown. Note that the validation set was used for model selection, given that its distribution was more similar to the training distribution. We compared the following methods: baseline model with no augmentations; ‘Color augm.’ for a model that uses color augmentations; ‘Label conditioning’ and ‘Label and property conditioning’ for our proposed approach of a generative model conditioned on the diagnostic label and both the diagnostic label and the hospital ID, respectively; ‘L cond. + color augm.’ and ‘L and P cond. + color augm.’ were used to apply color augmentations on the images generated with the diffusion models. Combining color augmentation with synthetic data performed best across all settings. Data are presented as mean ± s.d. across five technical replicates.
Fig. 2 Results on histopathology dataset. a, In-distribution fairness gap (in percentage) between the best and worst performing hospital versus overall prediction accuracy for the presence of breast cancer metastases in histopathology images on n = 33,560 samples. In the less skewed setting, we included 1,000 labeled samples from hospitals 3 and 4, while in the more skewed setting we included only 100 samples. b, OOD distribution results on n = 85,054 samples. Prediction accuracy (x axis) on the validation and test hospitals when training the generative model on all in-distribution labeled examples is shown. Note that the validation set was used for model selection, given that its distribution was more similar to the training distribution. We compared the following methods: baseline model with no augmentations; ‘Color augm.’ for a model that uses color augmentations; ‘Label conditioning’ and ‘Label and property conditioning’ for our proposed approach of a generative model conditioned on the diagnostic label and both the diagnostic label and the hospital ID, respectively; ‘L cond. + color augm.’ and ‘L and P cond. + color augm.’ were used to apply color augmentations on the images generated with the diffusion models. Combining color augmentation with synthetic data performed best across all settings. Data are presented as mean ± s.d. across five technical replicates.CC BY · PubMed Central
Fig. 3 Results on chest radiology datasets. a,b, Comparison of average AUC versus fairness (AUC) gap across different baselines for radiology for sex (a) and race (b) for in-distribution (n = 23,261 samples) and OOD (n = 17,723 samples) datasets. Race labels are not available for the OOD dataset. For a, we report results in-distribution (left) and OOD (right) on CheXpert and ChestX-ray14 datasets, respectively. We marked the baseline ‘Pretrained on JFT’ with black. Label conditioning corresponds to the model that used synthetic images from a diffusion model conditioned on only the diagnostic labels. We further compared to other strong contenders, that is, a BiT-ResNet model pretrained on ImageNet-21K (Pretrained on IN-21K), a model pretrained on JFT using RandAugment heuristic augmentations (RandAugment), a model trained with RandAugment on top of standard ImageNet augmentations (RandAugment + IN Augms) and a model trained with focal loss (Focal loss). To ensure a fair comparison, all methods were trained and finetuned for the same number of steps and with the same batch size. For the fairness gap, smaller values are preferable. Data are presented as the mean ± s.d. across five technical replicates.
Fig. 3 Results on chest radiology datasets. a,b, Comparison of average AUC versus fairness (AUC) gap across different baselines for radiology for sex (a) and race (b) for in-distribution (n = 23,261 samples) and OOD (n = 17,723 samples) datasets. Race labels are not available for the OOD dataset. For a, we report results in-distribution (left) and OOD (right) on CheXpert and ChestX-ray14 datasets, respectively. We marked the baseline ‘Pretrained on JFT’ with black. Label conditioning corresponds to the model that used synthetic images from a diffusion model conditioned on only the diagnostic labels. We further compared to other strong contenders, that is, a BiT-ResNet model pretrained on ImageNet-21K (Pretrained on IN-21K), a model pretrained on JFT using RandAugment heuristic augmentations (RandAugment), a model trained with RandAugment on top of standard ImageNet augmentations (RandAugment + IN Augms) and a model trained with focal loss (Focal loss). To ensure a fair comparison, all methods were trained and finetuned for the same number of steps and with the same batch size. For the fairness gap, smaller values are preferable. Data are presented as the mean ± s.d. across five technical replicates.CC BY · PubMed Central
Fig. 4 Results on dermatology datasets. Comparison of high-risk sensitivity (for basal cell carcinoma, melanoma and SCC/SCCIS) versus fairness gap with regard to sex in dermatology across different baselines. We report results in-distribution (left) and OOD (right) for OOD 1, as well as for the less skewed (top) and more skewed (bottom) setting. We marked the baseline ‘Pretrained on JFT’ with black. ‘Label conditioning’ and ‘Label and property conditioning’ correspond to the models that used synthetic images sampled from a diffusion model conditioned on only the label, and the label and sensitive attribute, respectively. We further compared to other strong contenders, that is, a BiT-ResNet model pretrained on ImageNet-21K (Pretrained on IN-21K), a model pretrained on JFT using RandAugment heuristic augmentations (RandAugment), a model trained with RandAugment on top of standard ImageNet augmentations (RandAugment + IN Augms), a model trained on a resampled version of the training dataset that is more balanced with regard to the sensitive attribute (Oversampling) and a model trained with focal loss (Focal loss). To ensure a fair comparison, all methods were trained and finetuned for the same number of steps and with the same batch size. For the fairness gap, smaller values are preferable. There are n = 1,349 samples in the in-distribution dataset and n = 6,639 samples in the OOD dataset. Data are presented as the mean ± s.d. across five technical replicates.
Fig. 4 Results on dermatology datasets. Comparison of high-risk sensitivity (for basal cell carcinoma, melanoma and SCC/SCCIS) versus fairness gap with regard to sex in dermatology across different baselines. We report results in-distribution (left) and OOD (right) for OOD 1, as well as for the less skewed (top) and more skewed (bottom) setting. We marked the baseline ‘Pretrained on JFT’ with black. ‘Label conditioning’ and ‘Label and property conditioning’ correspond to the models that used synthetic images sampled from a diffusion model conditioned on only the label, and the label and sensitive attribute, respectively. We further compared to other strong contenders, that is, a BiT-ResNet model pretrained on ImageNet-21K (Pretrained on IN-21K), a model pretrained on JFT using RandAugment heuristic augmentations (RandAugment), a model trained with RandAugment on top of standard ImageNet augmentations (RandAugment + IN Augms), a model trained on a resampled version of the training dataset that is more balanced with regard to the sensitive attribute (Oversampling) and a model trained with focal loss (Focal loss). To ensure a fair comparison, all methods were trained and finetuned for the same number of steps and with the same batch size. For the fairness gap, smaller values are preferable. There are n = 1,349 samples in the in-distribution dataset and n = 6,639 samples in the OOD dataset. Data are presented as the mean ± s.d. across five technical replicates.CC BY · PubMed Central
Fig. 5 Generated images in the dermatology setting. Each row of images corresponds to a different condition. a, Generated images for cyst, melanocytic nevus and seborrheic dermatitis. b, Generated images for folliculitis, hidradenitis and alopecia areata.
Fig. 5 Generated images in the dermatology setting. Each row of images corresponds to a different condition. a, Generated images for cyst, melanocytic nevus and seborrheic dermatitis. b, Generated images for folliculitis, hidradenitis and alopecia areata.CC BY · PubMed Central
Extended Data Fig. 1 Additional results for histopathology and radiology. (a) Prediction accuracy for the presence of breast cancer metastases in whole-slide images of histological lymph node sections. In these experiments we vary the number of labeled samples N used to train the diffusion model from in-distribution datasets on in-distribution and out-of-distribution hospitals. Performance improvements are particularly striking in the low data regime (that is, small values of N). Combining color augmentation with synthetic data performs best across all settings (in and out-of-distribution). (b) Additional results for chest radiology. Data are presented as mean values ± SD across 5 technical replicates.
Extended Data Fig. 1 Additional results for histopathology and radiology. (a) Prediction accuracy for the presence of breast cancer metastases in whole-slide images of histological lymph node sections. In these experiments we vary the number of labeled samples N used to train the diffusion model from in-distribution datasets on in-distribution and out-of-distribution hospitals. Performance improvements are particularly striking in the low data regime (that is, small values of N). Combining color augmentation with synthetic data performs best across all settings (in and out-of-distribution). (b) Additional results for chest radiology. Data are presented as mean values ± SD across 5 technical replicates.CC BY · PubMed Central
Extended Data Fig. 2 Generated images in simple settings. (a) Synthetic images for healthy cells in histopathology. (b) Synthetic images for cancerous cells in histopathology. (c) Synthetic images of chest X-rays for healthy individuals. (d) Synthetic images of chest X-rays with the following conditions: atelectasis, cardiomegaly, consolidation, pleural effusion, pulmonary edema (one condition per row).
Extended Data Fig. 2 Generated images in simple settings. (a) Synthetic images for healthy cells in histopathology. (b) Synthetic images for cancerous cells in histopathology. (c) Synthetic images of chest X-rays for healthy individuals. (d) Synthetic images of chest X-rays with the following conditions: atelectasis, cardiomegaly, consolidation, pleural effusion, pulmonary edema (one condition per row).CC BY · PubMed Central
Extended Data Fig. 3 Chest radiology data statistics. Age histogram (a) and normalized label distributions (b) for the five conditions we consider in the CheXpert training dataset. Normalized co-occurrence of conditions in the CheXpert training (c) and the ChestX-ray14 evaluation (d) datasets. For each condition on the row r of each plot, the corresponding column c indicates the ratio of all samples with condition r that also have condition c. Note that more than two conditions can be present at once. We observe that in the training set it is much more common that more than one condition is present simultaneously.
Extended Data Fig. 3 Chest radiology data statistics. Age histogram (a) and normalized label distributions (b) for the five conditions we consider in the CheXpert training dataset. Normalized co-occurrence of conditions in the CheXpert training (c) and the ChestX-ray14 evaluation (d) datasets. For each condition on the row r of each plot, the corresponding column c indicates the ratio of all samples with condition r that also have condition c. Note that more than two conditions can be present at once. We observe that in the training set it is much more common that more than one condition is present simultaneously.CC BY · PubMed Central
Extended Data Fig. 4 Dermatology data statistics. Condition distributions for in-distribution and out-of-distribution dermatology datasets. In-domain and OOD 2 distributions are much more similar in comparison to OOD 1. In particular, 3 out of 4 of the most prevalent conditions (that is acne, eczema and other) in the in-distribution dataset are also the most prevalent in OOD 2. However, there are only few examples of high-risk conditions like basal cell carcinoma and SCC/SCCIS, which are the two most prevalent conditions in OOD 1. We can see that overall the right-most dataset has a similar label distribution to the training dataset and, hence, is ‘less’ out-of-distribution than the other one.
Extended Data Fig. 4 Dermatology data statistics. Condition distributions for in-distribution and out-of-distribution dermatology datasets. In-domain and OOD 2 distributions are much more similar in comparison to OOD 1. In particular, 3 out of 4 of the most prevalent conditions (that is acne, eczema and other) in the in-distribution dataset are also the most prevalent in OOD 2. However, there are only few examples of high-risk conditions like basal cell carcinoma and SCC/SCCIS, which are the two most prevalent conditions in OOD 1. We can see that overall the right-most dataset has a similar label distribution to the training dataset and, hence, is ‘less’ out-of-distribution than the other one.CC BY · PubMed Central
Extended Data Fig. 5 Generated images in dermatology are canonical examples of the condition. (Left) The diffusion model can produce an infinite amount of synthetic images for a particular condition that is inherently more scarce (we have fewer than 50 samples of melanoma in our training dataset). (Middle) The images that experts have identified with melanoma with high confidence (a combination of individual’s confidence in diagnosis and expert consensus). (Right) The nearest neighbours from the training samples identified for each of the synthetic images based on l-norm in pixel space.
Extended Data Fig. 5 Generated images in dermatology are canonical examples of the condition. (Left) The diffusion model can produce an infinite amount of synthetic images for a particular condition that is inherently more scarce (we have fewer than 50 samples of melanoma in our training dataset). (Middle) The images that experts have identified with melanoma with high confidence (a combination of individual’s confidence in diagnosis and expert consensus). (Right) The nearest neighbours from the training samples identified for each of the synthetic images based on l-norm in pixel space.CC BY · PubMed Central
Extended Data Fig. 6 Misclassification rates for different subgroups. Mis-classification analysis on individuals with high-risk conditions depending on their sex attribute. We investigate the impact of a skewed training dataset with respect to the underrepresented sex on correctly detecting high-risk conditions for that subgroup in- and out-of-distribution. In each plot, we show the test sample index on the x-axis to investigate which individuals are consistently misclassified by all approaches. Sample indices are re-ordered such as to form contiguous blocks of correctly and incorrectly classified samples. It is worth noting that the source domain is more skewed towards females (see Table A1 for details).
Extended Data Fig. 6 Misclassification rates for different subgroups. Mis-classification analysis on individuals with high-risk conditions depending on their sex attribute. We investigate the impact of a skewed training dataset with respect to the underrepresented sex on correctly detecting high-risk conditions for that subgroup in- and out-of-distribution. In each plot, we show the test sample index on the x-axis to investigate which individuals are consistently misclassified by all approaches. Sample indices are re-ordered such as to form contiguous blocks of correctly and incorrectly classified samples. It is worth noting that the source domain is more skewed towards females (see Table A1 for details).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 73 references

Research works printed but not linked: 14

  1. Irvin, J. et al. Proc. AAAI Conference on Artificial Intelligence (AAAI, 2019).
  2. Wang, X. et al. Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2017).
  3. Cheng, V., Suriyakumar, V. M., Dullerud, N., Joshi, S. & Ghassemi, M. Proc. 2021 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery, 2021).
  4. Tomasev, N., McKee, K. R., Kay, J. & Mohamed, S. Proc. 2021 AAAI/ACM Conference on AI, Ethics, and Society (Association for Computing Machinery, 2021).
  5. Goodfellow, I. et al. Generative adversarial nets. In Advances in Neural Information Processing Systems (eds. Ghahramani, Z. et al.) 2672–2680 (2014).
  6. Rashid, H., Tanveer, M. A. & Khan, H. A. Proc. 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE, 2019).
  7. Gowal, S. et al. Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE, 2020).
  8. Bissoto, A., Valle, E. & Avila, S. Proc. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (IEEE, 2021).
  9. Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. Proc. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2023).
  10. Sun, C., Shrivastava, A., Singh, S. & Gupta, A. Proc. 2017 IEEE International Conference on Computer Vision (ICCV) (IEEE, 2017).
  11. Deng, J. et al. Proc. 2009 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2009).
  12. Cubuk, E. D., Zoph, B., Shlens, J. & Le, Q. V. Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (CVPRW) (IEEE, 2020).
  13. Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Dollár, P. Proc. IEEE International Conference on Computer Vision (ICCV) (IEEE, 2017).
  14. Bommasani, R., Creel, K., Kumar, A., Jurafsky, D. & Liang, P. Picking on the same person: does algorithmic monoculture lead to outcome homogenization? In Proc. Advances in Neural Information Processing Systems (eds. Koyejo, S. et al.) 3663–3678 (Curran Associates, Inc., 2022)

Cited by

See all 209

Metadata

Authors
Ktena, I., Wiles, O., Albuquerque, I., Rebuffi, S.-A., Tanno, R., Roy, A. G., Azizi, S., Belgrave, D., Kohli, P., Cemgil, T., Karthikesalingam, A., & Gowal, S.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • Google (United Kingdom) (GB)
  • GlaxoSmithKline (United Kingdom) (GB)
Corresponding author
Wiles, Olivia
Venue
Nature Medicine
Published
2024-04
Volume
30
Issue
4
Pages
1166-1173
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
OpenAlex
W4394683903
Cited by
165
References
73
Citations by year
2026: 58 · 2025: 78 · 2024: 28
FWCI
15.36
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
Keywords
generative models, diffusion models, domain generalization, distribution shift, algorithmic fairness, synthetic data augmentation, underrepresented groups, medical image classification, histopathology, chest radiography, dermatological images, model robustness, label efficiency
MeSH terms
Machine Learning, Artificial Intelligence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Generative models improve fairness of medical classifiers under distribution shifts. https://app.cochat.ai/papers/10.1038/s41591-024-02838-6
The paper
Ktena, I., Wiles, O., Albuquerque, I., Rebuffi, S.-A., Tanno, R., Roy, A. G., Azizi, S., Belgrave, D., Kohli, P., Cemgil, T., Karthikesalingam, A., & Gowal, S. (2024). Generative models improve fairness of medical classifiers under distribution shifts. Nature Medicine. https://doi.org/10.1038/s41591-024-02838-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.