CoChat
Sign Up
Cited by 125Artificial Intelligence in Healthcare and EducationExplainable Artificial Intelligence (XAI)Multimodal Machine Learning Applications

The DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index.

Transparent medical image AI via an image–text foundation model grounded in medical literature

Open access
Kim, Chanwoo, Gadgil, Soham U., DeGrave, Alex J., Omiye, Jesutofunmi A., Cai, Zhuo Ran, Daneshjou, Roxana, Lee, Su-In·2024·Nature Medicine
DOI 10.1038/s41591-024-02887-xPMC PMC10312868

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Abstract Building trustworthy and transparent image-based medical AI systems requires the ability to interrogate data and models at all stages of the development pipeline: from training models to post-deployment monitoring. Ideally, the data and associated AI systems could be described using terms already familiar to physicians, but this requires medical datasets densely annotated with semantically meaningful concepts. Here, we present a foundation model approach, named MONET ( M edical c ON cept r ET riever), which learns how to connect medical images with text and generates dense concept annotations to enable tasks in AI transparency from model auditing to model interpretation. Dermatology provides a demanding use case for the versatility of MONET, due to the heterogeneity in diseases, skin tones, and imaging modalities. We trained MONET on the basis of 105,550 dermatological images paired with natural language descriptions from a large collection of medical literature. MONET can accurately annotate concepts across dermatology images as verified by board-certified dermatologists, outperforming supervised models built on previously concept-annotated dermatology datasets. We demonstrate how MONET enables AI transparency across the entire AI development pipeline from dataset auditing to model auditing to building inherently interpretable models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41591-024-02887-x

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 | Overview of MONET framework and its usage examples. (A) Training procedure. MONET is trained using contrastive learning on an extensive set of dermatology image and text pairs collected from PubMed articles and medical textbooks. During the training process, the paired image and text are forced to be close in the joint representation space, while those from different pairs are forced to be far apart. (B) Automatic concept generation. MONET can map medical concepts and images onto a joint representation space, allowing it to determine the degree to which a concept is present in an image for any given concept by measuring the distance between the image and concept text prompts in the representation space. Its concept generation capability enables various concept-driven analyses at multiple stages of the medical AI pipeline. (C) Concept-level data auditing. MONET’s automatic concept generation capability makes it possible to explain the distinguishing features between two sets of data in the language of human-interpretable concepts. This approach facilitates the auditing of large-scale datasets with ease. (D) Concept-level model auditing. MONET can be used to identify which input characteristic leads to the errors of medical AI. (E) Developing inherently interpretable models. MONET can be used to develop inherently interpretable medical AI models that operate on human-interpretable concepts aligning with physicians’ expectations. These models allow physicians to easily decipher the factors influencing the models’ decisions, ensuring high transparency.
Fig. 1 | Overview of MONET framework and its usage examples. (A) Training procedure. MONET is trained using contrastive learning on an extensive set of dermatology image and text pairs collected from PubMed articles and medical textbooks. During the training process, the paired image and text are forced to be close in the joint representation space, while those from different pairs are forced to be far apart. (B) Automatic concept generation. MONET can map medical concepts and images onto a joint representation space, allowing it to determine the degree to which a concept is present in an image for any given concept by measuring the distance between the image and concept text prompts in the representation space. Its concept generation capability enables various concept-driven analyses at multiple stages of the medical AI pipeline. (C) Concept-level data auditing. MONET’s automatic concept generation capability makes it possible to explain the distinguishing features between two sets of data in the language of human-interpretable concepts. This approach facilitates the auditing of large-scale datasets with ease. (D) Concept-level model auditing. MONET can be used to identify which input characteristic leads to the errors of medical AI. (E) Developing inherently interpretable models. MONET can be used to develop inherently interpretable medical AI models that operate on human-interpretable concepts aligning with physicians’ expectations. These models allow physicians to easily decipher the factors influencing the models’ decisions, ensuring high transparency.CC BY-NC · PubMed Central
Fig. 2 | Images with high concept presence scores calculated using MONET. The concept presence score represents the degree to which a concept is present in an image. Each row displays the top 10 images for each concept. (A) Clinical images from the Fitzpatrick17k and DDI datasets. We exclude images inappropriate for public display due to the inclusion of sensitive body parts; for completeness, we denote the filenames of these files in Supplementary Table 1 (B) Dermoscopy images from the ISIC dataset.
Fig. 2 | Images with high concept presence scores calculated using MONET. The concept presence score represents the degree to which a concept is present in an image. Each row displays the top 10 images for each concept. (A) Clinical images from the Fitzpatrick17k and DDI datasets. We exclude images inappropriate for public display due to the inclusion of sensitive body parts; for completeness, we denote the filenames of these files in Supplementary Table 1 (B) Dermoscopy images from the ISIC dataset.CC BY-NC · PubMed Central
Fig. 3 | Concept-level data auditing. (A) We perform concept differential analysis between malignant images and benign images in the ISIC dataset. We show the top 10 concepts with positive values and the top 5 concepts with negative values. A positive value means the concept was more present in the malignant images than in the benign images, and vice versa. (B) We perform concept differential analysis between malignant and benign images per data source in the ISIC dataset to identify data-source-specific trends. The purple bar represents the output from the Medical University of Vienna, and the green bar represents the output from the Hospital Clinic de Barcelona. We show the top 15 concepts based on their absolute differences between the two cohorts. (C) Examples of red images in each cohort. We display 10 randomly selected images from the top 100 images in each cohort that had high concept expression scores for redness. (D) Precision-recall curve for images in each cohort. The images in each cohort are sorted based on their concept presence scores for redness and then compared to their malignancy labels. Precision is defined as the proportion of malignant images above a certain threshold out of all images above that threshold, while recall rate is defined as the proportion of malignant images above the threshold out of all malignant images. The top 500 and top 1000 red images from Barcelona Hospital still contain more malignant than benign samples.
Fig. 3 | Concept-level data auditing. (A) We perform concept differential analysis between malignant images and benign images in the ISIC dataset. We show the top 10 concepts with positive values and the top 5 concepts with negative values. A positive value means the concept was more present in the malignant images than in the benign images, and vice versa. (B) We perform concept differential analysis between malignant and benign images per data source in the ISIC dataset to identify data-source-specific trends. The purple bar represents the output from the Medical University of Vienna, and the green bar represents the output from the Hospital Clinic de Barcelona. We show the top 15 concepts based on their absolute differences between the two cohorts. (C) Examples of red images in each cohort. We display 10 randomly selected images from the top 100 images in each cohort that had high concept expression scores for redness. (D) Precision-recall curve for images in each cohort. The images in each cohort are sorted based on their concept presence scores for redness and then compared to their malignancy labels. Precision is defined as the proportion of malignant images above a certain threshold out of all images above that threshold, while recall rate is defined as the proportion of malignant images above the threshold out of all malignant images. The top 500 and top 1000 red images from Barcelona Hospital still contain more malignant than benign samples.CC BY-NC · PubMed Central
Fig. 4 | Concept-level model auditing. (A) We perform a benchmark analysis to see how well “model auditing with MONET” (MA-MONET) can identify the semantically meaningful concepts that lead to model error. To this end, we generate settings where we know the ground truth (i.e., concepts that lead to model errors); we create a training and test dataset with spurious correlation. We use MA-MONET to identify which concepts lead to model error for an AI model trained on this confounded dataset. MA-MONET returns a ranked list of concepts that explain model errors. (B) The frequency of the known spurious correlation being recovered by MA-MONET is shown. (D)-(E) Each row displays one of the top 5 clusters, sorted by high error rates. For each cluster, we show the misclassified images and the corresponding concepts associated with errors. We represent the true and predicted labels for each image by the color of the upper left and lower right triangles in the small box, respectively. The numbers at the top right compare the number of malignant and benign samples for the true and predicted labels. The 5 misclassified images shown for each are selected based on the average concept presence of the identified concepts.
Fig. 4 | Concept-level model auditing. (A) We perform a benchmark analysis to see how well “model auditing with MONET” (MA-MONET) can identify the semantically meaningful concepts that lead to model error. To this end, we generate settings where we know the ground truth (i.e., concepts that lead to model errors); we create a training and test dataset with spurious correlation. We use MA-MONET to identify which concepts lead to model error for an AI model trained on this confounded dataset. MA-MONET returns a ranked list of concepts that explain model errors. (B) The frequency of the known spurious correlation being recovered by MA-MONET is shown. (D)-(E) Each row displays one of the top 5 clusters, sorted by high error rates. For each cluster, we show the misclassified images and the corresponding concepts associated with errors. We represent the true and predicted labels for each image by the color of the upper left and lower right triangles in the small box, respectively. The numbers at the top right compare the number of malignant and benign samples for the true and predicted labels. The 5 misclassified images shown for each are selected based on the average concept presence of the identified concepts.CC BY-NC · PubMed Central
Fig. 5 | Concept bottleneck model. (A) Concept bottleneck model built using concepts generated by MONET (blue). The model first generates concepts using MONET and then predicts disease labels by combining them via a linear model. Concept bottleneck model built using concepts manually labeled by experts (green). The model uses manually annotated concept labels to predict disease labels using a linear model. Manual annotations take a lot longer than concept generation using MONET. (B)-(C) Performance of a malignancy prediction model trained using manual labels with respect to the number of concepts and the number of expert-labeled samples. (D)-(E) Performance of a melanoma prediction model trained using manual labels with respect to the number of concepts and the number of expert-labeled samples. (B)-(E) MONET+CBM is shown as a cross mark because it can utilize all concepts without expert annotation. The shaded area represents the 95% confidence interval. (F) Performance comparison of malignancy prediction models. (G) Performance comparison of melanoma prediction models. (F)-(G) Unlike (B)-(E), MONET+CBM uses task-relevant concepts curated by dermatologists. Each dot represents the AUC measure for individual runs with a different train-test split. The box represents the interquartile range with its lower and upper bounds corresponding to the first quartile and third quartile, respectively. p values derived from one-sided paired t-tests comparing MONET+CBM and other methods are indicated: *<0.05, **<0.01, ***<0.001; n=20 runs of each method. (H) Coefficient of the linear model in MONET+CBM for malignancy prediction. (I) Coefficient of the linear model in MONET+CBM for melanoma prediction. (H)-(I) The error bars present the 95% confidence interval.
Fig. 5 | Concept bottleneck model. (A) Concept bottleneck model built using concepts generated by MONET (blue). The model first generates concepts using MONET and then predicts disease labels by combining them via a linear model. Concept bottleneck model built using concepts manually labeled by experts (green). The model uses manually annotated concept labels to predict disease labels using a linear model. Manual annotations take a lot longer than concept generation using MONET. (B)-(C) Performance of a malignancy prediction model trained using manual labels with respect to the number of concepts and the number of expert-labeled samples. (D)-(E) Performance of a melanoma prediction model trained using manual labels with respect to the number of concepts and the number of expert-labeled samples. (B)-(E) MONET+CBM is shown as a cross mark because it can utilize all concepts without expert annotation. The shaded area represents the 95% confidence interval. (F) Performance comparison of malignancy prediction models. (G) Performance comparison of melanoma prediction models. (F)-(G) Unlike (B)-(E), MONET+CBM uses task-relevant concepts curated by dermatologists. Each dot represents the AUC measure for individual runs with a different train-test split. The box represents the interquartile range with its lower and upper bounds corresponding to the first quartile and third quartile, respectively. p values derived from one-sided paired t-tests comparing MONET+CBM and other methods are indicated: *<0.05, **<0.01, ***<0.001; n=20 runs of each method. (H) Coefficient of the linear model in MONET+CBM for malignancy prediction. (I) Coefficient of the linear model in MONET+CBM for melanoma prediction. (H)-(I) The error bars present the 95% confidence interval.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 72 references

Research works printed but not linked: 16

  1. Daneshjou, R., Yuksekgonul, M., Cai, Z. R., Novoa, R. & Zou, J. Y. SkinCon: a skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. In Advances in Neural Information Processing Systems (eds Koyejo, S. et al.) 18157–18167 (Curran Associates, Inc., 2022).
  2. Combalia, M. et al. BCN20000: dermoscopic lesions in the wild. Preprint at https://arxiv.org/abs/1908.02288 (2019). · arXiv
  3. Singh, C., Balakrishnan, G. & Perona, P. Matched sample selection with GANs for mitigating attribute confounding. Preprint at https://arxiv.org/abs/2103.13455 (2021). · arXiv
  4. Goel, K., Gu, A., Li, Y. & Ré, C. Model patching: closing the subgroup performance gap with data augmentation. In 9th International Conference on Learning Representations (ICLR, 2021); https://openreview.net/forum?id=9YlaeLfuhJF
  5. Sagawa, S., Koh, P. W., Hashimoto, T. B. & Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations (ICLR, 2020); https://dblp.org/rec/conf/iclr/SagawaKHL20.html?view=bibtex
  6. Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. In Proc. 31st International Conference on Neural Information Processing Systems (eds Guyon, I. et al.) 4768–4777 (Curran Associates Inc., 2017).
  7. Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. In Proc. 34th International Conference on Machine Learning (eds Precup, D. & Teh, Y. W.) 3319–3328 (PMLR, 2017).
  8. Crabbé, J. & van der Schaar, M. Concept activation regions: a generalized framework for concept-based explanations. In Advances in Neural Information Processing Systems (eds Koyejo, S. et al.) 2590–2607 (Curran Associates Inc., 2022).
  9. Abid, A., Yuksekgonul, M. & Zou, J. Meaningfully debugging model mistakes using conceptual counterfactual explanations. In Proc. 39th International Conference on Machine Learning (eds Chaudhuri, K. et al.) 66–68 (PMLR, 2022).
  10. Yuksekgonul, M., Wang, M. & Zou, J. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations (ICLR, 2023); https://dblp.org/rec/conf/iclr/YuksekgonulW023.html?view=bibtex
  11. PMC Open Access Subset. National Library of Medicine www.ncbi.nlm.nih.gov/pmc/tools/openftlist (2022).
  12. Tan, M. & Le, Q. V. EfficientNetV2: smaller models and faster training. In Proc. 38th International Conference on Machine Learning, ICML 2021 (eds Meila, M. & Zhang, T.) 10096–10106 (PMLR, 2021).
  13. Dosovitskiy, A. et al. An image is worth 16 × 16 words: transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021 (ICLR, 2021); https://openreview.net/forum?id=YicbFdNTTy
  14. Sennrich, R., Haddow, B. & Birch, A. Neural machine translation of rare words with subword units. In Proc. 54th Annual Meeting of the Association for Computational Linguistics, Vol. 1: Long Papers 1715–1725 (Association for Computational Linguistics, 2016). · DOI · Paper page
  15. Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015 (eds Bengio, Y. & LeCun, Y.) (ICLR, 2015).
  16. Wu, C., Parbhoo, S., Havasi, M. & Doshi-Velez, F. Learning optimal summaries of clinical time-series with concept bottleneck models. In Proc. 7th Machine Learning for Healthcare Conference (eds Lipton, Z. C. et al.) 648–672 (PMLR, 2022).

Cited by

Sorted by citations

See all 125

Related

Metadata

Authors
Kim, C., Gadgil, S. U., DeGrave, A. J., Omiye, J. A., Cai, Z. R., Daneshjou, R., & Lee, S.-I.
Affiliations
  • University of Washington (US)
  • Stanford Medicine (US)
  • Stanford University (US)
Corresponding author
  • Daneshjou, Roxana
  • Lee, Su-In
ORCID
Venue
Nature Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-04
Volume
30
Issue
4
Pages
1154-1165
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
DOI
10.1038/s41591-024-02887-x
PMC
PMC10312868
OpenAlex
W4394832225
Cited by
125
References
72
Citations by year
2026: 48 · 2025: 50 · 2024: 25
FWCI
11.52
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc)
Funders
  • National Science Foundation (DBI-1552309, DBI-1759487)
  • U.S. Department of Health & Human Services | National Institutes of Health (R01 AG061132, R35 GM 128638, 5T32 AR007422-38, AR007422-38)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
vision-language models, medical literature, dermatological images, model auditing, data auditing, model interpretability, AI transparency, self-supervised learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review