Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.

Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4

Open access

Published version of a preprint

Abstract

Large language models (LLMs) are seen to have tremendous potential in advancing medical diagnosis recently, particularly in dermatological diagnosis, which is a very important task as skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases. Here we present SkinGPT-4, which is an interactive dermatology diagnostic system based on multimodal large language models. We have aligned a pre-trained vision transformer with an LLM named Llama-2-13b-chat by collecting an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors’ notes, and designing a two-step training strategy. We have quantitatively evaluated SkinGPT-4 on 150 real-life cases with board-certified dermatologists. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identify the characteristics and categories of the skin conditions, perform in-depth analysis, and provide interactive treatment recommendations.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-024-50043-3

Cites

79 of 79 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Illustration of SkinGPT-4. SkinGPT-4 is an interactive dermatology diagnostic system based on multimodal large language models. To implement SkinGPT-4, we have designed a framework that aligned a pre-trained vision transformer with a large language model named Llama-2-13b-chat. SkinGPT-4 was trained on a vast collection (52,929) of both public and in-house skin disease images, accompanied by clinical concepts and doctors’ notes. With SkinGPT-4, users could upload their own skin photos for diagnosis, and SkinGPT-4 could autonomously determine the characteristics and categories of skin conditions, perform analysis, provide treatment recommendations, and allow interactive diagnosis. On the right is an example of interactive diagnosis.
Fig. 1 Illustration of SkinGPT-4. SkinGPT-4 is an interactive dermatology diagnostic system based on multimodal large language models. To implement SkinGPT-4, we have designed a framework that aligned a pre-trained vision transformer with a large language model named Llama-2-13b-chat. SkinGPT-4 was trained on a vast collection (52,929) of both public and in-house skin disease images, accompanied by clinical concepts and doctors’ notes. With SkinGPT-4, users could upload their own skin photos for diagnosis, and SkinGPT-4 could autonomously determine the characteristics and categories of skin conditions, perform analysis, provide treatment recommendations, and allow interactive diagnosis. On the right is an example of interactive diagnosis.CC BY · PubMed Central
Fig. 2 Illustration of our datasets for two-step training of SkinGPT-4. The notes below each image indicate clinical concepts and types of skin diseases. In addition, we have detailed descriptions from the board-certified dermatologists for images in the step 2 dataset. To avoid causing discomfort, we used a translucent grey box to obscure the displayed skin disease images.
Fig. 2 Illustration of our datasets for two-step training of SkinGPT-4. The notes below each image indicate clinical concepts and types of skin diseases. In addition, we have detailed descriptions from the board-certified dermatologists for images in the step 2 dataset. To avoid causing discomfort, we used a translucent grey box to obscure the displayed skin disease images.CC BY · PubMed Central
Fig. 3 Diagnosis generated by SkinGPT-4, SkinGPT-4 (step 1 only), SkinGPT-4 (step 2 only), MiniGPT-4, and dermatologists. This figure shows a case of actinic keratosis.
Fig. 3 Diagnosis generated by SkinGPT-4, SkinGPT-4 (step 1 only), SkinGPT-4 (step 2 only), MiniGPT-4, and dermatologists. This figure shows a case of actinic keratosis.CC BY · PubMed Central
Fig. 4 Clinical evaluation of SkinGPT-4 by board-certified offline and online dermatologists. a Questionnaire-based assessment of SkinGPT-4 by offline dermatologists. The barplot represents the percentage of skin disease cases that the dermatologists agree on. b Response time of SkinGPT-4 (n = 20) is lower compared to consulting dermatologists online (n = 20) (two-tailed Student’s t test with P < 0.00001). All boxplots represent the first quartile, the median, and the third quartile. The upper whisker indicates the maximum value no further than 1.5 times the interquartile range from the third quartile. The lower whisker indicates the minimum value no further than 1.5 times the interquartile range from the first quartile. Source data are provided as a Source Data file. c Consistency test of SkinGPT-4’s responses. The x axis indicates test samples, and the y axis indicates the diagnostic results.
Fig. 4 Clinical evaluation of SkinGPT-4 by board-certified offline and online dermatologists. a Questionnaire-based assessment of SkinGPT-4 by offline dermatologists. The barplot represents the percentage of skin disease cases that the dermatologists agree on. b Response time of SkinGPT-4 (n = 20) is lower compared to consulting dermatologists online (n = 20) (two-tailed Student’s t test with P < 0.00001). All boxplots represent the first quartile, the median, and the third quartile. The upper whisker indicates the maximum value no further than 1.5 times the interquartile range from the third quartile. The lower whisker indicates the minimum value no further than 1.5 times the interquartile range from the first quartile. Source data are provided as a Source Data file. c Consistency test of SkinGPT-4’s responses. The x axis indicates test samples, and the y axis indicates the diagnostic results.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 79 references

Research works printed but not linked: 5

  1. Mijwil, M., Aljanabi, M. & Ali, A. H. Chatgpt: exploring the role of cybersecurity in the protection of medical information. Mesop. J. Cybersecur. 2023, 18–21 (2023).
  2. Lund, B., Agbaji, D. & Teel, Z. A. Information literacy, data literacy, privacy literacy, and ChatGPT: technology literacies align with perspectives on emerging technology adoption within communities. Lund., B, Agbaji, D., Teel, ZA (2023) Inf. Lit., Data Lit., Priv. Lit., ChatGPT: Technol. Literacies Align Perspect. Emerg. Technol. Adoption Commun. Hum. Technol. 19, 163–177 (2023).
  3. Ordonez, V., Kulkarni, G. & Berg, T. Im2text: describing images using 1 million captioned photographs. Adv. Neural Inf. Process. Syst. 24 (2011).
  4. Daneshjou, R., Yuksekgonul, M., Cai, Z. R., Novoa, R. & Zou, J. Y. Skincon: a skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. Adv. Neural Inf. Process. Syst. 35, 18157–18167 (2022).
  5. Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017).

Cited by

See all 189

Metadata

Authors
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., & Gao, X.
Affiliations
  • King Abdullah University of Science and Technology (SA)
  • Capital Medical University (CN)
  • Brigham and Women's Hospital (US)
Corresponding author
He, Xiaonan
Venue
Nature Communications
Published
2024-07-05
Volume
15
Issue
1
Pages
5649-5649
ISSN
2041-1723 (ISSN-L)
OpenAlex
W4400362569
Cited by
189
References
79
Citations by year
2026: 84 · 2025: 97 · 2024: 8
FWCI
17.32
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Touro Scholar (Touro College) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
King Abdullah University of Science and Technology (FCC/1/1976-44-01)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
dermatological diagnosis, multimodal large language models, clinical notes, treatment guidelines
MeSH terms
Machine Learning, Computer Simulation, Dermatology, Models, Biological, Skin Diseases, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4. https://app.cochat.ai/papers/10.1038/s41467-024-50043-3
The paper
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., & Gao, X. (2024). Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4. Nature Communications. https://doi.org/10.1038/s41467-024-50043-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.