CoChat
Sign Up
PreprintCited by 17Artificial Intelligence in Healthcare and EducationCutaneous Melanoma Detection and ManagementArtificial Intelligence in Healthcare

The DOI matches its DataCite record; 9 of the 9 references it cites were found in a registry or an index.

SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Model

Open access
Zhou, Juexiao, He, Xiaonan, Sun, Liyuan, Xu, Jiannan, Chen, Xiuying, Chu, Yuetan, Zhou, Longxi, Liao, Xingyu, Zhang, Bin, Gao, Xin·2023·arXiv
DOI 10.48550/arxiv.2304.10691arXiv 2304.10691
Open this paper in CoChat — free accountOpen PDF

Abstract

Skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases, impacting a considerable portion of the population. Nonetheless, the field of dermatology diagnosis faces three significant hurdles. Firstly, there is a shortage of dermatologists accessible to diagnose patients, particularly in rural regions. Secondly, accurately interpreting skin disease images poses a considerable challenge. Lastly, generating patient-friendly diagnostic reports is usually a time-consuming and labor-intensive task for dermatologists. To tackle these challenges, we present SkinGPT-4, which is the world's first interactive dermatology diagnostic system powered by an advanced visual large language model. SkinGPT-4 leverages a fine-tuned version of MiniGPT-4, trained on an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors' notes. We designed a two-step training process to allow SkinGPT to express medical features in skin disease images with natural language and make accurate diagnoses of the types of skin diseases. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identifies the characteristics and categories of the skin conditions, performs in-depth analysis, and provides interactive treatment recommendations. Meanwhile, SkinGPT-4's local deployment capability and commitment to user privacy also render it an appealing choice for patients in search of a dependable and precise diagnosis of their skin ailments. To demonstrate the robustness of SkinGPT-4, we conducted quantitative evaluations on 150 real-life cases, which were independently reviewed by certified dermatologists, and showed that SkinGPT-4 could provide accurate diagnoses of skin diseases.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.10691

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 57

  1. R. J. Hay, N. E. Johns, H. C. Williams, I. W. Bolliger, R. P. Dellavalle, D. J. Margolis, R. Marks, L. Naldi, M. A. Weinstock, S. K. Wulf et al., "The global burden of skin disease in 2010: an analysis of the prevalence and impact of skin conditions," Journal of Investigative Dermatology, vol. 134, no. 6, pp. 1527-1534, 2014.
  2. H. Feng, J. Berk-Krauss, P. W. Feng, and J. A. Stein, "Comparison of dermatologist density between urban and rural counties in the united states," JAMA dermatology, vol. 154, no. 11, pp. 1265-1271, 2018.
  3. J. Resneck Jr and A. B. Kimball, "The dermatology workforce shortage," Journal of the American Academy of Dermatology, vol. 50, no. 1, pp. 50-54, 2004.
  4. Y. Liu, A. Jain, C. Eng, D. H. Way, K. Lee, P. Bui, K. Kanada, G. de Oliveira Marinho, J. Gallegos, S. Gabriele et al., "A deep learning system for differential diagnosis of skin diseases," Nature medicine, vol. 26, no. 6, pp. 900-908, 2020.
  5. D. Seth, K. Cheldize, D. Brown, and E. E. Freeman, "Global burden of skin disease: inequities and innovations," Current dermatology reports, vol. 6, pp. 204-210, 2017.
  6. D. G. Federman, J. Concato, and R. S. Kirsner, "Comparison of dermatologic diagnoses by primary care practitioners and derma- tologists: a review of the literature," Archives of family medicine, vol. 8, no. 2, p. 170, 1999.
  7. G. Moreno, H. Tran, A. L. Chia, A. Lim, and S. Shumack, "Prospec- tive study to assess general practitioners' dermatological diagnos- tic skills in a referral setting," Australasian journal of dermatology, vol. 48, no. 2, pp. 77-82, 2007.
  8. K. M. Yim, A. G. Florek, D. H. Oh, K. McKoy, and A. W. Arm- strong, "Teledermatology in the united states: an update in a dynamic era," Telemedicine and e-Health, vol. 24, no. 9, pp. 691-697, 2018.
  9. P. R. Kshirsagar, H. Manoharan, S. Shitharth, A. M. Alshareef, N. Albishry, and P. K. Balachandran, "Deep learning approaches for prognosis of automated skin disease," Life, vol. 12, no. 3, p. 426, 2022.
  10. F. Martora, A. Ruggiero, G. Fabbrocini, and A. Villani, "Patient satisfaction with remote dermatology consultations during the covid-19 pandemic. comment on 'a qualitative assessment of patient satisfaction with remote dermatology consultations used during the uk's first wave of the covid-19 pandemic in a single, secondary-care dermatology department'," Clinical and Experimen- tal Dermatology, vol. 47, no. 11, pp. 2037-2038, 2022.
  11. R. L ópez-Liria, M. Á. Valverde-Martínez, A. L ópez-Villegas, R. J. Bautista-Mesa, F. A. Vega-Ramírez, S. Peir ó, and C. Leal-Costa, "Teledermatology versus face-to-face dermatology: An analysis of cost-effectiveness from eight studies from europe and the united states," International Journal of Environmental Research and Public Health, vol. 19, no. 5, p. 2534, 2022.
  12. N. Lakdawala, C. Gronbeck, and H. Feng, "Workforce charac- teristics of nonphysician clinicians in dermatology in the united states," Journal of the American Academy of Dermatology, vol. 87, no. 5, pp. 1108-1110, 2022.
  13. I. K. Pious and R. Srinivasan, "A review on early diagnosis of skin cancer detection using deep learning techniques," in 2022 International Conference on Computer, Power and Communications (ICCPC). IEEE, 2022, pp. 247-253.
  14. P. Puri, N. Comfere, L. A. Drage, H. Shamim, S. A. Bezalel, M. R. Pittelkow, M. D. Davis, M. Wang, A. R. Mangold, M. M. Tollefson et al., "Deep learning for dermatologists: Part ii. current applica- tions," Journal of the American Academy of Dermatology, vol. 87, no. 6, pp. 1352-1360, 2022.
  15. S. Reshma and S. Reeja, "A review of computer assistance in der- matology," in 2023 International Conference on Intelligent and Inno- vative Technologies in Computing, Electrical and Electronics (IITCEE). IEEE, 2023, pp. 66-71.
  16. S. S. Han, I. Park, S. E. Chang, W. Lim, M. S. Kim, G. H. Park, J. B. Chae, C. H. Huh, and J.-I. Na, "Augmented intelligence derma- tology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders," Journal of Investigative Dermatology, vol. 140, no. 9, pp. 1753-1761, 2020.
  17. D. Popescu, M. El-Khatib, H. El-Khatib, and L. Ichim, "New trends in melanoma detection using neural networks: a systematic review," Sensors, vol. 22, no. 2, p. 496, 2022.
  18. A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, "Dermatologist-level classification of skin cancer with deep neural networks," nature, vol. 542, no. 7639, pp. 115- 118, 2017.
  19. S. S. Han, M. S. Kim, W. Lim, G. H. Park, I. Park, and S. E. Chang, "Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm," Journal of Investigative Dermatology, vol. 138, no. 7, pp. 1529-1538, 2018.
  20. H. A. Haenssle, C. Fink, R. Schneiderbauer, F. Toberer, T. Buhl, A. Blum, A. Kalloo, A. B. H. Hassen, L. Thomas, A. Enk et al., "Man against machine: diagnostic performance of a deep learning con- volutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists," Annals of oncology, vol. 29, no. 8, pp. 1836-1842, 2018.
  21. M. A. Marchetti, N. C. Codella, S. W. Dusza, D. A. Gutman, B. Helba, A. Kalloo, N. Mishra, C. Carrera, M. E. Celebi, J. L. DeFazio et al., "Results of the 2016 international skin imaging collaboration international symposium on biomedical imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images," Journal of the American Academy of Dermatology, vol. 78, no. 2, pp. 270-277, 2018.
  22. T. J. Brinker, A. Hekler, A. Hauschild, C. Berking, B. Schilling, A. H. Enk, S. Haferkamp, A. Karoglan, C. von Kalle, M. Weichenthal et al., "Comparing artificial intelligence algorithms to 157 german dermatologists: the melanoma classification benchmark," European Journal of Cancer, vol. 111, pp. 30-37, 2019.
  23. J. Yap, W. Yolland, and P. Tschandl, "Multimodal skin lesion clas- sification using deep learning," Experimental dermatology, vol. 27, no. 11, pp. 1261-1267, 2018.
  24. s. L. P. Aggarwal, "Data augmentation in dermatology image recognition using machine learning," Skin Research and Technology, vol. 25, no. 6, pp. 815-820, 2019.
  25. P. Tschandl, C. Rosendahl, B. N. Akay, G. Argenziano, A. Blum, R. P. Braun, H. Cabo, J.-Y. Gourhant, J. Kreusch, A. Lallas et al., "Expert-level diagnosis of nonpigmented skin cancer by combined convolutional neural networks," JAMA dermatology, vol. 155, no. 1, pp. 58-65, 2019.
  26. S. S. Han, I. J. Moon, W. Lim, I. S. Suh, S. Y. Lee, J.-I. Na, S. H. Kim, and S. E. Chang, "Keratinocytic skin cancer detection on the face using region-based convolutional neural network," JAMA dermatology, vol. 156, no. 1, pp. 29-37, 2020.
  27. O. Jones, R. Matin, M. van der Schaar, K. P. Bhayankaram, C. Ran- muthu, M. Islam, D. Behiyat, R. Boscott, N. Calanzani, J. Emery et al., "Artificial intelligence and machine learning algorithms for early detection of skin cancer in community and primary care settings: a systematic review," The Lancet Digital Health, vol. 4, no. 6, pp. e466-e476, 2022.
  28. A. Hekler, J. S. Utikal, A. H. Enk, C. Berking, J. Klode, D. Schaden- dorf, P. Jansen, C. Franklin, T. Holland-Letz, D. Krahl et al., "Pathologist-level classification of histopathological melanoma images with deep neural networks," European Journal of Cancer, vol. 115, pp. 79-83, 2019.
  29. Y. Jiang, J. Xiong, H. Li, X. Yang, W. Yu, M. Gao, X. Zhao, Y. Ma, W. Zhang, Y. Guan et al., "Recognizing basal cell carcinoma on smartphone-captured digital histopathology images with a deep neural network," British Journal of Dermatology, vol. 182, no. 3, pp. 754-762, 2020.
  30. A. Hekler, J. S. Utikal, A. H. Enk, W. Solass, M. Schmitt, J. Klode, D. Schadendorf, W. Sondermann, C. Franklin, F. Bestvater et al., "Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images," European Journal of Cancer, vol. 118, pp. 91-96, 2019.
  31. D. Roffman, G. Hart, M. Girardi, C. J. Ko, and J. Deng, "Predicting non-melanoma skin cancer via a multi-parameterized artificial neural network," Scientific reports, vol. 8, no. 1, p. 1701, 2018.
  32. J. P. Lott, D. M. Boudreau, R. L. Barnhill, M. A. Weinstock, E. Knopp, M. W. Piepkorn, D. E. Elder, S. R. Knezevich, A. Baer, A. N. Tosteson et al., "Population-based analysis of histologically confirmed melanocytic proliferations using natural language pro- cessing," JAMA dermatology, vol. 154, no. 1, pp. 24-29, 2018.
  33. S. S. Han, G. H. Park, W. Lim, M. S. Kim, J. I. Na, I. Park, and S. E. Chang, "Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagno- sis: Automatic construction of onychomycosis datasets by region- based convolutional deep neural network," PloS one, vol. 13, no. 1, p. e0191493, 2018.
  34. E. Bernardis and L. Castelo-Soccio, "Quantifying alopecia areata via texture analysis to automate the salt score computation," in Journal of Investigative Dermatology Symposium Proceedings, vol. 19, no. 1. Elsevier, 2018, pp. S34-S40.
  35. A. H. Thieme, Y. Zheng, G. Machiraju, C. Sadee, M. Mittermaier, M. Gertler, J. L. Salinas, K. Srinivasan, P. Gyawali, F. Carrillo-Perez et al., "A deep-learning algorithm to classify skin lesions from mpox virus infection," Nature Medicine, vol. 29, no. 3, pp. 738-747, 2023.
  36. A. A. Cruz-Roa, J. E. Arevalo Ovalle, A. Madabhushi, and F. A. González Osorio, "A deep learning architecture for im- age representation, visual interpretability and automated basal- cell carcinoma cancer detection," in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2013: 16th International Conference, Nagoya, Japan, September 22-26, 2013, Proceedings, Part II 16. Springer, 2013, pp. 403-410.
  37. Y. Yuan, M. Chao, and Y.-C. Lo, "Automatic skin lesion seg- mentation using deep fully convolutional networks with jaccard distance," IEEE transactions on medical imaging, vol. 36, no. 9, pp. 1876-1886, 2017.
  38. P. Tschandl, N. Codella, B. N. Akay, G. Argenziano, R. P. Braun, H. Cabo, D. Gutman, A. Halpern, B. Helba, R. Hofmann-Wellenhof et al., "Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classifi- cation: an open, web-based, international, diagnostic study," The lancet oncology, vol. 20, no. 7, pp. 938-947, 2019.
  39. X. Sun, J. Yang, M. Sun, and K. Wang, "A benchmark for auto- matic visual classification of clinical skin disease images," in Com- puter Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14. Springer, 2016, pp. 206-222.
  40. T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepa ño, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo et al., "Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models," PLoS digital health, vol. 2, no. 2, p. e0000198, 2023.
  41. M. Sallam, N. Salim, M. Barakat, and A. Al-Tammemi, "Chat- gpt applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations," Narra J, vol. 3, no. 1, pp. e103-e103, 2023.
  42. P. Lee, S. Bubeck, and J. Petro, "Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine," New England Journal of Medicine, vol. 388, no. 13, pp. 1233-1239, 2023.
  43. M. Balas and E. B. Ing, "Conversational ai models for ophthalmic diagnosis: Comparison of chatgpt and the isabel pro differential diagnosis generator," JFO Open Ophthalmology, vol. 1, p. 100005, 2023.
  44. M. Mijwil, M. Aljanabi, and A. H. Ali, "Chatgpt: Exploring the role of cybersecurity in the protection of medical information," Mesopotamian journal of cybersecurity, vol. 2023, pp. 18-21, 2023.
  45. R. K. Sinha, A. D. Roy, N. Kumar, H. Mondal, and R. Sinha, "Ap- plicability of chatgpt in assisting to solve higher order problems in pathology," Cureus, vol. 15, no. 2, 2023.
  46. R. Vaishya, A. Misra, and A. Vaish, "Chatgpt: Is this version good for healthcare and research?" Diabetes & Metabolic Syndrome: Clinical Research & Reviews, vol. 17, no. 4, p. 102744, 2023.
  47. E. J. Beltrami and J. M. Grant-Kels, "Consulting chatgpt: Ethical dilemmas in language model artificial intelligence," Journal of the American Academy of Dermatology, 2023.
  48. P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol, "Ai in health and medicine," Nature medicine, vol. 28, no. 1, pp. 31-38, 2022.
  49. J. Zhou, S. Chen, Y. Wu, H. Li, B. Zhang, L. Zhou, Y. Hu, Z. Xiang, Z. Li, N. Chen et al., "Ppml-omics: a privacy-preserving federated machine learning system protects patients' privacy from omic data," bioRxiv, pp. 2022-03, 2022.
  50. J. Zhou, L. Zhou, D. Wang, X. Xu, H. Li, Y. Chu, W. Han, and X. Gao, "Personalized and privacy-preserving federated hetero- geneous medical image analysis with pppml-hmi," medRxiv, pp. 2023-02, 2023.
  51. J. Zhou, H. Li, X. Liao, B. Zhang, W. He, Z. Li, L. Zhou, and X. Gao, "Audit to forget: A unified method to revoke patients' private data in intelligent healthcare," bioRxiv, pp. 2023-02, 2023.
  52. P. Sharma, N. Ding, S. Goodman, , and R. Soricut, "Concep- tual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning," In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018.
  53. V. Ordonez, G. Kulkarni, and T. Berg, "Im2text: Describing im- ages using 1 million captioned photographs," Advances in neural information processing systems, vol. 24, 2011.
  54. C. Preiksaitis, C. A. Sinsky, and C. Rose, "Chatgpt is not the solution to physicians' documentation burden," Nature Medicine, pp. 1-2, 2023.
  55. W. Fu and C. J. Cockerell, "The actinic (solar) keratosis: a 21st- century perspective," Archives of dermatology, vol. 139, no. 1, pp. 66-70, 2003.
  56. G. R. Kanthraj, "The twenty factors that made teledermatology consultation a matured application: A systematic review," Clinical Dermatology Review, vol. 7, no. 1, pp. 10-15, 2023.
  57. R. Daneshjou, M. Yuksekgonul, Z. R. Cai, R. Novoa, and J. Y. Zou, "Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis," Advances in Neural Information Processing Systems, vol. 35, pp. 18 157-18 167, 2022.

Cited by

Sorted by citations

See all 17

Related

See all

Metadata

Authors
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., & Gao, X.
Affiliations
  • King Abdullah University of Science and Technology (SA)
  • Capital Medical University (CN)
ORCID
Zhou, Juexiao
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.10691
arXiv
2304.10691
OpenAlex
W4366850928
Cited by
17
References
0
Citations by year
2026: 3 · 2025: 6 · 2024: 5 · 2023: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
dermatological diagnosis, skin disease classification, skin lesion image analysis, treatment guidelines, local deployment, patient privacy, dermatologist shortage, rural health care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review