CoChat
Sign Up
PreprintCited by 16Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationCutaneous Melanoma Detection and Management

Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology

Open access
Yan, Siyuan, Hu, Ming, Jiang, Yiwen, Li, Xieji, Fei, Hao, Tschandl, Philipp, Kittler, Harald, Ge, Zongyuan·2025·arXiv
DOI 10.48550/arxiv.2503.14911arXiv 2503.14911
Open this paper in CoChat — free accountOpen PDF

Abstract

The emergence of vision-language models has transformed medical AI, enabling unprecedented advances in diagnostic capability and clinical applications. However, progress in dermatology has lagged behind other medical domains due to the lack of standard image-text pairs. Existing dermatological datasets are limited in both scale and depth, offering only single-label annotations across a narrow range of diseases instead of rich textual descriptions, and lacking the crucial clinical context needed for real-world applications. To address these limitations, we present Derm1M, the first large-scale vision-language dataset for dermatology, comprising 1,029,761 image-text pairs. Built from diverse educational resources and structured around a standard ontology collaboratively developed by experts, Derm1M provides comprehensive coverage for over 390 skin conditions across four hierarchical levels and 130 clinical concepts with rich contextual information such as medical history, symptoms, and skin tone. To demonstrate Derm1M potential in advancing both AI research and clinical application, we pretrained a series of CLIP-like models, collectively called DermLIP, on this dataset. The DermLIP family significantly outperforms state-of-the-art foundation models on eight diverse datasets across multiple tasks, including zero-shot skin disease classification, clinical and artifacts concept identification, few-shot/full-shot learning, and cross-modal retrieval. Our dataset and code will be publicly available at https://github.com/SiyuanYan1/Derm1M upon acceptance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.14911

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The development of foundation models is reshaping dermatology, with multiple models such as SkinGPT4, PanDerm, and Derm1M.

    “The development of foundation models is also reshaping the dermatology domain, with multiple models such as SkinGPT4 2 , PanDerm 4 , Derm1M 3 , paving the way for an increased use of such models in the domain.”

    From Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets (2026) · cites this paper as [3]

    The Introduction (p. 2) says Derm1M advances dermatology AI and reports results for a model trained on it, supporting its relevance to foundation-model development. But the Conclusion (p. 8) identifies Derm1M as a vision-language dataset, not a model; the models are called DermLIP. Thus the citation supports Derm1M’s role in this development, but not its inclusion in a list of models. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Most public dermatology datasets are small, unimodal, and paired with heterogeneous labels and metadata, restricting effective model development.

    “Most publicly available dermatology datasets are unimodal, limited in size, and composed mainly of dermoscopic or clinical images 3,4,13,20,21,42-47 , often photographs, paired with heterogeneous metadata or categorical labels that are lacking expressive textual information 48 .”

    From Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets (2026) · cites this paper as [3]

    The Introduction (p. 2) describes existing datasets as limited in size and clinical context, and says the scarcity of image-text pairs constrains model development. The Abstract (p. 1) says they commonly offer single-label annotations rather than rich textual descriptions. These passages support the central point, but do not establish that their labels or metadata are heterogeneous. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 43

  1. [1][1] Michael Binder, Heinz Kittler, A Seeber, A Steiner, H Pehamberger, and K Wolff. Epiluminescence microscopybased classification of pigmented skin lesions using computerized image analysis and an artificial neural network. Melanoma research , 8(3):261-266, 1998. 2 -
  2. [2][2] Otto Braun-Falco, Gerd Plewig, Helmut H Wolff, and Richard K Winkelmann. Dermatology . Springer Science &Business Media, 2013. 2 -
  3. [3][3] Souradip Chakraborty, Ekaba Bisong, Shweta Bhatt, Thomas Wagner, Riley Elliott, and Francesco Mosconi. Biomedbert: A pre-trained biomedical language model for qa and ir. In Proceedings of the 28th international conference on computational linguistics , pages 669-679, 2020. 6 -
  4. [4][4] SI Cho, S Sun, J-H Mun, C Kim, SY Kim, S Cho, SW Youn, HC Kim, and JH Chung. Dermatologist-level classification of malignant lip diseases using a deep convolutional neural network. British Journal of Dermatology , 182(6):13881394, 2020. 2 -
  5. [5][5] Matthew Christensen, Milos Vukadinovic, Neal Yuan, and David Ouyang. Vision-language foundation model for echocardiogram interpretation. Nature Medicine , 30(5): 1481-1488, 2024. 8 -
  6. [6][6] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: Achallenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018) , pages 168-172. IEEE, 2018. 2, 3, 6, 7, 14, 17 -
  7. [7][7] Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Y Zou. Skincon: A skin disease dataset densely annotated by domain experts for finegrained debugging and analysis. Advances in Neural Information Processing Systems , 35:18157-18167, 2022. 2, 3, 6, 7 -
  8. [9][9] David Doukhan, Jean Carrive, F´ elicien Vallet, Anthony Larcher, and Sylvain Meignier. An open-source speaker gender detection framework for monitoring gender equality. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 5214-5218. IEEE, 2018. 4 -
  9. [10][10] Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and 11. Serena Yeung-Levy. Describing differences in image sets with natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24199-24208, 2024. 8 -
  10. [11][11] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature , 542(7639):115-118, 2017. 2 -
  11. [12][12] Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher R´ e. Domino: Discovering systematic errors with cross-modal embeddings. arXiv preprint arXiv:2203.14960 , 2022. 8 - · arXiv
  12. [13][13] Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1820-1828, 2021. 6, 8 -
  13. [14][14] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) , 3(1): 1-23, 2021. 6 -
  14. [15][15] Roderick J Hay, Nicole E Johns, Hywel C Williams, Ian W Bolliger, Robert P Dellavalle, David J Margolis, Robin Marks, Luigi Naldi, Martin A Weinstock, Sarah K Wulf, et al. The global burden of skin disease in 2010: an analysis of the prevalence and impact of skin conditions. Journal of investigative dermatology , 134(6):1527-1534, 2014. 2 -
  15. [16][16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 2, 6 -
  16. [17][17] Ming Hu, Kun Yuan, Yaling Shen, Feilong Tang, Xiaohao Xu, Lin Zhou, Wei Li, Ying Chen, Zhongxing Xu, Zelin Peng, et al. Ophclip: Hierarchical retrieval-augmented learning for ophthalmic surgical video-language pretraining. arXiv preprint arXiv:2411.15421 , 2024. 2 - · arXiv
  17. [18][18] Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology. Advances in neural information processing systems , 36, 2024. 2, 3, 9, 11 -
  18. [19][19] Jeremy Kawahara, Sara Daneshvar, Giuseppe Argenziano, and Ghassan Hamarneh. Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics , 23(2): 538-546, 2018. 3, 6, 7 -
  19. [21][21] Chanwoo Kim, Soham U Gadgil, Alex J DeGrave, Jesutofunmi A Omiye, Zhuo Ran Cai, Roxana Daneshjou, and Su-In Lee. Transparent medical image ai via an image-text foundation model grounded in medical literature. Nature Medicine , pages 1-12, 2024. 2, 6, 7, 8, 9, 12, 18 -
  20. [22][22] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning , pages 5338-5348. PMLR, 2020. 8 -
  21. [23][23] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large languageand-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems , 36:28541-28564, 2023. 8 -
  22. [24][24] Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive language-image pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 525-536. Springer, 2023. 3, 6, 8, 9, 12, 18 -
  23. [25][25] Yuan Liu, Ayush Jain, Clara Eng, David H Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, et al. A deep learning system for differential diagnosis of skin diseases. Nature medicine , 26(6):900-908, 2020. 2 -
  24. [26][26] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visuallanguage foundation model for computational pathology. Nature Medicine , 30(3):863-874, 2024. 8 -
  25. [27][27] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and TsuiWei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06129 , 2023. 8 - · arXiv
  26. [29][29] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. 6 -
  27. [30][30] Alec Radford, Jong Wook Kim, Chris Hallacy, and et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 2, 3, 5, 6, 8, 17, 18 -
  28. [31][31] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning , pages 28492-28518. PMLR, 2023. 4, 11 -
  29. [32][32] Yue Shen, Huanyu Li, Can Sun, Hongtao Ji, Daojun Zhang, Kun Hu, Yiqi Tang, Yu Chen, Zikun Wei, and Junwei Lv. Optimizing skin disease diagnosis: harnessing online community data with contrastive learning and clustering techniques. NPJ Digital Medicine , 7(1):28, 2024. 2 -
  30. [33][33] Xiaoxiao Sun, Jufeng Yang, Ming Sun, and Kai Wang. A benchmark for automatic visual classification of clinical skin disease images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14 , pages 206-222. Springer, 2016. 3, 8, 16 -
  31. [34][34] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception archi- 36. tecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016. 2 -
  32. [35][35] Hue Tran, Keng Chen, Adrian C Lim, James Jabbour, and Stephen Shumack. Assessing diagnostic skill in dermatology: a comparison between general practitioners and dermatologists. Australasian journal of dermatology , 46(4):230234, 2005. 2 -
  33. [36][36] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data , 5(1):1-9, 2018. 2, 3, 6, 17 -
  34. [37][37] Abbi Ward, Jimmy Li, Julie Wang, and et al. Creating an empirical dermatology dataset through crowdsourcing with web search advertisements. JAMA Network Open , 7(11): e2446615-e2446615, 2024. 3, 14 -
  35. [38][38] Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine. arXiv preprint arXiv:2408.02900 , 2024. 3 - · arXiv
  36. [39][39] Siyuan Yan, Zhen Yu, Xuelin Zhang, Dwarikanath Mahapatra, Shekhar S. Chandra, Monika Janda, Peter Soyer, and Zongyuan Ge. Towards trustable skin cancer diagnosis via rewriting model's decision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11568-11577, 2023. 2, 8 -
  37. [40][40] Siyuan Yan, Zhen Yu, Clare Primiero, Cristina Vico-Alonso, Zhonghua Wang, Litao Yang, Philipp Tschandl, Ming Hu, Gin Tan, Vincent Tang, et al. A general-purpose multimodal foundation model for dermatology. arXiv preprint arXiv:2410.15038 , 2024. 2, 6 - · arXiv
  38. [41][41] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19187-19197, 2023. 8 -
  39. [42][42] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022. 6, 8, 17, 18 - · arXiv
  40. [43][43] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11975-11986, 2023. 6, 8, 17, 18 -
  41. [44][44] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 , 2022. 4 - · arXiv
  42. [46][46] Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision-language foundation 49. model for diverse biomedical tasks. Nature Medicine , pages 1-13, 2024. 6 -
  43. [48][48] Juexiao Zhou, Liyuan Sun, Yan Xu, Wenbin Liu, Shawn Afvari, Zhongyi Han, Jiaoyan Song, Yongzhi Ji, Xiaonan He, and Xin Gao. Skincap: A multi-modal dermatology dataset annotated with rich medical captions. arXiv preprint arXiv:2405.18004 , 2024. 1, 2, 3, 6, 8 · arXiv

Cited by

Sorted by citations

Related

Authors
Yan, S., Hu, M., Jiang, Y., Li, X., Fei, H., Tschandl, P., Kittler, H., & Ge, Z.
Affiliations
  • Monash University (AU)
  • National University of Singapore (SG)
  • Medical University of Vienna (AT)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
12681-12690
DOI
10.48550/arxiv.2503.14911
arXiv
2503.14911
OpenAlex
W4414902259
Cited by
13
References
0
Citations by year
2026: 13
FWCI
9.79
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
Keywords
dermatology, skin disease classification, clinical ontologies, image-text pairs, CLIP, zero-shot classification, cross-modal retrieval, skin tone

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review