The DOI matches its Crossref record; 52 of the 55 references it cites were found in a registry or an index.
FusionM4Net: A multi-stage multi-modal learning algorithm for multi-label skin lesion classification
Abstract
Skin disease is one of the most common diseases in the world. Deep learning-based methods have achieved excellent skin lesion recognition performance, most of which are based on only dermoscopy images. In recent works that use multi-modality data (patient's meta-data, clinical images, and dermoscopy images), the methods adopt a one-stage fusion approach and only optimize the information fusion at the feature level. These methods do not use information fusion at the decision level and thus cannot fully use the data of all modalities. This work proposes a novel two-stage multi-modal learning algorithm (FusionM4Net) for multi-label skin diseases classification. At the first stage, we construct a FusionNet, which exploits and integrates the representation of clinical and dermoscopy images at the feature level, and then uses a Fusion Scheme 1 to conduct the information fusion at the decision level. At the second stage, to further incorporate the patient's meta-data, we propose a Fusion Scheme 2, which integrates the multi-label predictive information from the first stage and patient's meta-data information to train an SVM cluster. The final diagnosis is formed by the fusion of the predictions from the first and second stages. Our algorithm was evaluated on the seven-point checklist dataset, a well-established multi-modality multi-label skin disease dataset. Without using the patient's meta-data, the proposed FusionM4Net's first stage (FusionM4Net-FS) achieved an average accuracy of 75.7% for multi-classification tasks and 74.9% for diagnostic tasks, which is more accurate than other state-of-the-art methods. By further fusing the patient's meta-data at FusionM4Net's second stage (FusionM4Net-SS), the entire FusionM4Net finally boosts the average accuracy to 77.0% and the diagnostic accuracy to 78.5%, which indicates its robust and excellent classification performance on the label-imbalanced dataset. The corresponding code is available at: https://github.com/pixixiaonaogou/MLSDR.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These studies suggest that patient metadata provides complementary information that can support skin lesion classification when combined with image features.”
From Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata (2026) · cites this paper as [26]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989
Harris, C., Millman, K., van der Walt, S., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N., et al. (2020) · RePEc: Research Papers in Economics · Cited by 17,844
Final Version of 2009 AJCC Melanoma Staging and Classification
Balch, C. M., Gershenwald, J. E., Soong, S.-J., Thompson, J. F., Atkins, M. B., Byrd, D. R., Buzaid, A. C., Cochran, A. J., Coit, D. G., Ding, S., et al. (2009) · Journal of Clinical Oncology · doi:10.1200/jco.2009.23.4799 · Cited by 4,653
Tschandl, P., Rosendahl, C., & Kittler, H. (2018) · Scientific Data · doi:10.1038/sdata.2018.161 · Cited by 3,628
(2011) · Cited by 2,752
Albumentations: fast and flexible image augmentations
Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., & Kalinin, A. A. (2020) · Information · doi:10.3390/info11020125 · Cited by 2,714
Codella, N. C. F., Gutman, D., Celebi, M. E., Helba, B., Marchetti, M. A., Dusza, S. W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363547 · Cited by 2,620
Multi-Label Image Recognition With Graph Convolutional Networks
Chen, Z.-M., Wei, X.-S., Wang, P., & Guo, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00532 · Cited by 1,364
Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks
Yu, L., Chen, H., Dou, Q., Qin, J., & Heng, P.-A. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2642839 · Cited by 1,140
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Epiluminescence Microscopy for the Diagnosis of Doubtful Melanocytic Skin Lesions
Argenziano, G., Fabbrocini, G., Carli, P., GIORGI, V. D., Sammarco, E., & Delfino, M. (1998) · Archives of Dermatology · doi:10.1001/archderm.134.12.1563 · Cited by 898
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
BCN20000: Dermoscopic Lesions in the Wild
Combalia, M., Codella, N., Rotemberg, V. M., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A. C., Puig, S., et al. (2019) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · doi:10.48550/arxiv.1908.02288 · Cited by 607
Attention Residual Learning for Skin Lesion Classification
Zhang, J., Xie, Y., Xia, Y., & Shen, C. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2893944 · Cited by 585
Seven-point checklist and skin lesion classification using multitask multimodal neural nets
Kawahara, J., Daneshvar, S., Argenziano, G., & Hamarneh, G. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2824327 · Cited by 584
Skin lesion classification with ensembles of deep convolutional neural networks
Harangi, B. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.08.006 · Cited by 460
Skin cancer and new treatment perspectives: A review
Simões, M. F., Sousa, J., & Pais, A. A. C. C. (2014) · Cancer Letters · doi:10.1016/j.canlet.2014.11.001 · Cited by 421
Dermoscopy of pigmented skin lesions
Braun, R. P., Rabinovitz, H. S., Oliviero, M., Kopf, A. W., & Saurat, J. ‐. (2004) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2001.11.001 · Cited by 401
Deep features to classify skin lesions
Kawahara, J., BenTaieb, A., & Hamarneh, G. (2016) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2016.7493528 · Cited by 356
Multimodal skin lesion classification using deep learning
Yap, J., Yolland, W., & Tschandl, P. (2018) · Experimental Dermatology · doi:10.1111/exd.13777 · Cited by 326
Pacheco, A. G. C., & Krohling, R. A. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3062002 · Cited by 274
Melanoma Recognition in Dermoscopy Images via Aggregated Deep Convolutional Features
Yu, Z., Jiang, X., Zhou, F., Qin, J., Ni, D., Chen, S., Lei, B., & Wang, T. (2018) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2018.2866166 · Cited by 253
Integrative data analysis of multi-platform cancer data with a multimodal deep learning approach
Liang, M., Li, Z., Chen, T., & Zeng, J. (2015) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2014.2377729 · Cited by 231
Knowledge transfer for melanoma screening with deep learning
Menegola, A., Fornaciali, M., Pires, R., Bittencourt, F. V., Avila, S., & Junior, E. A. D. V. (2017) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2017.7950523 · Cited by 223
Unpaired Multi-Modal Segmentation via Knowledge Distillation
Dou, Q., Liu, Q., Heng, P., & Glocker, B. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2963882 · Cited by 216
Improving Dermoscopy Image Classification Using Color Constancy
Barata, A. F., Celebi, M. E., & Marques, J. S. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2014.2336473 · Cited by 212
Efficient skin lesion segmentation using separable-Unet with stochastic weight averaging
Tang, P., Liang, Q., Yan, X., Xiang, S., Sun, W., Zhang, D., & Coppola, G. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.07.005 · Cited by 190
Self-citationSharable and Individual Multi-View Metric Learning
Hu, J., Lu, J., & Tan, Y. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2749576 · Cited by 170
Seven-point checklist of dermoscopy revisited
Argenziano, G., Catricalà, C., Ardigò, M., Buccini, P., Simone, P. D., Eibenschutz, L., Ferrari, A., Mariani, G., Silipo, V., Sperduti, I., et al. (2010) · British Journal of Dermatology · doi:10.1111/j.1365-2133.2010.10194.x · Cited by 169
Ge, Z., Demyanov, S., Chakravorty, R., Bowling, A., & Garnavi, R. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_29 · Cited by 145
Tang, P., Liang, Q., Yan, X., Xiang, S., & Zhang, D. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2977013 · Cited by 144
Self-citationImage Classification of Melanoma, Nevus and Seborrheic Keratosis by Deep Neural Network EnsemblePreprint
Matsunaga, K., Hamada, A., Minagawa, A., & Koga, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03108 · Cited by 133
Qiu, S., Chang, G. H., Panagia, M., Gopal, D. M., Au, R., & Kolachalama, V. B. (2018) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1016/j.dadm.2018.08.013 · Cited by 110
Bi, L., Feng, D. D., FULHAM, M. J., & Kim, J. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107502 · Cited by 109
Fusing Metadata and Dermoscopy Images for Skin Disease Diagnosis
Li, W., Zhuang, J., Wang, R., Zhang, J., & Zheng, W. (2020) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi45749.2020.9098645 · Cited by 85
Model-Based Classification Methods of Global Patterns in Dermoscopic Images
Sáez, A., Serrano, C., & Acha, B. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2305769 · Cited by 74
Zhang, H., Zuelsdorf, C., Penneys, D. S., Fan, S., Kofsky, J., & Song, B. (2018) · Scientific Data · doi:10.1038/sdata.2018.290 · Cited by 68
European Handbook of Dermatological Treatments
D’Erme, A. M., Dessinioti, C., Lotti, T., & Katsambas, A. (2015) · Springer eBooks · doi:10.1007/978-3-662-45139-7 · Cited by 63
Liao, X., Qian, Y., Chen, Y., Xiong, X., Wang, Q., & Heng, P. (2020) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2020.101785 · Cited by 49
Sun, B., Li, L., Wu, X., Zuo, T., Chen, Y., Zhou, G., He, J. Y. D., & Zhu, X. (2015) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0203-6 · Cited by 46
Hierarchical Multimodal Metric Learning for Multimodal Classification
Zhang, H., Patel, V. M., & Chellappa, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.312 · Cited by 41
Analysis of the contour structural irregularity of skin lesions using wavelet decomposition
Ma, L., & Staunton, R. C. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2012.07.001 · Cited by 35
Dermoscopy of Pigmented Skin Lesions
Дума, С. (2015) · Springer eBooks · doi:10.1007/978-3-662-45139-7_116 · Cited by 11
Research works printed but not linked: 1
- Qiu, 2018, Fusion of deep learning models of MRI scans, Assess. Dis. Monit., 10, 737
Cited by
Sorted by citations
Current advances and future perspectives of image fusion: A comprehensive review
Karim, S., Tong, G., Li, J., Qadir, A., Farooq, U., & Yu, Y. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.09.019 · Cited by 290
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274
A survey, review, and future trends of skin lesion segmentation and classification
Hasan, M. K., Ahamad, M. A., Yap, C. H., & Yang, G. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106624 · Cited by 161
Zhang, S., Zhang, J., Tian, B., Lukasiewicz, T., & Xu, Z. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102656 · Cited by 145
Wang, Y., Wang, Y., Cai, J., Lee, T. K., Miao, C., & Wang, Z. J. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102693 · Cited by 144
Metadata
- Authors
- Tang, P., Yan, X., Nan, Y., Xiang, S., Krammer, S., & Lasser, T.
- Affiliations
- Technical University of Munich (DE)
- Wuhan University (CN)
- LMU Klinikum (DE)
- Corresponding author
- Tang, Peng
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2022-02
- Volume
- 76
- Pages
- 102307
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W3215150387
- Semantic Scholar
- 9bab4cf354505a691237550507918821d7656002
- Cited by
- 131
- References
- 55
- Citations by year
- 2026: 30 · 2025: 39 · 2024: 31 · 2023: 20 · 2022: 11
- FWCI
- 5.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Medical Image Analysis (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Bundesministerium für Gesundheit (2520DAT920)
- China Scholarship Council
- Topics
- Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- skin lesion classification, multi-label classification, multimodal learning, dermoscopic images, decision fusion, feature-level fusion, two-stage fusion, label imbalance, dermatological diagnosis
- MeSH terms
- Algorithms, Humans, Skin Diseases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
