Seven-point checklist and skin lesion classification using multitask multimodal neural nets
Abstract
We propose a multi-task deep convolutional neural network, trained on multi-modal data (clinical and dermoscopic images, and patient meta-data), to classify the 7-point melanoma checklist criteria and perform skin lesion diagnosis. Our neural network is trained using several multi-task loss functions, where each loss considers different combinations of the input modalities, which allows our model to be robust to missing data at inference time. Our final model classifies the 7-point checklist and skin condition diagnosis, produces multi-modal feature vectors suitable for image retrieval, and localizes clinically discriminant regions. We benchmark our approach using 1011 lesion cases, and report comprehensive results over all 7-point criteria and diagnosis. We also make our dataset (images and metadata) publicly available online at http://derm.cs.sfu.ca.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Kawahara et al. used CNNs as feature extractors for imaging modalities but experimented with a unique multimodal multi-task loss function that considers multiple combinations of the input modalities.
“Kawahara et al.39 also used CNNs as feature extractors for imaging modalities but experimented with a unique multimodal multi-task loss function that considers multiple combinations of the input modalities.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [39]
Kawahara et al., Yala et al., and Yoo et al. reported an improvement in performance using fusion compared to image-only models.
“Kawahara et al., Yala et al. and Yoo et al. reported an improvement in performance using fusion compared to image-only models (Table 1).”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [39]
The complexity of the non-imaging data used in multimodal fusion studies was limited, with most studies using basic demographic information (age, gender), a limited range of categorical clinical history (e.g., hypertension, smoking status), or disease-specific features strongly associated with the disease of interest (e.g., APOE4 for Alzheimer's, PSA for prostate cancer).
“The complexity of the non-imaging data in multimodal fusion work was limited, particularly in the context of available feature-rich and time-series data in the EHR. Instead, most studies focused primarily on basic demographic information such as age and gender25,27,39, a limited range of categorical clinical history such as hypertension or smoking status32,34 or disease-specific clinical features”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [39]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · see citing works
A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings
Friedman, M. (1940) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731944 · Cited by 2,785 · cited 1× · see citing works
“, x-balanced), we apply a Friedman test [35] using AUROC scores for each category, where the AUROC scores are averaged within each category, and obtained a statistically significant difference between the two models (p = 0.”
Method(2011) · Cited by 2,752 · see citing works
Codella, N. C. F., Gutman, D., Celebi, M. E., Helba, B., Marchetti, M. A., Dusza, S. W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363547 · Cited by 2,620 · see citing works
Diagnostic accuracy of dermoscopy
Kittler, H., Pehamberger, H., Wolff, K., & Binder, M. (2002) · The Lancet Oncology · doi:10.1016/s1470-2045(02)00679-4 · Cited by 1,349 · see citing works
Dermoscopy of pigmented skin lesions: Results of a consensus meeting via the Internet
Argenziano, G., Soyer, H. P., Chimenti, S., Talamini, R., Corona, R., Sera, F., Binder, M., Cerroni, L., Rosa, G. D., Ferrara, G., et al. (2003) · Journal of the American Academy of Dermatology · doi:10.1067/mjd.2003.281 · Cited by 1,211 · see citing works
Self-citationAutomated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks
Yu, L., Chen, H., Dou, Q., Qin, J., & Heng, P.-A. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2642839 · Cited by 1,140 · see citing works
(2014) · arXiv (Cornell University) · Cited by 942 · see citing works
Epiluminescence Microscopy for the Diagnosis of Doubtful Melanocytic Skin Lesions
Argenziano, G., Fabbrocini, G., Carli, P., GIORGI, V. D., Sammarco, E., & Delfino, M. (1998) · Archives of Dermatology · doi:10.1001/archderm.134.12.1563 · Cited by 898 · see citing works
Self-citationNachbar, F., Stolz, W., Merkle, T., Cognetta, A. B., VOGT, T. M., Landthaler, M., Bilek, P., Braun‐Falco, O., & Plewig, G. (1994) · Journal of the American Academy of Dermatology · doi:10.1016/s0190-9622(94)70061-3 · Cited by 893 · see citing works
Nock, R., & Nielsen, F. (2004) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2004.110 · Cited by 822 · see citing works
Skin Cancer: Epidemiology, Disease Burden, Pathophysiology, Diagnosis, and Therapeutic Approaches
Apalla, Z., Nashan, D., Weller, R., & Castellsagué, X. (2017) · Dermatology and Therapy · doi:10.1007/s13555-016-0165-y · Cited by 466 · see citing works
Dermoscopy of pigmented skin lesions
Braun, R. P., Rabinovitz, H. S., Oliviero, M., Kopf, A. W., & Saurat, J. ‐. (2004) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2001.11.001 · Cited by 401 · see citing works
Gutman, D. A., Codella, N., Celebi, E., Helba, B., Marchetti, M. A., Mishra, N. K., & Halpern, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01397 · Cited by 365 · see citing works
Deep features to classify skin lesions
Kawahara, J., BenTaieb, A., & Hamarneh, G. (2016) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2016.7493528 · Cited by 356 · see citing works
Self-citationBallerini, L., Fisher, R. B., Aldridge, B., & Rees, J. (2012) · Lecture notes in computational vision and biomechanics · doi:10.1007/978-94-007-5389-1_4 · Cited by 289 · see citing works
Knowledge transfer for melanoma screening with deep learning
Menegola, A., Fornaciali, M., Pires, R., Bittencourt, F. V., Avila, S., & Junior, E. A. D. V. (2017) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2017.7950523 · Cited by 218 · see citing works
Shaban, A., & Yousefi, S. (2024) · Springer optimization and its applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 206 · see citing works
Seven-point checklist of dermoscopy revisited
Argenziano, G., Catricalà, C., Ardigò, M., Buccini, P., Simone, P. D., Eibenschutz, L., Ferrari, A., Mariani, G., Silipo, V., Sperduti, I., et al. (2010) · British Journal of Dermatology · doi:10.1111/j.1365-2133.2010.10194.x · Cited by 169 · see citing works
Self-citationGe, Z., Demyanov, S., Chakravorty, R., Bowling, A., & Garnavi, R. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_29 · Cited by 145 · see citing works
The Melanoma Epidemic: Res Ipsa Loquitur
Beddingfield, F. C. (2003) · The Oncologist · doi:10.1634/theoncologist.8-5-459 · Cited by 137 · see citing works
Carli, P., Quercioli, E., Sestini, S., Stante, M., Ricci, L., Brunasso, A. M. G., & GIORGI, V. D. (2003) · British Journal of Dermatology · doi:10.1046/j.1365-2133.2003.05023.x · Cited by 132 · see citing works
Detection and Analysis of Irregular Streaks in Dermoscopic Images of Skin Lesions
Sadeghi, M., Lee, T. K., McLean, D. I., Lui, H., & Atkins, M. S. (2013) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2013.2239307 · Cited by 127 · see citing works
Total body skin examination for skin cancer screening in patients with focused symptoms
Argenziano, G., Zalaudek, I., Hofmann‐Wellenhof, R., et al. (2011) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2010.12.039 · Cited by 77 · see citing works
Self-citationImplementation of the 7-point checklist for melanoma detection on smart handheld devices
Wadhawan, T., Situ, N., Rui, H., Lancaster, K., Yuan, X., & Zouridakis, G. (2011) · 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2011.6090866 · Cited by 72 · see citing works
Haenssle, H. A., Korpas, B., Hansen-Hagge, C., et al. (2010) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2009.08.049 · Cited by 59 · see citing works
Interactive Atlas of Dermoscopy
Lio, P., & Nghiem, P. (2004) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2003.07.029 · Cited by 45 · see citing works
Mirzaalian, H., Lee, T. K., & Hamarneh, G. (2012) · doi:10.1109/mmbia.2012.6164758 · Cited by 35 · see citing works
Self-citationAutomatic Detection of Blue-White Veil by Discrete Colour Matching in Dermoscopy Images
Madooei, A., Drew, M. S., Sadeghi, M., et al. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40760-4_57 · Cited by 33 · see citing works
Automatic Diagnosis of Melanoma Based on the 7-Point Checklist
Fabbrocini, G., Vita, V. D., Cacciapuoti, S., et al. (2013) · Series in bioengineering · doi:10.1007/978-3-642-39608-3_4 · Cited by 22 · see citing works
Hierarchical Classification of Ten Skin Lesion Classes
Leo, C. D., Bevilacqua, V., Ballerini, L., et al. (2015) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 9 · see citing works
Graph Geodesics to Find Progressively Similar Skin Lesion Images
Kawahara, J., Moriarty, K. P., & Hamarneh, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67675-3_4 · Cited by 4 · see citing works
Self-citationThe Little Black Book of Dermatology (The Little Black Book Series)
Bennett, M. L. (2001) · Archives of Dermatology · doi:10.1001/archderm.137.6.831 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934 · see citing works
“Kawahara et al.(39) also used CNNs as feature extractors for imaging modalities but experimented with a unique multimodal multi-task loss function that considers multiple combinations of the input modalities.”
MethodFederated learning for medical image analysis: A survey
Guan, H., Yap, P.-T., Bozoki, A., & Liu, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110424 · Cited by 518 · see citing works
Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I., & Lungren, M. P. (2020) · Scientific Reports · doi:10.1038/s41598-020-78888-w · Cited by 246 · see citing works
Han, S. S., Park, I., Eun Chang, S., Lim, W., Kim, M. S., Park, G. H., Chae, J. B., Huh, C. H., & Na, J.-I. (2020) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2020.01.019 · Cited by 248 · see citing works
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274 · see citing works
Review of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235 · see citing works
Guidelines and evaluation of clinical explainable AI in medical image analysis
Jin, W., Li, X., Fatehi, M., & Hamarneh, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102684 · Cited by 199 · see citing works
Self-citationYan, R., Qu, L., Wei, Q., Huang, S.-C., Shen, L., Rubin, D. L., Xing, L., & Zhou, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3233574 · Cited by 183 · see citing works
Skin disease diagnosis with deep learning: a review
Li, H., Pan, Y., Zhao, J., & Zhang, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.096 · Cited by 178 · see citing works
Karthik, R., Vaichole, T. S., Kulkarni, S. K., Yadav, O., & Khan, F. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103406 · Cited by 152 · see citing works
FusionM4Net: a multi-stage multi-modal learning algorithm for multi-label skin lesion classification
Tang, P., Yan, X., Nan, Y., Xiang, S., Krammer, S., & Lasser, T. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102307 · Cited by 131 · see citing works
Transparent medical image AI via an image–text foundation model grounded in medical literature
Kim, C., Gadgil, S. U., DeGrave, A. J., Omiye, J. A., Cai, Z. R., Daneshjou, R., & Lee, S.-I. (2024) · Nature Medicine · doi:10.1038/s41591-024-02887-x · Cited by 125 · see citing works
Schouten, D., Nicoletti, G., Dille, B., Chia, C., Vendittelli, P., Schuurmans, M., Litjens, G., & Khalili, N. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103621 · Cited by 108 · see citing works
Höhn, J., Krieghoff-Henning, E., Jutzi, T. B., von Kalle, C., Utikal, J. S., Meier, F., Gellrich, F. F., Hobelsberger, S., Hauschild, A., Schlager, J. G., et al. (2021) · European Journal of Cancer · doi:10.1016/j.ejca.2021.02.032 · Cited by 106 · see citing works
A Multimodal Vision Foundation Model for Clinical Dermatology
Yan, S., Yu, Z., Primiero, C., Vico-Alonso, C., Wang, Z., Yang, L., Tschandl, P., Hu, M., Ju, L., Tan, G., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03747-y · Cited by 117 · see citing works
Bibi, S., Khan, M. A., Shah, J. H., Damaševičius, R., Alasiry, A., Marzougui, M., Alhaisoni, M., & Masood, A. (2023) · Diagnostics · doi:10.3390/diagnostics13193063 · Cited by 88 · see citing works
“The capacity of dermatologists to discriminate between melanoma and non-melanoma images has been improved by the development of several traditional approaches, including the ABCD rule [17], 7-point checklist [18], Menzies procedure [19], and CASH [20].”
MethodSimon, B. D., Ozyoruk, K. B., Gelikman, D. G., Harmon, S. A., & Türkbey, B. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242631 · Cited by 73 · see citing works
Nunnari, F., Bhuvaneshwara, C., Ezema, A. O., & Sonntag, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_11 · Cited by 24 · see citing works
Multimodal dual-stage feature refinement for robust skin lesion classification
Khurshid, M., Singh, R., & Vatsa, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-14839-7 · Cited by 13 · see citing works
- Authors
- Kawahara, J., Daneshvar, S., Argenziano, G., & Hamarneh, G.
- Affiliations
- Simon Fraser University (CA)
- University of Campania "Luigi Vanvitelli" (IT)
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2019-03
- Volume
- 23
- Issue
- 2
- Pages
- 538-546
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W2797527544
- Cited by
- 584
- References
- 38
- Citations by year
- 2026: 99 · 2025: 112 · 2024: 93 · 2023: 84 · 2022: 84 · 2021: 63 · 2020: 30 · 2019: 15 · 2018: 4
- FWCI
- 10.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Journal of Biomedical and Health Informatics (publishedVersion)
- PubMed (publishedVersion)
- Archivio Istituzionale della Ricerca - Università degli Studi della Campania "Luigi Vanvitelli" (submittedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- melanoma, skin lesion classification, dermoscopic images, multi-task learning, multimodal neural networks, missing data robustness, image retrieval, lesion localization, clinical metadata
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
