The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 30 of the 49 sources its claims cite were checked for support.
Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata
Open accessAbstract
Introduction: Accurate and early classification of skin lesions is essential for the early detection of disease and improved clinical outcomes. However, automated multiclass classification is still hindered by severe class imbalance, high inter-class visual similarity, and the comparatively under-explored use of complementary clinical metadata relative to image-only pipelines, despite growing interest in such metadata fusion. In line with the growing demand for artificial intelligence (AI) tools that integrate heterogeneous data in clinical scenarios, this study presents a multimodal machine learning-based computer-aided diagnosis (ML-CAD) framework that fuses dermoscopic images with patient metadata. Methods: The framework follows a five-phase pipeline. A structured multimodal balancing strategy combines class-wise Synthetic Minority Oversampling Technique for Nominal and Continuous features (SMOTENC) for metadata with controlled image augmentation. This is followed by cross-modal alignment to preserve clinical consistency. A fine-tuned Vision Transformer (ViT-B/16) at reduced input resolution performs feature extraction and late fusion with encoded metadata, yielding an 800-dimensional multimodal representation. Individual and stacked ensemble classifiers are subsequently trained on the fused features, with Bayesian optimization based on Gaussian process surrogates used for hyperparameter tuning. Gradient-weighted Class Activation Mapping (Grad-CAM) provides a visual interpretation across all seven lesion categories. Results: Experiments on the HAM10000 dataset demonstrate that the proposed SVM+KNN stacking ensemble achieved an accuracy of 98.55% and a ROC-AUC of 99.88%, achieving competitive accuracy among recently reported methods under broadly similar balanced settings. Discussion: The findings underscore the necessity of an AI framework that integrates imaging and clinical data to provide interpretable clinical decision support. This framework represents a step toward providing clinicians with multimodal, interpretable decision support. We emphasize, however, that external and prospective clinical validation remain necessary before translation into routine dermatological practice.
Overview
CoChat Summary
## TL;DR
This paper presents a multimodal skin-lesion classification framework that combines dermoscopic images with patient metadata on HAM10000. The best model, an SVM+KNN stacking ensemble, reached 98.55% accuracy and 99.88% ROC-AUC, with Grad-CAM used for visual interpretability.
## Key findings
- On HAM10000 (10,015 dermoscopic images, 7 lesion classes), the proposed SVM+KNN stacking ensemble achieved 98.55% accuracy and 99.88% ROC-AUC.
- The fine-tuned ViT feature extractor alone achieved 95.02% accuracy and 99.66% ROC-AUC.
- In the modality ablation, metadata-only classification reached 40.59% accuracy, while image-only ViT reached 90.27%; adding metadata improved accuracy to 95.02%, with the largest single metadata gain coming from lesion localization (93.54%).
- In input-resolution ablation, accuracy increased from 95.02% at 48 × 48 to 95.57% at 128 × 128 and 95.78% at 224 × 224, but training time rose from 45 minutes to 2.5 hours and 6.5 hours.
## Methods
- Retrospective experiment on the HAM10000 dataset: 10,015 dermoscopic images across 7 lesion classes, with metadata for age, sex, and lesion localization.
- Multimodal pipeline: class-wise SMOTENC for metadata, controlled image augmentation, lesion-ID-level split before augmentation, ViT-B/16 feature extraction with late fusion into an 800-dimensional multimodal representation.
- Classification and tuning: individual classifiers (SVM, DT, KNN, LR), bagging/boosting/stacking ensembles, and Bayesian optimization with Gaussian process surrogates; interpretability via Grad-CAM.
## Limitations
- The paper explicitly states that external and prospective validation are still necessary before any clinical translation; all results come from a single retrospective dataset (HAM10000).
- It also states that clinical utility/generalizability is not established because the model was evaluated only on an internal held-out test set, not independent cohorts.
- Evident from the text: the available metadata are limited to age, sex, and lesion localization, and the authors note that richer clinical context is unavailable; this may constrain performance on visually similar classes.
- Evident from the text: the 48 × 48 input resolution may discard fine-grained lesion details, even though it improves computational efficiency.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
15 Valid · 4 Not valid · 5 Not checked · 93 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The dataset includes clinical metadata: patient age, sex, and lesion localization.”
§ Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata > Materials and methods > Dataset description
What the sources say
- Partly supportedcites [30] Tschandl, 2018, The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions, Sci Data sourceExperimental — accuracy not yet measured
The Data Records section (page not provided) says HAM10000 images and metadata are accessible through the ISIC archive. Our copy does not specify that the metadata include patient age, sex, or lesion localization; searches for those fields found no confirming passage. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The HAM10000 dataset (30) contains 10,015 dermoscopic images of seven pigmented skin lesion types: actinic keratoses (AKIEC), basal cell carcinoma (BCC), benign keratosis-like lesions (BKL), dermatofibroma (DF), melanoma (MEL), melanocytic nevus (NV), and vascular lesions (VASC).”
§ Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata > Materials and methods > Dataset description
What the sources say
- Supportedcites [30] Tschandl, 2018, The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions, Sci Data source
The Methods section states that the HAM10000 training set contains 10,015 dermatoscopic images and describes seven diagnostic classes. The Usage Notes identify the listed classes, including AKIEC, BCC, BKL, DF, MEL, NV, and VASC. No page numbers are available in our copy. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“These studies suggest that patient metadata provides complementary information that can support skin lesion classification when combined with image features.”
§ Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata > Related work
What the sources say
- Not checked — no full text of the sourcecites [26] Tang, 2022, FusionM4Net: a multi-stage multi-modal learning algorithm for multi-label skin lesion classification, Med Image Anal source
- Supportedcites [27] Khurshid, 2025, Multimodal dual-stage feature refinement for robust skin lesion classification, Sci Rep source
The Introduction describes clinical and demographic metadata as complementary to image data and says jointly using both can help address skin-lesion classification challenges. The Methods describe combining visual and metadata features for classification, while the Discussion reports enhanced diagnostic accuracy when rich metadata is available. (LLM, conf=0.94)
- Supportedcites [28] Saeed, 2025, Multimodal deep learning ensemble framework for skin cancer detection, Sci Rep source
The Proposed model section says that structured metadata alongside image-derived features enriches contextual understanding and improves skin-lesion classification. The Related work > Metadata usage section also reports that combining patient metadata with images can improve classification accuracy in some scenarios. (LLM, conf=0.94)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Most studies focus on binary classification rather than clinically relevant multiclass categorization (9, 10).”
§ Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata > Introduction
What the sources say
- Partly supportedcites [9] Bechelli, 2022, Machine learning and deep learning algorithms for skin cancer classification from dermoscopic images, Bioengineering sourceExperimental — accuracy not yet measured
The Conclusions and Results (pp. not available) show that this paper studies binary malignant-versus-benign classification. They do not establish that most studies use binary classification or compare its clinical relevance with multiclass categorization. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [10] Khan, 2025, An advanced deep learning framework for skin cancer classification, Rev Socionetw Strat source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In the classification stage, most prior studies relied primarily on the fully connected layer for final classification (7, 14, 16–18, 20).”
§ Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata > Discussion > Classifier design and optimization
What the sources say
- Partly supportedcites [7] Yang, 2023, A novel vision transformer model for skin cancer classification, Neural Proc Lett sourceExperimental — accuracy not yet measured
The Classification block section compares the paper’s dense-layer classification block with the original ViT’s MLP head, supporting the narrower point that an earlier model used such a head. The paper does not establish that most prior studies relied primarily on a fully connected layer for final classification; no page number is available in our copy. (LLM, conf=0.72)
- Could not verifycites [14] Shetty, 2022, Skin lesion classification of dermoscopic images using machine learning and convolutional neural network, Sci Rep sourceExperimental — accuracy not yet measured
The Proposed methodology section lists dense layers for this paper’s own classifier, but that does not establish what most prior studies used. The Literature survey describes several classification approaches without saying that most relied primarily on a fully connected layer; searches for that claim and a dense-layer paraphrase found no supporting passage. The indexed copy provides no page numbers. (LLM, conf=0.72)
- Not checked — no full text of the sourcecites [16] Govindu, 2026, Cutting-edge CNN-based skin cancer detection with batch normalization and advanced imbalance learning for superior medical image classification, Biomed Signal Process Control source
- Partly supportedcites [17] Alwakid, 2022, Melanoma detection using deep learning-based classifications, Healthcare sourceExperimental — accuracy not yet measured
Section 3.2 (page unavailable) describes a fully connected layer as part of the basic CNN structure and says its proposed model outputs classification results through that layer. This supports the paper as an example, but it does not establish that most prior studies relied primarily on fully connected layers for final classification. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [18] Abbas, 2025, A robust skin cancer classification using deep learning, Sustain Mach Intell J source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
50 more claims · 5 with their cited sources checked
Create a free account to see all 117 claims and every source verdict.
- Probably not a substantial claim. 27
- It may state two separate claims as one. 45
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Nearest neighbor pattern classification
Cover, T., & Hart, P. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1053964 · Cited by 16,936
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300 · cited 1×
“Three ensemble strategies were employed (39): (1) Bagging : Random Forest (RF) (40) and Extremely Randomized Trees (ET) (41); (2) Boosting : Extreme Gradient Boosting (XGB) (42) and Light Gradient Boosting Machine (LGBM) (43); (3) Stacking : All possible combinations of the four individual…”
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
Tschandl, P., Rosendahl, C., & Kittler, H. (2018) · Scientific Data · doi:10.1038/sdata.2018.161 · Cited by 3,628
Research works printed but not linked: 2
- Breiman, 1984, Classification and Regression Trees
- Ke, 2017, LightGBM: a highly efficient gradient boosting decision tree, Advances in neural information processing systems, 3146
Metadata
- Authors
- Sonuç, E., Saihood, Q., Yargı Baydilli, Y., & Özcan, E.
- Affiliations
- University of Nottingham (GB)
- Karabük University (TR)
- Department of Computer Engineering, Hakkari University
- Corresponding author
- Sonuç, Emrullah
- Venue
- Frontiers in Medicine
- Publisher
- Frontiers Media SA
- Published
- 2026-07-29
- Volume
- 13
- Pages
- 1871689-1871689
- ISSN
- 2296-858X (ISSN-L)
- PMC
- PMC13461512
- OpenAlex
- W7171674036
- Semantic Scholar
- 372c09977d095d0355dc2df723a148657e4473a7
- Cited by
- 0
- References
- 44
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- skin lesion classification, dermoscopic images, patient metadata, multimodal fusion, computer-aided diagnosis, class imbalance, SMOTE-NC, image augmentation, cross-modal alignment, vision transformer, decision fusion, ensemble classifiers, Bayesian optimization, Grad-CAM, interpretability, HAM10000 dataset, dermatology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata. https://app.cochat.ai/papers/10.3389/fmed.2026.1871689
- The paper
- Sonuç, E., Saihood, Q., Yargı Baydilli, Y., & Özcan, E. (2026). Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata. Frontiers in Medicine. https://doi.org/10.3389/fmed.2026.1871689
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.