Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 0Cutaneous Melanoma Detection and ManagementAI in cancer detectionArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 30 of the 49 sources its claims cite were checked for support.

Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata

Open access
Sonuç, Emrullah, Saihood, Qusay, Yargı Baydilli, Yusuf, Özcan, Ender·2026·Frontiers in Medicine

Abstract

Introduction: Accurate and early classification of skin lesions is essential for the early detection of disease and improved clinical outcomes. However, automated multiclass classification is still hindered by severe class imbalance, high inter-class visual similarity, and the comparatively under-explored use of complementary clinical metadata relative to image-only pipelines, despite growing interest in such metadata fusion. In line with the growing demand for artificial intelligence (AI) tools that integrate heterogeneous data in clinical scenarios, this study presents a multimodal machine learning-based computer-aided diagnosis (ML-CAD) framework that fuses dermoscopic images with patient metadata. Methods: The framework follows a five-phase pipeline. A structured multimodal balancing strategy combines class-wise Synthetic Minority Oversampling Technique for Nominal and Continuous features (SMOTENC) for metadata with controlled image augmentation. This is followed by cross-modal alignment to preserve clinical consistency. A fine-tuned Vision Transformer (ViT-B/16) at reduced input resolution performs feature extraction and late fusion with encoded metadata, yielding an 800-dimensional multimodal representation. Individual and stacked ensemble classifiers are subsequently trained on the fused features, with Bayesian optimization based on Gaussian process surrogates used for hyperparameter tuning. Gradient-weighted Class Activation Mapping (Grad-CAM) provides a visual interpretation across all seven lesion categories. Results: Experiments on the HAM10000 dataset demonstrate that the proposed SVM+KNN stacking ensemble achieved an accuracy of 98.55% and a ROC-AUC of 99.88%, achieving competitive accuracy among recently reported methods under broadly similar balanced settings. Discussion: The findings underscore the necessity of an AI framework that integrates imaging and clinical data to provide interpretable clinical decision support. This framework represents a step toward providing clinicians with multimodal, interpretable decision support. We emphasize, however, that external and prospective clinical validation remain necessary before translation into routine dermatological practice.

Overview

CoChat Summary

## TL;DR

This paper presents a multimodal skin-lesion classification framework that combines dermoscopic images with patient metadata on HAM10000. The best model, an SVM+KNN stacking ensemble, reached 98.55% accuracy and 99.88% ROC-AUC, with Grad-CAM used for visual interpretability.

## Key findings

  • On HAM10000 (10,015 dermoscopic images, 7 lesion classes), the proposed SVM+KNN stacking ensemble achieved 98.55% accuracy and 99.88% ROC-AUC.
  • The fine-tuned ViT feature extractor alone achieved 95.02% accuracy and 99.66% ROC-AUC.
  • In the modality ablation, metadata-only classification reached 40.59% accuracy, while image-only ViT reached 90.27%; adding metadata improved accuracy to 95.02%, with the largest single metadata gain coming from lesion localization (93.54%).
  • In input-resolution ablation, accuracy increased from 95.02% at 48 × 48 to 95.57% at 128 × 128 and 95.78% at 224 × 224, but training time rose from 45 minutes to 2.5 hours and 6.5 hours.

## Methods

  • Retrospective experiment on the HAM10000 dataset: 10,015 dermoscopic images across 7 lesion classes, with metadata for age, sex, and lesion localization.
  • Multimodal pipeline: class-wise SMOTENC for metadata, controlled image augmentation, lesion-ID-level split before augmentation, ViT-B/16 feature extraction with late fusion into an 800-dimensional multimodal representation.
  • Classification and tuning: individual classifiers (SVM, DT, KNN, LR), bagging/boosting/stacking ensembles, and Bayesian optimization with Gaussian process surrogates; interpretability via Grad-CAM.

## Limitations

  • The paper explicitly states that external and prospective validation are still necessary before any clinical translation; all results come from a single retrospective dataset (HAM10000).
  • It also states that clinical utility/generalizability is not established because the model was evaluated only on an internal held-out test set, not independent cohorts.
  • Evident from the text: the available metadata are limited to age, sex, and lesion localization, and the authors note that richer clinical context is unavailable; this may constrain performance on visually similar classes.
  • Evident from the text: the 48 × 48 input resolution may discard fine-grained lesion details, even though it improves computational efficiency.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fmed.2026.1871689

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Schematic of the proposed methodology: data preprocessing with multimodal balancing, ViT-based feature extraction with late fusion, Bayesian-optimized ML classifiers, and visual interpretation.
Figure 1 Schematic of the proposed methodology: data preprocessing with multimodal balancing, ViT-based feature extraction with late fusion, Bayesian-optimized ML classifiers, and visual interpretation.CC BY · PubMed Central
Figure 2 Representative sample images from each skin lesion class in the HAM10000 dataset.
Figure 2 Representative sample images from each skin lesion class in the HAM10000 dataset.CC BY · PubMed Central
Figure 3 Distribution of patient age, sex, and lesion localization in the HAM10000 metadata.
Figure 3 Distribution of patient age, sex, and lesion localization in the HAM10000 metadata.CC BY · PubMed Central
Figure 4 Examples of original and augmented skin lesion images.
Figure 4 Examples of original and augmented skin lesion images.CC BY · PubMed Central
Figure 5 Normalized confusion matrix of the best-performing stacked SVM+KNN model for seven-class skin lesion classification.
Figure 5 Normalized confusion matrix of the best-performing stacked SVM+KNN model for seven-class skin lesion classification.CC BY · PubMed Central
Figure 6 Representative Grad-CAM visualizations for MEL (malignant), AKIEC (precancerous), and NV (benign) lesions. Correctly classified and misclassified examples are shown with the original image, Grad-CAM heatmap, and overlay, highlighting the regions that contributed most strongly to the model's predictions.
Figure 6 Representative Grad-CAM visualizations for MEL (malignant), AKIEC (precancerous), and NV (benign) lesions. Correctly classified and misclassified examples are shown with the original image, Grad-CAM heatmap, and overlay, highlighting the regions that contributed most strongly to the model's predictions.CC BY · PubMed Central

Verification

Claims the paper makes

15 Valid · 4 Not valid · 5 Not checked · 93 Own results and interpretations

50 more claims · 5 with their cited sources checked

Create a free account to see all 117 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 27
  • It may state two separate claims as one. 45

Create a free account to see these claims.

Citations

Cites

Show all 44 references

Research works printed but not linked: 2

  1. Breiman, 1984, Classification and Regression Trees
  2. Ke, 2017, LightGBM: a highly efficient gradient boosting decision tree, Advances in neural information processing systems, 3146

Metadata

Authors
Sonuç, E., Saihood, Q., Yargı Baydilli, Y., & Özcan, E.
Affiliations
  • University of Nottingham (GB)
  • Karabük University (TR)
  • Department of Computer Engineering, Hakkari University
Corresponding author
Sonuç, Emrullah
Venue
Frontiers in Medicine
Published
2026-07-29
Volume
13
Pages
1871689-1871689
ISSN
2296-858X (ISSN-L)
OpenAlex
W7171674036
Cited by
0
References
44
FWCI
0.00
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
skin lesion classification, dermoscopic images, patient metadata, multimodal fusion, computer-aided diagnosis, class imbalance, SMOTE-NC, image augmentation, cross-modal alignment, vision transformer, decision fusion, ensemble classifiers, Bayesian optimization, Grad-CAM, interpretability, HAM10000 dataset, dermatology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata. https://app.cochat.ai/papers/10.3389/fmed.2026.1871689
The paper
Sonuç, E., Saihood, Q., Yargı Baydilli, Y., & Özcan, E. (2026). Interpretable multimodal fusion for skin lesion classification using dermoscopic images and patient metadata. Frontiers in Medicine. https://doi.org/10.3389/fmed.2026.1871689

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.