Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index; 5 of the 27 sources its claims cite were checked for support.

Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets

Open access
Marini, Niccolo, Liang, Zhaohui, Rajaraman, Sivaramakrishnan, Xue, Zhiyun, Antani, Sameer · 2026 · npj Digital Medicine

Abstract

Multimodal (MM) algorithms and foundation models have shown strong potential for automated skin lesion analysis in dermatology, however, their translation to clinical practice is hindered by the limited availability of large, high-quality image-text datasets. Most public dermatology datasets are small, unimodal, and paired with heterogeneous labels and metadata, restricting effective model development. Large Language Models (LLMs) provide an opportunity to synthesize clinical notes from existing unimodal datasets, but their application is limited by hallucinations that can hinder MM training. This paper investigates and evaluates strategies to generate reliable clinical notes using existing LLMs and metadata paired with real images, aiming to mitigate hallucinations and improve performance on downstream tasks. We train a MM architecture on real dermatology images paired with synthesized notes and evaluate it across 15 datasets (6 internal, 9 external) on cross-modal retrieval and zero-shot learning tasks. Results demonstrate improved robustness and generalization compared to state-of-the-art medical foundation models, under specific synthesis conditions.

Overview

CoChat Summary

TL;DR

Using metadata-conditioned LLM prompts to synthesize clinical notes from unimodal dermatology images, the authors trained a multimodal architecture on ~16,000 lesions and evaluated it on 15 datasets. The metadata-guided LLM strategy (L) improved cross-modal retrieval and zero-shot learning over template-only metadata (M) and unguided LLM prompting (OL), and the small multimodal model outperformed larger dermatology foundation models on most datasets.

Key findings

  • M (metadata-filled template) beat all OL (LLM without lesion metadata) variants on cross-modal retrieval on every dataset, with absolute mAP gaps of 0.10–0.40; e.g., BCN20000 image-to-text: M 0.769 ± 0.048 vs OL GPT 0.456 ± 0.018, MedGemma 0.340 ± 0.016, SkinGPT4 0.380 ± 0.029, DermLIP 0.596 ± 0.016. OL degradation was more evident on external datasets.
  • L (LLM prompted with metadata) generally matched or outperformed M, especially in text-to-image retrieval and on external datasets; in zero-shot learning, all L models significantly outperformed M across nearly all datasets in the 8-class setup, with amplified gains in the 15-class setup. Example BCN20000 8-class: M 0.554 ± 0.037 vs L GPT 0.738 ± 0.014, MedGemma 0.718 ± 0.050, SkinGPT4 0.724 ± 0.045, DermLIP 0.691 ± 0.053; 15-class BCN20000: M 0.269 ± 0.023 vs DermLIP 0.590 ± 0.009.
  • The MM architecture trained with L outperformed BioMedCLIP, MedImageInsights, MONET, and Derm1M on cross-modal retrieval and zero-shot learning, with mAP improvements of 0.10–0.30 and zero-shot F1 gaps of 0.20–0.40 on dermoscopic datasets. Example BCN20000 image-to-text: BioMedCLIP 0.320, MedImageInsights 0.682, MONET 0.333, Derm1M 0.403 vs ours DermLIP 0.791 ± 0.012.
  • LLM note content: L with GPT/MedGemma often included the correct lesion type, while OL dropped sharply; e.g., BCN20000 correct lesion type: GPT L 1.000 vs OL 0.284; SkinGPT4 L 0.793 vs OL 0.001; DermLIP L 0.619 vs OL 0.201. On PH2, non-medical features such as symmetry and color were described more accurately than dermatology-specific features.

Methods

  • Design: a multimodal architecture with a PanDerm image encoder (ViT, 85M parameters, 224×224×3, 24 transformer blocks, fully finetuned) and a frozen PubMedBERT text encoder (512 tokens, 12 transformer blocks, ~110M parameters), projected to 128 dimensions and passed through shared fully connected layers with an 8-class classifier; trained with L1, cosine similarity, and NT-Xent losses, alternating modalities.
  • Data/sample: ~16,000 skin lesions from six public datasets for training/validation; 15 datasets total (6 internal, 9 external) for evaluation; eight diagnostic classes; dermoscopic and clinical images; PH2 (200 images) for lesion-feature validation. Clinical notes were synthesized via M (metadata template), L (LLM prompted with metadata), and OL (LLM without metadata) using GPT-4o-mini, MedGemma, SkinGPT4, and DermLIP.
  • Analysis: cross-modal retrieval (mAP), zero-shot learning (weighted F1 for 8- and 15-class setups), and clinical note generation (BERTScore, weighted F1); ten runs reported as mean ± SD; Wilcoxon signed-rank test with p < 0.05; foundation models evaluated at fixed checkpoints without retraining.

Limitations

  • Stated: the method requires a supervised classifier, limiting task-agnostic use and generalization to completely different diseases; there is no clinical validation against real clinical notes, reader studies, prospective evaluation, workflow integration testing, or safety analysis; synthesized notes were not reviewed by medical experts, only lesion type was validated, and the proportion of noisy or unreliable notes cannot be estimated.
  • Stated: report variability is constrained by the synthesis strategies; real-image size is limited to 224×224; the framework was not evaluated on non-medical imaging modalities; and cross-modal retrieval uses lesion-level classes as a proxy for alignment quality, not finer-grained morphology.
  • Evident/stated: automated metrics are proxies—BERTScore is insensitive to factual clinical errors, and weighted F1 only detects lesion type, not other incorrect lesion characteristics; foundation-model comparisons lack statistical tests because models were evaluated at fixed checkpoints. The text also notes this is an unedited article-in-press version that may contain errors.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-026-03040-3

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
From the PDF
Image 2
From the PDF
Image 3
From the PDF
Image 4
From the PDF
ARTICLE IN PRESS Table 12. Overview of the dataset composition. The dataset contains dermatology images paired with synthesized clinical notes, collected from multiple publicly available sources. Each sample is labeled with one of eight classes: BEK (Benign Keratosis), DF (Dermatofibroma), NEV (Benign Nevus), VASC (Vascular Lesion), ACK (Pre-cancer Actinic Keratosis), BCC (Basal Cell Carcinoma), MEL (Melanoma), SCC (Squamous Cell Cancer). Images are either dermoscopic (D) or clinical (C). Data are split into three partitions: training, validation, testing. Training and validation partitions include four datasets, to exploit both dermoscopic and clinical samples. The testing partition includes ten datasets: the testing partitions of the four aforementioned datasets (internal partition) and six independent datasets (external partitions). BEK DF NEV VASC Training partition
ARTICLE IN PRESS Table 12. Overview of the dataset composition. The dataset contains dermatology images paired with synthesized clinical notes, collected from multiple publicly available sources. Each sample is labeled with one of eight classes: BEK (Benign Keratosis), DF (Dermatofibroma), NEV (Benign Nevus), VASC (Vascular Lesion), ACK (Pre-cancer Actinic Keratosis), BCC (Basal Cell Carcinoma), MEL (Melanoma), SCC (Squamous Cell Cancer). Images are either dermoscopic (D) or clinical (C). Data are split into three partitions: training, validation, testing. Training and validation partitions include four datasets, to exploit both dermoscopic and clinical samples. The testing partition includes ten datasets: the testing partitions of the four aforementioned datasets (internal partition) and six independent datasets (external partitions). BEK DF NEV VASC Training partitionFrom the PDF

Verification

Claims the paper makes

2 Valid · 4 Not valid · 6 Not checked · 160 Own results and interpretations

67 more claims

Create a free account to see all 172 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 53
  • Probably not a substantial claim. 62
  • The quote only partly states this claim. 7

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Metadata

Authors
Marini, N., Liang, Z., Rajaraman, S., Xue, Z., & Antani, S.
Affiliations
National Institutes of Health (US)
Corresponding author
Antani, Sameer
Venue
npj Digital Medicine
Published
2026-07-17
ISSN
2398-6352 (ISSN-L)
OpenAlex
W7169528786
Cited by
0
References
0
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
the Intramural Research Program of the National Institutes of Health
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
multimodal AI, dermatological images, skin lesion analysis, large language models, hallucination mitigation, image-text dataset, cross-modal retrieval, zero-shot learning, medical foundation models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets. https://app.cochat.ai/papers/10.1038/s41746-026-03040-3
The paper
Marini, N., Liang, Z., Rajaraman, S., Xue, Z., & Antani, S. (2026). Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets. npj Digital Medicine. https://doi.org/10.1038/s41746-026-03040-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.