The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index; 5 of the 27 sources its claims cite were checked for support.
Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets
Open accessAbstract
Multimodal (MM) algorithms and foundation models have shown strong potential for automated skin lesion analysis in dermatology, however, their translation to clinical practice is hindered by the limited availability of large, high-quality image-text datasets. Most public dermatology datasets are small, unimodal, and paired with heterogeneous labels and metadata, restricting effective model development. Large Language Models (LLMs) provide an opportunity to synthesize clinical notes from existing unimodal datasets, but their application is limited by hallucinations that can hinder MM training. This paper investigates and evaluates strategies to generate reliable clinical notes using existing LLMs and metadata paired with real images, aiming to mitigate hallucinations and improve performance on downstream tasks. We train a MM architecture on real dermatology images paired with synthesized notes and evaluate it across 15 datasets (6 internal, 9 external) on cross-modal retrieval and zero-shot learning tasks. Results demonstrate improved robustness and generalization compared to state-of-the-art medical foundation models, under specific synthesis conditions.
Overview
CoChat Summary
TL;DR
Using metadata-conditioned LLM prompts to synthesize clinical notes from unimodal dermatology images, the authors trained a multimodal architecture on ~16,000 lesions and evaluated it on 15 datasets. The metadata-guided LLM strategy (L) improved cross-modal retrieval and zero-shot learning over template-only metadata (M) and unguided LLM prompting (OL), and the small multimodal model outperformed larger dermatology foundation models on most datasets.
Key findings
- M (metadata-filled template) beat all OL (LLM without lesion metadata) variants on cross-modal retrieval on every dataset, with absolute mAP gaps of 0.10–0.40; e.g., BCN20000 image-to-text: M 0.769 ± 0.048 vs OL GPT 0.456 ± 0.018, MedGemma 0.340 ± 0.016, SkinGPT4 0.380 ± 0.029, DermLIP 0.596 ± 0.016. OL degradation was more evident on external datasets.
- L (LLM prompted with metadata) generally matched or outperformed M, especially in text-to-image retrieval and on external datasets; in zero-shot learning, all L models significantly outperformed M across nearly all datasets in the 8-class setup, with amplified gains in the 15-class setup. Example BCN20000 8-class: M 0.554 ± 0.037 vs L GPT 0.738 ± 0.014, MedGemma 0.718 ± 0.050, SkinGPT4 0.724 ± 0.045, DermLIP 0.691 ± 0.053; 15-class BCN20000: M 0.269 ± 0.023 vs DermLIP 0.590 ± 0.009.
- The MM architecture trained with L outperformed BioMedCLIP, MedImageInsights, MONET, and Derm1M on cross-modal retrieval and zero-shot learning, with mAP improvements of 0.10–0.30 and zero-shot F1 gaps of 0.20–0.40 on dermoscopic datasets. Example BCN20000 image-to-text: BioMedCLIP 0.320, MedImageInsights 0.682, MONET 0.333, Derm1M 0.403 vs ours DermLIP 0.791 ± 0.012.
- LLM note content: L with GPT/MedGemma often included the correct lesion type, while OL dropped sharply; e.g., BCN20000 correct lesion type: GPT L 1.000 vs OL 0.284; SkinGPT4 L 0.793 vs OL 0.001; DermLIP L 0.619 vs OL 0.201. On PH2, non-medical features such as symmetry and color were described more accurately than dermatology-specific features.
Methods
- Design: a multimodal architecture with a PanDerm image encoder (ViT, 85M parameters, 224×224×3, 24 transformer blocks, fully finetuned) and a frozen PubMedBERT text encoder (512 tokens, 12 transformer blocks, ~110M parameters), projected to 128 dimensions and passed through shared fully connected layers with an 8-class classifier; trained with L1, cosine similarity, and NT-Xent losses, alternating modalities.
- Data/sample: ~16,000 skin lesions from six public datasets for training/validation; 15 datasets total (6 internal, 9 external) for evaluation; eight diagnostic classes; dermoscopic and clinical images; PH2 (200 images) for lesion-feature validation. Clinical notes were synthesized via M (metadata template), L (LLM prompted with metadata), and OL (LLM without metadata) using GPT-4o-mini, MedGemma, SkinGPT4, and DermLIP.
- Analysis: cross-modal retrieval (mAP), zero-shot learning (weighted F1 for 8- and 15-class setups), and clinical note generation (BERTScore, weighted F1); ten runs reported as mean ± SD; Wilcoxon signed-rank test with p < 0.05; foundation models evaluated at fixed checkpoints without retraining.
Limitations
- Stated: the method requires a supervised classifier, limiting task-agnostic use and generalization to completely different diseases; there is no clinical validation against real clinical notes, reader studies, prospective evaluation, workflow integration testing, or safety analysis; synthesized notes were not reviewed by medical experts, only lesion type was validated, and the proportion of noisy or unreliable notes cannot be estimated.
- Stated: report variability is constrained by the synthesis strategies; real-image size is limited to 224×224; the framework was not evaluated on non-medical imaging modalities; and cross-modal retrieval uses lesion-level classes as a proxy for alignment quality, not finer-grained morphology.
- Evident/stated: automated metrics are proxies—BERTScore is insensitive to factual clinical errors, and weighted F1 only detects lesion type, not other incorrect lesion characteristics; foundation-model comparisons lack statistical tests because models were evaluated at fixed checkpoints. The text also notes this is an unedited article-in-press version that may contain errors.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
2 Valid · 4 Not valid · 6 Not checked · 160 Own results and interpretations
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While lesion type information is reliably provided by metadata”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Counterintuitively, LLMs specialized in dermatology tend to output more hallucinated content than general-purpose (GPT) or general biomedical (MedGemma) models.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, when textual supervision is limited, as in the M strategy that focuses only on lesion type, generalization performance is reduced.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite reaching lower performance in the 15-classes setup, the relative performance drop from 8-class to 15class setups is significantly smaller for MM architectures compared to state-of-the-art foundation models”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When multiple lesion descriptions are used during training, the model achieves robust performance across several external datasets.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
67 more claims
Create a free account to see all 172 claims and every source verdict.
- It may state two separate claims as one. 53
- Probably not a substantial claim. 62
- The quote only partly states this claim. 7
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ ARTICLE IN PRESS · p. 9
“In both cases, M achieves the highest performance on every dataset, including both internal and external partitions.”
§ ARTICLE IN PRESS · p. 8
“| HAM10000 | 0.289 ± 0.008 | 0.458 ± 0.035 | 0.571 ± 0.028 | 0.253 ± 0.003 | 0.484 ± 0.036 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ ARTICLE IN PRESS · p. 20
“the MM architecture trained with LLM-synthesized clinical notes achieve consistently higher F1-scores across all datasets and class granularity setups.”
§ ARTICLE IN PRESS · p. 19
“| SD198 | 0.166 | 0.048 | 0.385 | 0.458 | 0.275 ± 0.021 | 0.237 ± 0.033 | 0.268 ± 0.035 | 0.285 ± 0.018 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ ARTICLE IN PRESS · p. 18
“The results on cross-modal retrieval, for both image-to-text and text-to-image retrieval tasks, show that our MM architecture, trained with L, achieves higher average F1-scores than largescale foundation models”
§ MM architecture reaches higher performance than state-of-the-art dermatology foundation models · p. 16
“Performance is reported as average mAP ± standard deviation (ten repetitions) for image-to-text (upper part) and text-to-image (lower part) retrieval across datasets.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Metadata
- Authors
- Marini, N., Liang, Z., Rajaraman, S., Xue, Z., & Antani, S.
- Affiliations
- National Institutes of Health (US)
- Corresponding author
- Antani, Sameer
- Venue
- npj Digital Medicine
- Published
- 2026-07-17
- ISSN
- 2398-6352 (ISSN-L)
- OpenAlex
- W7169528786
- Semantic Scholar
- e8bc906c446c0006863b1d4bb23a662660c5b846
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (am)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- the Intramural Research Program of the National Institutes of Health
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- multimodal AI, dermatological images, skin lesion analysis, large language models, hallucination mitigation, image-text dataset, cross-modal retrieval, zero-shot learning, medical foundation models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets. https://app.cochat.ai/papers/10.1038/s41746-026-03040-3
- The paper
- Marini, N., Liang, Z., Rajaraman, S., Xue, Z., & Antani, S. (2026). Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets. npj Digital Medicine. https://doi.org/10.1038/s41746-026-03040-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.