Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index; 10 of the 18 sources its claims cite were checked for support.

A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings

Open access

Abstract

Large foundation models show promise in biomedicine but face challenges in clinical use due to performance gaps, accessibility, cost, and lack of scalable evaluation. Here we show that open-source small multimodal models can bridge these gaps in radiology by generating free-text findings from chest X-ray images. ... To enable robust, clinically relevant evaluation, we develop and validate CheXprompt, a GPT-4-based metric for assessing factual accuracy aligned with radiologists’ evaluations.

Overview

CoChat Summary

## TL;DR

This paper introduces LLaVA-Rad, a 7B open-source multimodal model for generating chest X-ray findings, trained with a data-centric pipeline on 697,435 image-report pairs. It also proposes CheXprompt, a GPT-4-based factuality metric that correlates better with radiologist error counts than existing automated report metrics.

## Key findings

  • LLaVA-Rad was trained using 697K curated chest X-ray image-text pairs from 7 datasets, sourced from approximately 258,639 patients.
  • On MIMIC-CXR report-generation benchmarks, LLaVA-Rad outperformed larger models including GPT-4V and Med-PaLM M (84B); the paper reports 12.1% relative improvement in ROUGE-L and 10.1% in F1-RadGraph over Med-PaLM M.
  • CheXprompt, evaluated on ReXval with annotations from 6 board-certified radiologists over 200 report pairs, achieved Kendall’s Tau-b correlations > 0.75 for total errors and > 0.70 for clinically significant errors; existing metrics did not exceed 0.57.
  • CheXprompt’s GPT-4 Turbo version had lower mean absolute difference than a left-out radiologist on average (0.55 MAD vs 0.71 MAD), and LLaVA-Rad had the highest share of error-free reports: 6.79% free of clinically significant errors and 2.58% free of all errors.

## Methods

  • The model uses a three-stage training pipeline: pretrain a domain-specific vision encoder (BiomedCLIP-CXR) on chest X-ray image-text pairs, align the image encoder with a language model using a lightweight MLP adapter, then fine-tune with LoRA on MIMIC-CXR report data.
  • Training data were assembled from 7 public datasets into CXR-697K; where free-text reports were missing or incomplete, the authors used GPT-4 to synthesize or structure reports, and they used MIMIC-CXR train split data for later training stages.
  • Evaluation included cross-modal retrieval (recall@K), report-generation metrics (BLEU-4, ROUGE-L, F1-CheXbert, F1-RadGraph), attention visualization, and CheXprompt validation against radiologist annotations in ReXval.

## Limitations

  • The paper states that LLaVA-Rad is “not immediately ready for real-time clinical deployment” and is presented as a step toward clinically adaptable multimodal AI.
  • The scope is limited to chest X-rays; the authors note that future work should test other anatomies and modalities.
  • The text makes evident that performance may vary across datasets and populations because the evaluation sets differ in disease prevalence and setting; this is explicitly noted by the authors as a generalizability concern.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-025-58344-x

Cites

66 of 67 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 LLaVA-Rad overview. a To train LLaVA-Rad, we assemble a large dataset with over 697 thousand chest X-ray image-text pairs; GPT-4 is used to synthesize reports from labels, translate reports from Spanish, and process and structure the corresponding radiology reports. b We adopt a modular three-stage approach to train LLaVA-Rad, comprised of pre-training, alignment and fine-tuning. c A qualitative visualization of the model’s attention during its generative process. d For evaluation, we also propose a novel factual error scoring approach using GPT-4 and demonstrate its parity with expert evaluation. e LLaVA-Rad outperforms much larger generalist and specialized models like GPT-4V and Med-PaLM M on prior standard report evaluation metrics. MLP multi-layer perceptron. The example chest X-ray image in b is obtained from ref. 27 with permission for reproduction from the authors.
Fig. 1 LLaVA-Rad overview. a To train LLaVA-Rad, we assemble a large dataset with over 697 thousand chest X-ray image-text pairs; GPT-4 is used to synthesize reports from labels, translate reports from Spanish, and process and structure the corresponding radiology reports. b We adopt a modular three-stage approach to train LLaVA-Rad, comprised of pre-training, alignment and fine-tuning. c A qualitative visualization of the model’s attention during its generative process. d For evaluation, we also propose a novel factual error scoring approach using GPT-4 and demonstrate its parity with expert evaluation. e LLaVA-Rad outperforms much larger generalist and specialized models like GPT-4V and Med-PaLM M on prior standard report evaluation metrics. MLP multi-layer perceptron. The example chest X-ray image in b is obtained from ref. 27 with permission for reproduction from the authors.CC BY-NC-ND · PubMed Central
Fig. 2 Quantitative and qualitative evaluation of LLaVA-Rad using existing report generation benchmarks on MIMIC-CXR. a Comparison between LLaVA-Rad and open-source models according to existing factual correctness (F1-CheXbert-14, F1-RadGraph) and lexical similarity (ROUGE-L) metrics. b Comparison between LLaVA-Rad and closed-source models according to existing factual correctness and lexical similarity metrics. c Comparison between model size and factual correctness shows that LLaVA-Rad is both smaller and more factually correct compared to existing approaches. d Illustration of a sample generated report from LLaVA-Rad compared with that of LLaVA and LLaVA-Med. LLaVA-Rad’s generations that match reference findings are highlighted. e Comparison of the performance on cross-modal retrieval demonstrated by LLaVA-Rad, LLaVA-Med and LLaVA. In a–e values correspond to mean statistic in MIMIC-CXR test-set (n = 2461 image-report pairs) with the exception of MAIRA-1 and Med-PaLM M which are derived from their original publications. In a, b error bars correspond to 95% bootstrap confidence intervals derived from 500 samples. Source data are provided as a Source Data file.
Fig. 2 Quantitative and qualitative evaluation of LLaVA-Rad using existing report generation benchmarks on MIMIC-CXR. a Comparison between LLaVA-Rad and open-source models according to existing factual correctness (F1-CheXbert-14, F1-RadGraph) and lexical similarity (ROUGE-L) metrics. b Comparison between LLaVA-Rad and closed-source models according to existing factual correctness and lexical similarity metrics. c Comparison between model size and factual correctness shows that LLaVA-Rad is both smaller and more factually correct compared to existing approaches. d Illustration of a sample generated report from LLaVA-Rad compared with that of LLaVA and LLaVA-Med. LLaVA-Rad’s generations that match reference findings are highlighted. e Comparison of the performance on cross-modal retrieval demonstrated by LLaVA-Rad, LLaVA-Med and LLaVA. In a–e values correspond to mean statistic in MIMIC-CXR test-set (n = 2461 image-report pairs) with the exception of MAIRA-1 and Med-PaLM M which are derived from their original publications. In a, b error bars correspond to 95% bootstrap confidence intervals derived from 500 samples. Source data are provided as a Source Data file.CC BY-NC-ND · PubMed Central
Fig. 3 External validation results for LLaVA-Rad on held-out datasets. Open-I (a, b) CheXpert (c, d) and US-CXR (e, f). LLaVA-Rad outperforms baselines across all external validation datasets, as assessed by traditional factual correctness metrics (F1-CheXbert-14, F1-RadGraph) and lexical similarity (ROUGE-L). CheXprompt evaluation (b, d, f) further demonstrates that LLaVA-Rad produces fewer clinically significant and overall errors compared to baselines. Each dataset sample consists of image-report pairs (Open-I: n = 2163; CheXpert: n = 61; US-CXR: n = 1751). Values represent mean metric scores for each dataset, and error bars indicate 95% bootstrap confidence intervals derived from 500 resampling iterations. Source data are provided as a Source Data file.
Fig. 3 External validation results for LLaVA-Rad on held-out datasets. Open-I (a, b) CheXpert (c, d) and US-CXR (e, f). LLaVA-Rad outperforms baselines across all external validation datasets, as assessed by traditional factual correctness metrics (F1-CheXbert-14, F1-RadGraph) and lexical similarity (ROUGE-L). CheXprompt evaluation (b, d, f) further demonstrates that LLaVA-Rad produces fewer clinically significant and overall errors compared to baselines. Each dataset sample consists of image-report pairs (Open-I: n = 2163; CheXpert: n = 61; US-CXR: n = 1751). Values represent mean metric scores for each dataset, and error bars indicate 95% bootstrap confidence intervals derived from 500 resampling iterations. Source data are provided as a Source Data file.CC BY-NC-ND · PubMed Central
Fig. 4 Evaluating LLaVA-Rad using CheXprompt. a GPT-4 based CheXprompt is more similar to average left-in radiologists in total error quantification, compared to the left-out radiologist (mean absolute difference 0.55 vs 0.71). b Comparison between CheXprompt and existing metrics in terms of agreement with radiologist error quantification. c Comparison between LLaVA-Rad and competing methods using CheXprompt on the MIMIC-CXR test set. d Illustration of how CheXprompt can be used to evaluate a report generated by LLaVA-Rad, with errors highlighted. GPT-4T stands for GPT-4 Turbo. In a p values correspond to two-sided paired t-test. In b, c values represent mean metric scores and error bars correspond to 95% bootstrap confidence intervals. Source data are provided as a Source Data file.
Fig. 4 Evaluating LLaVA-Rad using CheXprompt. a GPT-4 based CheXprompt is more similar to average left-in radiologists in total error quantification, compared to the left-out radiologist (mean absolute difference 0.55 vs 0.71). b Comparison between CheXprompt and existing metrics in terms of agreement with radiologist error quantification. c Comparison between LLaVA-Rad and competing methods using CheXprompt on the MIMIC-CXR test set. d Illustration of how CheXprompt can be used to evaluate a report generated by LLaVA-Rad, with errors highlighted. GPT-4T stands for GPT-4 Turbo. In a p values correspond to two-sided paired t-test. In b, c values represent mean metric scores and error bars correspond to 95% bootstrap confidence intervals. Source data are provided as a Source Data file.CC BY-NC-ND · PubMed Central
Fig. 5 Analyzing the performance of LLaVA-Rad using ablation studies and attention visualization. a Comparison of using different image encoders (BiomedCLIP-CXR from LLaVA-Rad, BiomedCLIP continually pre-trained on MIMIC-CXR, BiomedCLIP, and OpenAI CLIP) to start the alignment and fine-tuning stages. b Ablation study on only using rule-processed MIMIC-CXR training data or GPT-4 processed training data in alignment and fine-tuning stages. c Attention visualization qualitatively demonstrates the appropriate grounding of LLaVA-Rad in-specific image regions when generating a word (bold text) as part of a specific finding (bottom row). In a, b values represent mean metric scores and error bars indicate 95% bootstrap confidence intervals derived from 500 resampling iterations. Source data are provided as a Source Data file.
Fig. 5 Analyzing the performance of LLaVA-Rad using ablation studies and attention visualization. a Comparison of using different image encoders (BiomedCLIP-CXR from LLaVA-Rad, BiomedCLIP continually pre-trained on MIMIC-CXR, BiomedCLIP, and OpenAI CLIP) to start the alignment and fine-tuning stages. b Ablation study on only using rule-processed MIMIC-CXR training data or GPT-4 processed training data in alignment and fine-tuning stages. c Attention visualization qualitatively demonstrates the appropriate grounding of LLaVA-Rad in-specific image regions when generating a word (bold text) as part of a specific finding (bottom row). In a, b values represent mean metric scores and error bars indicate 95% bootstrap confidence intervals derived from 500 resampling iterations. Source data are provided as a Source Data file.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

5 Valid · 5 Not valid · 6 Not checked · 85 Own results and interpretations

56 more claims

Create a free account to see all 101 claims and to verify the 6 references still open.

See all claims — free account
  • It may state two separate claims as one. 35
  • Probably not a substantial claim. 10
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Citations

Cites

Show all 56 references

Research works printed but not linked: 3

  1. Lehman, E. et al. Do we still need clinical language models? In Proc Conference on Health, Inference, and Learning, vol. 209 of Proceedings of Machine Learning Research, (eds Mortazavi, B. J. et al.) 578–597 (PMLR, 2023).
  2. Hu, E. J. et al. LoRA: Low-rank adaptation of large language models. In Proc International Conference on Learning Representations. https://openreview.net/forum?id=nZeVKeeFYf9 (2022).
  3. Ramesh, V., Chi, N. A. & Rajpurkar, P. Improving radiology report generation systems by removing hallucinated references to non-existent priors. In Machine Learning for Health, (eds Parziale, A. et al.) 456–473 (PMLR, 2022).

Cited by

See all 46

Metadata

Authors
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., Awadalla, H., Gong, J., Hu, H., Yang, J., Li, C., Gao, J., Gu, Y., Wong, C., Wei, M., Naumann, T., Chen, M., Lungren, M. P., Chaudhari, A., Yeung-Levy, S., Langlotz, C. P., Wang, S., & Poon, H.
Affiliations
  • Stanford University (US)
  • Microsoft (United States) (US)
  • University of Washington (US)
  • University of Southern California (US)
  • University of California, Davis (US)
Corresponding author
  • Wang, Sheng
  • Poon, Hoifung
Venue
Nature Communications
Published
2025-04-01
Volume
16
Issue
1
Pages
3108-3108
ISSN
2041-1723 (ISSN-L)
OpenAlex
W4409042270
Cited by
45
References
56
Citations by year
2026: 31 · 2025: 14
FWCI
15.63
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Microsoft Research
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
Keywords
chest X-ray interpretation, multimodal foundation models, small language models, radiology report generation, cross-modal alignment, clinical implementation of AI
MeSH terms
Algorithms, Humans, Radiography, Thoracic, Multimodal Imaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings. https://app.cochat.ai/papers/10.1038/s41467-025-58344-x
The paper
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., Awadalla, H., Gong, J., Hu, H., Yang, J., Li, C., Gao, J., Gu, Y., Wong, C., Wei, M., . . . Poon, H. (2025). A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings. Nature Communications. https://doi.org/10.1038/s41467-025-58344-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.