The DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index; 10 of the 18 sources its claims cite were checked for support.
A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings
Open accessAbstract
Large foundation models show promise in biomedicine but face challenges in clinical use due to performance gaps, accessibility, cost, and lack of scalable evaluation. Here we show that open-source small multimodal models can bridge these gaps in radiology by generating free-text findings from chest X-ray images. ... To enable robust, clinically relevant evaluation, we develop and validate CheXprompt, a GPT-4-based metric for assessing factual accuracy aligned with radiologists’ evaluations.
Overview
CoChat Summary
## TL;DR
This paper introduces LLaVA-Rad, a 7B open-source multimodal model for generating chest X-ray findings, trained with a data-centric pipeline on 697,435 image-report pairs. It also proposes CheXprompt, a GPT-4-based factuality metric that correlates better with radiologist error counts than existing automated report metrics.
## Key findings
- LLaVA-Rad was trained using 697K curated chest X-ray image-text pairs from 7 datasets, sourced from approximately 258,639 patients.
- On MIMIC-CXR report-generation benchmarks, LLaVA-Rad outperformed larger models including GPT-4V and Med-PaLM M (84B); the paper reports 12.1% relative improvement in ROUGE-L and 10.1% in F1-RadGraph over Med-PaLM M.
- CheXprompt, evaluated on ReXval with annotations from 6 board-certified radiologists over 200 report pairs, achieved Kendall’s Tau-b correlations > 0.75 for total errors and > 0.70 for clinically significant errors; existing metrics did not exceed 0.57.
- CheXprompt’s GPT-4 Turbo version had lower mean absolute difference than a left-out radiologist on average (0.55 MAD vs 0.71 MAD), and LLaVA-Rad had the highest share of error-free reports: 6.79% free of clinically significant errors and 2.58% free of all errors.
## Methods
- The model uses a three-stage training pipeline: pretrain a domain-specific vision encoder (BiomedCLIP-CXR) on chest X-ray image-text pairs, align the image encoder with a language model using a lightweight MLP adapter, then fine-tune with LoRA on MIMIC-CXR report data.
- Training data were assembled from 7 public datasets into CXR-697K; where free-text reports were missing or incomplete, the authors used GPT-4 to synthesize or structure reports, and they used MIMIC-CXR train split data for later training stages.
- Evaluation included cross-modal retrieval (recall@K), report-generation metrics (BLEU-4, ROUGE-L, F1-CheXbert, F1-RadGraph), attention visualization, and CheXprompt validation against radiologist annotations in ReXval.
## Limitations
- The paper states that LLaVA-Rad is “not immediately ready for real-time clinical deployment” and is presented as a step toward clinically adaptable multimodal AI.
- The scope is limited to chest X-rays; the authors note that future work should test other anatomies and modalities.
- The text makes evident that performance may vary across datasets and populations because the evaluation sets differ in disease prevalence and setting; this is explicitly noted by the authors as a generalizability concern.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
5 Valid · 5 Not valid · 6 Not checked · 85 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“However, while small language models have shown success in text domains14–16, small multimodal models (SMMs) still have significant performance gaps compared to larger models6,10.”
§ A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings > Introduction
What the sources say
- Disputedcites [6] Tu, T. et al. Towards generalist biomedical AI. NEJM AI 1, AIoa2300138 (2024). sourceExperimental — accuracy not yet measured
In §8, “Perspective on Generalist Biomedical AI” (page unavailable), the paper says the performance gap across three Med-PaLM M sizes was relatively small for chest X-ray report generation. It also says there was no overall evidence that a larger vision model outperformed a smaller one across its experiments. These findings run counter to the claimed significant performance gap. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
- Could not verifycites [10] Tanno, R. et al. Collaboration between clinicians and vision–language models in radiology report generation. Nat. Med. 31 1–10 (2024). sourceExperimental — accuracy not yet measured
Our copy of Tanno et al. (2025) contains no statement about small multimodal models lagging larger ones: searches for 'small multimodal models performance gap', 'model size scaling', 'built from scratch underperform large pretrained', and 'limited capabilities of smaller VLMs' returned only the Main paragraph on existing report-generation models being 'built from scratch, neglecting the likely useful transfer of knowledge from such pretrained models' (Main) — a pretraining argument, not a size argument. If anything, the paper runs counter to the claim's framing: Methods > Task (Architecture) states Flamingo-CXR 'is built using a parsimonious 400 million parameter version', and Table 1 (Results, Automated report generation metrics) shows it at 0.519 CheXpert F1 (all), above the 12B Med-PaLM-M (0.514), with the text reporting it 'outperforms the current SoTA method by 33%'. Since our copy neither states nor contradicts a general small-vs-large multimodal performance gap, the attribution cannot be settled from this source. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Benchmarked with CheXprompt and other standard factuality metrics, LLaVA-Rad (7B) achieves state-of-the-art performance, outperforming much larger models like GPT-4V and Med-PaLM M (84B).”
- Also stated in: § Introduction · plain
“The resulting LLaVA-Rad (7B) model attains state-of-the-art results on standard radiology tasks such as report generation and cross-modal retrieval, even outperforming much larger models such as GPT-4V and Med-PaLM M (84B)6.”
§ A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings > Abstract
What the sources say
- Could not verifycites [6] Tu, T. et al. Towards generalist biomedical AI. NEJM AI 1, AIoa2300138 (2024). sourceExperimental — accuracy not yet measured
Table A.7 and the Additional information section report state-of-the-art results for Med-PaLM M on clinical factuality metrics. Our copy does not report results for LLaVA-Rad or GPT-4V, use CheXprompt, or compare either model with Med-PaLM M (searched: LLaVA-Rad, GPT-4V, CheXprompt). It therefore cannot establish the attributed performance claim. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Finally, even the best models are still subject to errors such as hallucination, and existing automated evaluation methods of factual correctness exhibit limited correlation with expert assessments17.”
- Also stated in: § Evaluating LLaVA-Rad using CheXprompt, a GPT-4-based evaluation system · hedged
“It is well known that existing n-gram and findings-based automated report evaluation methods might be biased to pre-defined conditions and have limited correlation with expert assessments17.”
§ A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings > Introduction
What the sources say
- Supportedcites [17] Yu, F. et al. Evaluating progress in automatic chest x-ray radiology report generation. Patterns 4, 100802 (2023). source
The Introduction states that even metrics designed to assess the correctness of radiology information do not align with radiologists. The Discussion likewise says prior evaluations poorly approximate radiologists’ evaluations of reports. These passages support the claim about limited alignment with expert assessments. (LLM, conf=0.91)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Evaluating LLaVA-Rad using CheXprompt, a GPT-4-based evaluation system · hedged
Every source the paper cites for this claim was checked and supports it.
In the paper
“We evaluate the validity of the proposed CheXprompt using the ReXval dataset39, which is comprised of error annotations from 6 board-certified radiologists on 200 pairs of candidate and ground-truth reports”
§ A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings > Methods > Quality of generated reports
What the sources say
- Supportedcites [39] Yu, F. et al. Radiology report expert evaluation (rexval) dataset. PhysioNet (2023). source
The Abstract states the dataset 'contains annotations from 6 board certified radiologists' and that 'There are 4 candidate reports generated for 50 studies, translating to 200 pairs of candidate and ground-truth reports on which radiologists provided annotations.' The Methods (Data/Participants, p. ~1) independently confirms 50 MIMIC-CXR studies with 4 generated reports each (200 pairs) annotated by 6 board certified radiologists, and the Data Description section repeats the 6-radiologist error-annotation structure. The citing sentence's description of ReXVal matches our copy exactly. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we develop and validate CheXprompt, a GPT-4-based metric for assessing factual accuracy aligned with radiologists’ evaluations.”
§ A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings > Abstract
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
56 more claims
Create a free account to see all 101 claims and to verify the 6 references still open.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 10
- The quote only partly states this claim. 2
Citations
Cites
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 1×
“Automatically drafting high-quality radiology reports isa challenging but clinically meaningful task that could directly increase radiologist productivity and potentially improve communication and decreaseburnout 19 .”
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Self-citationFoundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · cited 2×
“MIMIC-CXR comprises images and their corresponding radiology reports sourced from radiographic studies conducted at the Beth Israel Deaconess Medical Center in Boston, MA, spanning the years 2011 to 2016 27 .”
Self-citationImproved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748 · cited 2×
“As noted by other works 9,29 and also demonstrated by our ablation studies, this strategy can substantially improve the alignment by forcing the decoder model to focus on the image alone.”
Self-citationImproved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2310.03744 · Cited by 1,733
Self-citationFlamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555 · cited 1×
“For more optimal positioning, the towould have to be pulled back almost 10cm and then hopefully redirected toward the lower stomach.”
Research works printed but not linked: 3
- Lehman, E. et al. Do we still need clinical language models? In Proc Conference on Health, Inference, and Learning, vol. 209 of Proceedings of Machine Learning Research, (eds Mortazavi, B. J. et al.) 578–597 (PMLR, 2023).
- Hu, E. J. et al. LoRA: Low-rank adaptation of large language models. In Proc International Conference on Learning Representations. https://openreview.net/forum?id=nZeVKeeFYf9 (2022).
- Ramesh, V., Chi, N. A. & Rajpurkar, P. Improving radiology report generation systems by removing hallucinated references to non-existent priors. In Machine Learning for Health, (eds Parziale, A. et al.) 456–473 (PMLR, 2022).
Cited by
Merlin: a computed tomography vision–language foundation model and dataset
Blankemeier, L., Kumar, A., Cohen, J., Liu, J., Liu, L., Veen, D. V., Gardezi, S. J. S., Yu, H., Paschali, M., Chen, Z., et al. (2026) · Nature · doi:10.1038/s41586-026-10181-8 · Cited by 201
Self-citationDrafting the Future: The Dawn of AI Report Generation in Radiology
Seah, J. C. Y., Tang, J. S. N., & Tran, A. (2025) · Radiology · doi:10.1148/radiol.243378 · Cited by 21
MAIRA-2: Grounded Radiology Report Generation
Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04449 · Cited by 19
“Hallucinations or omissions of such mentions would not be detected by CheXprompt.”
BackgroundMethodInfluentialSelf-citationAI-Driven Digital Pathology: Deep Learning and Multimodal Integration for Precision Oncology
Jang, H., & Lee, S. H. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms27010379 · Cited by 14
“Many models align image features with related medical reporting to support visual question answering (VQA), report generation, and structured reasoning [10–12].”
Huang, L., Lin, A., Li, H., Wang, Q., Shen, J., Jiang, A., Chang, Q., Gan, W., Zhu, L., Mou, W., et al. (2025) · Open Research (University of Surrey) · doi:10.1002/viw.20250077 · Cited by 9
Metadata
- Authors
- Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., Awadalla, H., Gong, J., Hu, H., Yang, J., Li, C., Gao, J., Gu, Y., Wong, C., Wei, M., Naumann, T., Chen, M., Lungren, M. P., Chaudhari, A., Yeung-Levy, S., Langlotz, C. P., Wang, S., & Poon, H.
- Affiliations
- Stanford University (US)
- Microsoft (United States) (US)
- University of Washington (US)
- University of Southern California (US)
- University of California, Davis (US)
- Corresponding author
- Wang, Sheng
- Poon, Hoifung
- ORCID
- Zambrano Chaves, Juan Manuel
- Huang, Shih-Cheng
- Xu, Yanbo
- Xu, Hanwen
- Usuyama, Naoto
- Zhang, Sheng
- Wang, Fei
- Khademi, Mahmoud
- Yang, Ziyi
- Awadalla, Hany
- Gong, Julia
- Yang, Jianwei
- Li, Chunyuan
- Gao, Jianfeng
- Wong, Cliff
- Wei, Mu
- Naumann, Tristan
- Chen, Muhao
- Lungren, Matthew P.
- Chaudhari, Akshay
- Yeung-Levy, Serena
- Langlotz, Curtis P.
- Wang, Sheng
- Poon, Hoifung
- Venue
- Nature Communications
- Published
- 2025-04-01
- Volume
- 16
- Issue
- 1
- Pages
- 3108-3108
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC11962106
- OpenAlex
- W4409042270
- Cited by
- 45
- References
- 56
- Citations by year
- 2026: 31 · 2025: 14
- FWCI
- 15.63
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Keywords
- chest X-ray interpretation, multimodal foundation models, small language models, radiology report generation, cross-modal alignment, clinical implementation of AI
- MeSH terms
- Algorithms, Humans, Radiography, Thoracic, Multimodal Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings. https://app.cochat.ai/papers/10.1038/s41467-025-58344-x
- The paper
- Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., Awadalla, H., Gong, J., Hu, H., Yang, J., Li, C., Gao, J., Gu, Y., Wong, C., Wei, M., . . . Poon, H. (2025). A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings. Nature Communications. https://doi.org/10.1038/s41467-025-58344-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.