CoChat
Sign Up
Cited by 76Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationRadiology practices and education

Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation

Open access
Li, Cheng-Yi, Chang, Kao-Jung, Yang, Cheng-Fu, Wu, Hsin-Yu, Chen, Wenting, Bansal, Hritik, Chen, Ling, Yang, Yi-Ping +6 more·2025·Nature Communications
DOI 10.1038/s41467-025-57426-0PMC PMC11885477

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Abstract Multi-modal large language models (MLLMs) have been given free rein to explore exciting medical applications with a primary focus on radiology report generation. Nevertheless, the preliminary MLLM successful attempts in 2D medical image-text pair captioning are incompetent to reflect the real-world diagnostic challenge in the volumetric 3D anatomy. Toward deploying MLLM for more applicable diagnostic context, we noticed that the (1) scarcity of 3D image training dataset, (2) the direct use of undifferentiated foundation MLLMs, and (3) the lack of pertinent caption evaluation metrics were independent domain-specific constraints that integratively hobbles the iteration of next-generation medical MLLM research. In this regard, this study collected a 3D-BrainCT dataset (18,885 text-scan pairs) and applied clinical visual instruction tuning (CVIT) to train volumetric anatomy-sensible BrainGPT models to generate radiology-adherent 3D brain CT reports. Statistically, our BrainGPT model scored BLEU-1 = 44.35, BLEU-4 = 20.38, METEOR = 30.13, ROUGE-L = 47.6, and CIDEr-R = 211.77 during internal testing and demonstrated an accuracy of 0.91 in captioning midline shifts on the external validation CQ500 dataset. By further inspecting the captioned report, we reported that the traditional metrics appeared to measure only the surface text similarity and failed to gauge the information density of the diagnostic purpose. To close this gap, we proposed a novel Feature-Oriented Radiology Task Evaluation (FORTE) to estimate the clinical relevance (lesion feature and landmarks) of the report. Notably, the BrainGPT model scored an average FORTE 0.71 F1-score (degree=0.661; landmark=0.706; feature=0.693, and impression=0.779). To demonstrate that BrainGPT models possess objective readiness to generate human-like radiology reports, we conducted a Turing test that enrolled 11 physician evaluators, and around 74% of the BrainGPT-generated captions were indistinguishable from those written by humans. While various computational intelligence researchers have advocated the avant-garde MLLM applications, our work embodies a holistic framework that showcased the first-hand experience of curating a 3D brain CT dataset, fine-tuning anatomy-sensible language models, and proposing robust radiology evaluation metrics. We deemed that the adventure of docking MLLM for 3D brain CT report generation may unfold new MLLM applications at the forefront of human-machine collaborated modern healthcare.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces BrainGPT, a clinically visual instruction-tuned multimodal LLM for 3D brain CT radiology report generation, trained on the curated 3D-BrainCT dataset of 18,885 text-scan pairs. It also proposes FORTE, a feature-oriented evaluation scheme, and reports that BrainGPT achieves an average FORTE F1-score of 0.71, with 74% of generated reports indistinguishable from human-written ground truth in a Turing-like test.

Key findings

  • BrainGPT-keyword achieved an average FORTE F1-score of 0.71 after negation removal, with category scores of 0.661 for degree, 0.706 for landmark, 0.693 for feature, and 0.779 for impression.
  • In a Turing-like test with 11 physicians and 66 evaluations, 74.24% of BrainGPT-generated captions were mistakenly identified as human-written; when CT image context was provided, misidentification fell to 56%.
  • On zero-shot external validation using CQ500 (n = 133), BrainGPT models reported mass effect with accuracy 0.71–0.75, midline shift 0.35–0.38, and hemorrhagic events 0.43–0.5; after negation removal, midline shift accuracy rose to 0.86–0.91 and hemorrhagic events to 0.59–0.61.
  • FORTE correlated with DocLens LLM-as-a-judge scores (r = 0.78, p < 0.001) but traditional metrics correlated poorly with DocLens (r < 0.4, p < 0.001), supporting FORTE as a more clinically aligned evaluation.

Methods

  • Design: retrospective development and evaluation of BrainGPT, an Otter-based multimodal LLM fine-tuned with regular visual instruction tuning (RVIT) or clinical visual instruction tuning (CVIT), using four instruction conditions: plain, example, template, and keyword.
  • Data: 3D-BrainCT included 18,885 brain CT scans (742,501 slices) from 9,689 patients with Alzheimer’s Disease (mean age 82.59 years, SD 9.3; 56.4% male) at Taipei Veterans General Hospital, 2010–2022. Training used 365,928 slices from 15,238 scans of 7,747 patients; testing used 87,312 slices from 3,638 scans of 1,938 patients. External validation used CQ500 non-contrast CT scans (n = 133).
  • Analysis: traditional language metrics (BLEU, METEOR, ROUGE-L, CIDEr-R), sentence pairing and negation removal preprocessing, FORTE keyword-category F1 scoring, DocLens LLM-as-a-judge, physician RADPEER-based ratings, and a linguistic-embedded Turing test.

Limitations

  • The paper states this is a pilot volumetric brain CT captioning study with no counterpart MLLM module to benchmark, so SOTA-level efficacy cannot be fully justified.
  • The paper states BrainGPT was trained on degeneration-oriented data and fails to capture malignant tumor and acute traumatic features in CQ500; it also did not test whether changing the model backbone could improve brain CT captioning.
  • Evident from the text: the training data come from a single institution and an Alzheimer’s Disease-enriched elderly population, which may limit generalizability; the Turing test involved 11 physicians and 66 evaluations, a relatively small human evaluation sample.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-025-57426-0

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 46 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    'No' was notably among the most frequently used words in BrainGPT reports, along with 'of,' 'and,' 'in,' and 'the.'

    “we found that “no” was notably among the most frequently used words, along with “of,” “and,” “in,” and “the.””
    Quote in the paperSubstantial claim
  • Findingown data

    The baseline Otter model scored the lowest in BLEU-4 (0) and CIDEr-R (5.9) metrics.

    “the baseline Otter model scored the lowest in the BLEU-4 (0) and CIDEr-R (5.9) metrics”
    Quote in the paperSubstantial claim
  • Findingown data

    Traditional LLM metrics failed to gauge the diagnostic quality of radiology report generation.

    “While we notice that the traditional LLM metrics failed to gauge the diagnostic quality of the RRG”
    • Also stated in: § Training BrainGPT with clinical visual instruction tuning · boosted
      “we demonstrated that traditional metrics are inherently insensitive to the clinical essence of the generated radiology reports.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
  • Findingnone given

    BrainGPT is a clinically visual instruction-tuned (CVIT) model designed for 3D CT radiology report generation.

    “develop BrainGPT, a clinically visual instruction-tuned (CVIT) model designed for 3D CT RRG”
    • Also stated in: § Introduction · plain
      “We introduced a clinical visual instruction tuning (CVIT) concept that enhances medical domain knowledge of the open-source Otter foundation model.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The utility of MLLM-based radiology report generation for 3D medical images remains largely unexplored.
    • Rests on an assumption: Current single-slice CT image-text data is manually pre-selected from 3D data, so MLLMs are largely tested on definite-positive lesion-included images.
    • Rests on an assumption: Testing MLLMs on definite-positive lesion-included images may result in a sharpshooter fallacy.
  • Findingown data

    No traditional metrics, except CIDEr-R, rated BrainGPT performance in accordance with the crescendo elaborateness of RVIT and CVIT.

    “no traditional metrics, except CIDEr-R, rated the BrainGPT performance in accordance with the crescendo elaborateness of RVIT and CVIT”
    Quote in the paperSubstantial claim
  • May combine two claims. 43
  • Classified as probably not a substantial claim. 33

  • May combine two claims.

    Findingown data

    Sentence pairing increased traditional metric scores by an average of 5.28 points in METEOR, 6.48 points in ROUGE-L, and 114 points in CIDEr-R.

    “the process of sentence pairing increased traditional metric scores by an average of 5.28 points in METEOR, 6.48 points in ROUGE-L, and an astonishing 114 points in CIDEr-R”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The baseline Otter model had a low recall rate for keywords, whereas BrainGPT models after visual instruction tuning demonstrated significant radiology keyword usage in image captioning.

    “We found that the baseline Otter model had a low recall rate for keywords, whereas the BrainGPT models, after visual instruction tuning, demonstrated significant radiology keyword usage in image captioning.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Traditional metrics had high intracorrelations (r > 0.7, p < 0.001), but a lower correlation with FORTE (r < 0.5, p < 0.001).

    “traditional metrics had high intracorrelations (r > 0.7, p < 0.001), but a lower correlation with the FORTE (r < 0.5, p < 0.001).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
    • Rests on an assumption: The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

  • May combine two claims.

    Findingown data

    BrainGPT's reports were overlaid with excessive negative descriptions rather than concise, context-specific language used by radiologists, a phenomenon termed 'Interpretation spree.'

    “BrainGPT’s reports were overlaid with excessive negative descriptions rather than the concise, context-specific language used by radiologists, a phenomenon we termed the “Interpretation spree.””
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    BrainGPT achieves FORTE F1-scores of 0.661 for degree, 0.706 for landmark, 0.693 for feature, and 0.779 for impression.

    “(degree = 0.661; landmark = 0.706; feature = 0.693, and impression = 0.779)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
    • Rests on an assumption: The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

  • Classified as probably not a substantial claim.

    Findingnone given

    FORTE is an evaluation scheme that captures the clinical essence of generated reports.

    “we propose feature-oriented radiology task evaluation (FORTE), an evaluation scheme that captures the clinical essence of the generated reports”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
    • Rests on an assumption: The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

  • May combine two claims.

    Findingown data

    Advanced CVIT models, BrainGPT-template and BrainGPT-keyword, scored higher average FORTE F1 scores than RVIT models did (Wilcoxon p < 0.05 in most categories).

    “the advanced CVIT models, BrainGPT-template and BrainGPT-keyword, scored higher average F1 scores than the RVIT models did. (Wilcoxon signed-rank test p-value < 0.05 in most categories”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
    • Rests on an assumption: The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

  • May combine two claims.

    Findingown data

    The four instruction-tuned BrainGPT models achieved accuracy in reporting mass effects (0.71-0.75), midline shifts (0.35-0.38), and hemorrhagic events (0.43-0.5) on CQ500.

    “the four instruction-tuned BrainGPT models achieved fair accuracy in reporting mass effects (0.71-0.75), midline shifts (0.35-0.38), and hemorrhagic events (0.43-0.5).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The initial FORTE evaluation showed all keyword categories were present in 20.9-27.4% of BrainGPT-generated reports, whereas the number decreased to 3.3-6.2% after negation removal.

    “The initial FORTE evaluation showed all keyword categories were present in 20.9–27.4% of the BrainGPT-generated reports, whereas the number decreased to 3.3–6.2% after negation removal.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Within the FORTE framework, BrainGPT achieves an F1 score of 0.589, comparable to the state-of-the-art (SOTA) performance in general medical disease identification, which reports an accuracy of 59.2% in prior benchmark studies.

    “Within the FORTE framework, BrainGPT achieves a commendable performance with an F1 score of 0.589, comparable to the state-of-the-art (SOTA) performance in general medical disease identification, which reports an accuracy of 59.2% in prior benchmark studies45.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] Liu, J., et al. A Spectrum evaluation benchmark for medical multi-modal large language models. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240211217L (2024).

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    DocLens had a moderate correlation with human expert opinion in terms of 'Diagnostic Accuracy', and a mild correlation in the remaining rated items.

    “we also noted a moderate correlation between DocLens and human expert opinion in terms of “Diagnostic Accuracy”, and a mild correlation in the remaining rated items.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    'Putamen' was misspelled as 'putmen,' likely due to a typo learned from the original training report.

    ““putamen” was misspelled as “putmen,” likely due to a typo learned from the original training report.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    BrainGPT requires only 12 h of fine-tuning on two NVIDIA A100 GPUs, achieving 3 epochs.

    “However, their customized transformer model required 7 days of training on a single NVIDIA A100 GPU, whereas our BrainGPT model requires only 12 h of fine-tuning on two NVIDIA A100 GPUs.”
    • Also stated in: § Clinical visual instruction tuning (CVIT) · plain
      “The training duration for each resulting model was 12 h on two NVIDIA A100 GPUs, achieving 3 epochs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    FORTE has moderate to high correlation with both human expert assessments and DocLens scores.

    “This is further evidenced by its moderate to high correlation with both human expert assessments and DocLens scores25.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [25] Irvin, J. et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Proc. AAAI Conf. Artif. Intell. 33, 590–597 (2019).

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    When CT input context was provided to evaluators, 18% of the misidentification was reduced, resulting in a 56% incidence of mistakes machine-written as human-written reports.

    “When CT input context was provided to the evaluators, 18% of the misidentification was reduced, resulting a 56% incidence of mistakes machine-written as human-written reports.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    'Degree and landmark' was present in 29.3-35.3% of BrainGPT-generated reports, whereas the number increased to 41.6-55% after negation removal.

    ““degree and landmark” was present in 29.3–35.3% of the BrainGPT-generated reports, whereas the number increased to 41.6–55% after negation removal.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Negation removal led to an overall average F1 score gain of 0.153, with BrainGPT-keyword F1 score rising from 0.576 to 0.71.

    “Negation removal led to an overall average F1 score gain of 0.153, with the best instruction-tuned model, BrainGPT-keyword, seeing its F1 score rise from 0.576 to 0.71.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
    • Rests on an assumption: The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

  • May combine two claims.

    Findingown data

    DocLens claim precision scores and FORTE scores were similar pre-visual instruction tuning (0.102 vs 0.102) and post-visual instruction tuning (0.589 vs 0.539).

    “by comparing the DocLens claim precision scores and the FORTE scores in the status pre- (0.102 vs 0.102) and post-visual instruction tuning (0.589 vs 0.539)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    DocLens is highly correlated with FORTE (r = 0.78, p < 0.001), but poorly correlated with traditional metrics (r < 0.4, p < 0.001).

    “Doclens is highly correlated with FORTE (r = 0.78, p-value < 0.001), but is poorly correlated with traditional metrics (r < 0.4, p-value < 0.001).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Negation removal improved BrainGPT FORTE keyword average F1 score by 0.118 for degrees, 0.194 for landmarks, 0.139 for features, and 0.163 for impressions.

    “negation removal improved the BrainGPT FORTE keyword average F1 score by 0.118 for degrees, 0.194 for landmarks, 0.139 for features, and 0.163 for impressions”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
    • Rests on an assumption: The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

  • May combine two claims.

    Findingcitation

    Traits of 'Familiarity and voice' (29%), 'Specificity or vagueness of details' (37.60%), 'Continuity and Coherence' (33.33%), and 'Writing quality at sentence level' (33.33%) were significant factors when medical experts attribute a human origin behind a radiology report.

    “traits of “Familiarity and voice” (29%), “Specificity or vagueness of details” (37.60%), “Continuity and Coherence” (33.33%), and “Writing quality at sentence level” (33.33%) were significant factors when medical experts attribute a human origin behind a radiology report”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [34] Casal, J. E. & Kessler, M. Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing. Res. Methods Appl. Linguist. 2, 100068 (2023). source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    BrainGPT uses an end-to-end open-source Otter framework (CLIP ViT-L/14 vision encoder and LlaMA-7B), which allows for experimental replication and checkpoint sharing.

    “In contrast, our BrainGPT uses an end-to-end open-source Otter framework (CLIP ViT-L/14 vision encoder39 and LlaMA-7B40), which allows for experimental replication and checkpoint sharing.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [39] Radford, A. et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. (PmLR, 2021).

      Not checked in the free run.

    • cites [40] Touvron, H. et al. LLaMA: open and efficient foundation language models. Preprint at https://ui.adsabs.harvard.edu/abs/2023arXiv230213971T (2023).

      Not checked in the free run.

  • May combine two claims.

    Findingreasoning

    The dataset includes images of normal brains, past infarcts that still show manifestations, chronic brain conditions, and acute brain lesions.

    “Since Alzheimer’s Disease is a progressive degenerative condition predominantly seen in the elderly, the dataset includes images of normal brains, past infarcts that still show manifestations, chronic brain conditions, and acute brain lesions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We collected 18,885 brain CT scans (742,501 slices) from 9689 patients with Alzheimer's Disease (mean age = 82.59 years [standard deviation = 9.3 years]; 56.4% male) at Taipei Veterans General Hospital in Taipei, Taiwan, between January 1, 2010, and December 31, 2022.

    “We collected 18,885 brain CT scans (742,501 slices) from 9689 patients with Alzheimer’s Disease (mean age = 82.59 years [standard deviation = 9.3 years]; 56.4% male) at Taipei Veterans General Hospital in Taipei, Taiwan, between January 1, 2010, and December 31, 2022.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Input case imbalance can influence the caption writing style, potentially related to the over-fitting observed during general model training.

    “Interestingly, we observed that input case imbalance can influence the caption writing style, potentially related to the over-fitting observed during general model training (see also Supplementary Fig. 4).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The CT images included a variety of common neurology conditions affecting the skull, brain parenchyma, nasal sinuses, and the eye, and were collected by radiologists who routinely obtain CT images and write image reports based on the images and the patient's medical records.

    “The CT images included a variety of common neurology conditions affecting the skull, brain parenchyma, nasal sinuses, and the eye, and were collected by radiologists who routinely obtain CT images and write image reports based on the images and the patient’s medical records.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The system was tested on 87,312 slices sampled from 145,166 slices of 3638 CT scans representing 1938 patients.

    “The system was then tested on 87,312 slices sampled from 145,166 slices of 3638 CT scans representing 1938 patients.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    BrainGPT is trained on degeneration-oriented data and thus it fails to capture the malignant tumor and acute traumatic features in CQ500.

    “Second, BrainGPT is trained on degeneration-oriented data and thus it fails to capture the malignant tumor and acute traumatic features in CQ500.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Clinical instruction-tuned BrainGPT models generate brain CT reports with a higher level of radiology term usage and better alignment with original diagnostic reports.

    “These collective results indicate that clinical instruction-tuned BrainGPT models generate brain CT reports with a higher level of radiology term usage and better alignment with original diagnostic reports.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
    • Rests on an assumption: The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    There are no generalized evaluation metrics available to gauge MLLM report information density and fidelity.

    “there are no generalized evaluation metrics available to gauge MLLM report information density and fidelity”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The utmost model capacity was not fully tested in interpreting volumetric scans.

    “the utmost model capacity was not fully tested in interpreting volumetric scans”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The most studied CXR modality lacks sufficient lesion diversity to reflect real-world diagnostic challenges.

    “the most studied CXR modality lacks sufficient lesion diversity to reflect real-world diagnostic challenges”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The Interpretation Spree behavior is detrimental because off-target effects may exclude the primary disease focus and expanded narration may dilute traditional metrics, leading to invalid evaluations.

    “This behavior is detrimental because (1) off-target effects may exclude the primary disease focus (e.g., stroke or brain tumor), and (2) expanded narration may dilute traditional metrics, leading to invalid evaluations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    BrainGPT generated accurate and informative radiologic reports, and FORTE effectively denotes the medical essence within the report.

    “In conclusion, BrainGPT generated accurate and informative radiologic reports, whilst the proposed FORTE effectively denotes the medical essence within the report.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.
    • Rests on an assumption: The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    DocLens mentioned the importance of the evaluation method to discriminate 'there is a focal lesion' from 'there is no focal lesion'.

    “(2) in coherence with the issue raised in our negation removal session, Doclens29 mentioned the importance of the evaluation method to discriminate “there is a focal lesion” from “there is no focal lesion””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] Xie, Y. et al. DocLens: Multi-aspect Fine-grained Medical Text Evaluation. In Proc. 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 649–679 (Association for Computational Linguistics, Bangkok, Thailand, 2024). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The FORTE framework is customizable and transferable across various medical tasks with no focus restrain.

    “Moreover, the FORTE framework is customizable and transferable across various medical tasks with no focus restrain, an interchangeable categorized keyword bank was provided as keyword JSON files on our GitHub page, some examples were demonstrated for chest X-ray, low-dose computed tomography (LDCT), abdomen CT, and Brain CT tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Generative models are competent in 3D chest CT report generation at the state-of-the-art level (BLEU-1 = 46, BLEU-4 = 36.9, METEOR = 29.5, ROUGE-L = 45.9).

    “Recent work by Hamamci et al. revealed that generative models are competent in 3D chest CT report generation at the state-of-the-art (SOTA) level (BLEU-1 = 46, BLEU-4 = 36.9, METEOR = 29.5, ROUGE-L = 45.9)35–37.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [35] Ethem Hamamci, I., Er, S. & Menze, B. CT2Rep: automated radiology report generation for 3D medical imaging. In proc. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. LNCS 15012, 476–486 (Springer Nature, Switzerland, 2024). source

      Not checked in the free run.

    • cites [36] Miura, Y., Zhang, Y., Tsai, E. B., Langlotz, C. P. & Jurafsky, D. Improving factual completeness and consistency of image-to-text radiology report generation. In Proc. 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. (eds Toutanova, K. et al.) 5288–5304 (Association for Computational Linguistics, 2021). source

      Not checked in the free run.

    • cites [37] Nicolson, A., Dowling, J. & Koopman, B. Improving chest X-ray report generation by leveraging warm starting. Artif. Intell. Med. 144, 102633 (2023). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Every recent LLM-based evaluation study aims to establish a more clinically precise, explainable, and generalizable metric.

    “(3) most importantly, ranging from sentence-level precision and recall to error counting, every study aims to establish a more clinically precise, explainable, and generalizable metric.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [29] Xie, Y. et al. DocLens: Multi-aspect Fine-grained Medical Text Evaluation. In Proc. 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 649–679 (Association for Computational Linguistics, Bangkok, Thailand, 2024). source

      Not checked in the free run.

    • cites [42] Huang, A., Banerjee, O., Wu, K., Pontes Reis, E., Rajpurkar, P. FineRadScore: a radiology report line-by-line evaluation technique generating corrections with severity scores. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240520613H (2024).

      Not checked in the free run.

    • cites [43] Zambrano Chaves, J. M., et al. Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240308002Z (2024).

      Not checked in the free run.

    • cites [44] Bannur, S., et al. MAIRA-2: Grounded radiology report generation. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240604449B (2024).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to distinguish synthesized clinical reports from human reports.

    “Therefore, related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that distinguish synthesized clinical reports from human reports46–49.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [46] Van Veen, D., et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30, 1134–1142 (2024). source

      Not checked in the free run.

    • cites [47] Yan, B. et al. Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023. 14676–14688 (Association for Computational Linguistics, Singapore, 2023). source

      Not checked in the free run.

    • cites [48] Boag W., Kané H., Rawat S., Wei J. & Goehler A. A Pilot Study in Surveying Clinical Judgments to Evaluate Radiology Report Generation. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery, 2021). source

      Not checked in the free run.

    • cites [49] Youssef, A. et al. External validation of AI models in health should be replaced with recurring local validation. Nat. Med. 29, 2686–2687 (2023). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The study assembled a holistic framework to address unmet needs and achieve robust report generation for 3D brain CT.

    “In this study, we assembled a holistic framework to address various unmet needs and achieve robust report generation for 3D brain CT.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: There are unmet needs in 3D brain CT report generation.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The high computational cost of using large-scale Google TPUv4 accelerator pods makes this approach infeasible for general research with limited resources.

    “The high computational cost of using large-scale Google TPUv4 accelerator pods makes this approach infeasible for general research with limited resources.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Google AI's Med-Gemini-3D can perform 3D CT report generation, but only 53% of its reports are considered clinically valid in human evaluations.

    “Additionally, Google AI’s Med-Gemini-3D can perform 3D CT report generation, but only 53% of its reports are considered clinically valid in human evaluations38.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38] Yang, L. et al. Advancing multimodal medical capabilities of Gemini. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240503162Y (2024).

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The reduced training cost of BrainGPT enables efficient visual instruction tuning, enhancing model performance and tailoring responses to specialized or stylistic requirements.

    “Moreover, the reduced training cost of BrainGPT enables efficient visual instruction tuning, enhancing model performance and tailoring responses to specialized or stylistic requirements (Supplementary Table 5).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Negation removal allowed evaluation metrics to focus on positive findings, enhanced average traditional scores and FORTE F1 scores, and avoided sparse and off-topic impressions in reports.

    “Together, we demonstrated that negation removal allowed evaluation metrics to focus on positive findings, enhanced average traditional scores and FORTE F1 scores, and avoided sparse and off-topic impressions in reports.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The value of key radiology descriptors outweighs grammatical filler words.

  • May combine two claims.

    Interpretationnone given

    FORTE served as an evaluation framework that consists of sentence-pairing, negation removal, and 4-category keyword extraction, which coherently restricts model hallucination, enhances the interpretation convergence, and provides instant radiology impressions for healthcare providers.

    “To this end, FORTE served as an evaluation framework that consists of sentence-pairing, negation removal, and 4-category keyword extraction, which coherently restricts model hallucination, enhances the interpretation convergence, and provides instant radiology impressions for healthcare providers.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    A 10-point Likert scale rating by 2 physicians reading 50 random sampled captions was constructed.

    “we constructed a 10-point Likert scale rating by 2 physicians reading 50 random sampled captions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Traditional metrics fall short of addressing semantic complexity in medical reports, and FORTE scoring works as efficiently as state-of-the-art DocLens evaluation but in a computation-light manner.

    “our results demonstrated that traditional metrics fall short of addressing the semantic complexity in medical reports, and the FORTE scoring concept work as efficient as the state-of-the-art DocLens evaluation but in a computation-light manner.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Medical image reports assist in differential diagnosis and are characterized by complex paraphrasing, high token counts (>100), and numerous negative descriptions, which conflict with common metric evaluation contexts.

    “Medical image reports assist in differential diagnosis and thus are characterized by complex paraphrasing, high token counts (>100), and numerous negative descriptions, which conflict with common metric evaluation contexts24,41.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [24] Liu, G. et al. Clinically Accurate Chest X-Ray Report Generation. In Proc. 4th Machine Learning for Healthcare Conference. 106, 249–269 (PMLR, 2019).

      Not checked in the free run.

    • cites [41] Abbasian, M. et al. Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI. NPJ Digital Med. 7, 82 (2024). source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The physician survey was composed of six caption pairs, each comprising a BrainGPT report and a radiologist report, including diverse disease instances such as lacunar infarct, subdural hemorrhage, brain atrophy, and midline (cerebral falx) shift.

    “The physician survey was composed of six caption pairs, each comprising a BrainGPT report and a radiologist report. These examples included diverse disease instances including lacunar infarct, subdural hemorrhage, brain atrophy, and midline (cerebral falx) shift, thereby encompassing a range of both acute and chronic cerebral alterations for expert evaluation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    BERT embeddings capture rich semantic information and are widely recognized as effective for sentence similarity tasks.

    “Previous literature showed the BERT embeddings54 capture rich semantic information and are widely recognized as effective for sentence similarity tasks55–57.”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [54] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, Vol 1 (long and short papers) (Association for Computational Linguistics, 2019).

      Not checked in the free run.

    • cites [55] Feng, F., Yang, Y., Cer, D., Arivazhagan, N. & Wang, W. Language-agnostic BERT Sentence Embedding. In Proc. 60th Annual Meeting of the Association for Computational Linguistics, Vol 1 (Long Papers), 878–891. (Association for Computational Linguistics, Dublin, Ireland, 2022) source

      Not checked in the free run.

    • cites [56] Li, B. et al. On the Sentence Embeddings from Pre-trained Language Models. In Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 9119–9130 Online. (Association for Computational Linguistics, 2020). source

      Not checked in the free run.

    • cites [57] Ethayarajh, K. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proc. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 55–65 (Association for Computational Linguistics, Hong Kong, China, 2019). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The significance of the medical report writing style was also highlighted in an independent prompting study.

    “The significance of the medical report writing style was also highlighted in an independent prompting study47.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] Yan, B. et al. Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023. 14676–14688 (Association for Computational Linguistics, Singapore, 2023). source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Sentence pairing releases the sequential constraints of disease descriptors, and negation removal reduces the false positives in evaluation reports.

    “By these means, sentence pairing releases the sequential constraints of disease descriptors, and negation removal reduces the false positives in evaluation reports.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    We averaged the reports for each reporter, yielding 27 reporter averages, to minimize the influence of reporter writing style.

    “To minimize the influence of reporter writing style (as demonstrated in Supplementary Fig. 4), we averaged the reports for each reporter, yielding 27 reporter averages.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    This is a pilot volumetric brain CT captioning study that has no counterpart MLLM module to benchmark, and hence we cannot justify the efficacy at the SOTA level.

    “First, this is a pilot volumetric brain CT captioning study that has no counterpart MLLM module to benchmark, and hence we cannot justify the efficacy at the SOTA level; however, we applied external validation to ensure the caption validity in the brain CT module.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    We utilized the DocLens metric, which employs GPT-4o mini to extract claims from generated reports, verify their presence in ground-truth reports, and compute a precision score for the generated content.

    “For the LLM-as-a-judge evaluation, we utilized the DocLens metric, which employs GPT-4o mini to extract claims from generated reports, verify their presence in ground-truth reports, and compute a precision score for the generated content.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [29] Xie, Y. et al. DocLens: Multi-aspect Fine-grained Medical Text Evaluation. In Proc. 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 649–679 (Association for Computational Linguistics, Bangkok, Thailand, 2024). source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Ground truth was based on a read.csv file from the CQ500 dataset, and the majority rule was applied among the three raters to summarize the 'Mass effect,' 'Hemorrhage event,' and 'Midline shift' labels.

    “Ground truth was based on a read.csv file from the CQ500 dataset, and the majority rule was applied among the three raters to summarize the “Mass effect,” “Hemorrhage event,” and “Midline shift” labels.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Intraclass correlation coefficients (ICCs) were calculated to measure the reliability of human evaluations across these criteria.

    “Intraclass correlation coefficients (ICCs) were calculated to measure the reliability of human evaluations across these criteria.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    RADPEER scores were dichotomized using arithmetic means as thresholds to compute Cohen's kappa coefficients with DocLens scores.

    “Additionally, RADPEER scores were dichotomized using arithmetic means as thresholds to compute Cohen’s kappa coefficients with DocLens scores.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    We provided the FORTE keyword JSON files for the Brain CT, chest X-ray, low-dose computed tomography (LDCT), and abdomen CT on our project GitHub page to demonstrate the generalizability of FORTE in facilitating the next generation of medical LLM.

    “We provided the FORTE keyword JSON files for the Brain CT, chest X-ray, low-dose computed tomography (LDCT), and abdomen CT on our project GitHub page to demonstrate the generalizability of FORTE in facilitating the next generation of medical LLM.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    We proposed the feature-oriented radiology task evaluation (FORTE) framework to capture radiology keywords in terms of Degree (size and intensity), Landmark (location and anatomy), Feature (disease traits and primary findings), and Impression (final diagnosis) of the disease.

    “We proposed the feature-oriented radiology task evaluation (FORTE) framework to capture radiology keywords in terms of Degree (size and intensity), Landmark (location and anatomy), Feature (disease traits and primary findings), and Impression (final diagnosis) of the disease.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The FORTE keyword categories (Degree, Landmark, Feature, Impression) adequately capture the clinical essence of radiology reports.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Informed consent was exempted due to the retrospective nature of the data collection and the use of deidentified CT images.

    “Informed consent was exempted due to the retrospective nature of the data collection and the use of deidentified CT images.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Cosine similarity is widely recognized as an effective generation metric and applied in the CheXbert vector similarity metric.

    “Cosine similarity, which measures the token-level contextual embeddings of reference and candidate sentences, is widely recognized as an effective generation metric and applied in the CheXbert vector similarity metric58.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [58] Yu, F. et al. Evaluating progress in automatic chest X-ray radiology report generation. Patterns 4, 100802 (2023). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductreasoning

    This ensures that slice thickness and details are similar to our training dataset and fit in the Otter framework.

    “This ensures that slice thickness and details are similar to our training dataset and fit in the Otter framework.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The FORTE keyword list includes synonyms, allowing the system to recognize a broader array of related terms that may appear in different contexts, addressing the challenge of lexical variability inherent in clinical reports.

    “This list is not just a compilation of keywords but also includes synonyms, allowing the system to recognize a broader array of related terms that may appear in different contexts, addressing the challenge of lexical variability inherent in clinical reports.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All data were collected under institutional review board approval (2023-10-002 BC) of the Taipei Veterans General Hospital.

    “All data were collected under institutional review board approval (2023-10-002 BC) of the Taipei Veterans General Hospital.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The 'sentence pairing' process was inspired by BERTScore, which leverages contextual embeddings and cosine similarity to achieve better semantic matching than traditional n-gram-based methods.

    “The “sentence pairing” process was inspired by BERTScore, which leverages contextual embeddings and cosine similarity to achieve better semantic matching than traditional n-gram-based methods53.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:190409675, (2019).

      Not checked in the free run.

66 of 66

  • The paper relies on this without support.

    The utility of MLLM-based radiology report generation for 3D medical images remains largely unexplored.

    “While 2D MLLM-based RRG has been well established, its utility for 3D medical images remains largely unexplored.”

  • The paper relies on this without support.

    Current single-slice CT image-text data is manually pre-selected from 3D data, so MLLMs are largely tested on definite-positive lesion-included images.

    “current single-slice CT image-text data is manually pre-selected from a 3D data, thus the MLLM is largely tested under a definite-positive lesion-included image.”

  • The paper relies on this without support.

    Testing MLLMs on definite-positive lesion-included images may result in a sharpshooter fallacy.

    “This may result in a sharpshooter fallacy in the MLLM testing.”

  • The paper relies on this without support.

    The value of key radiology descriptors outweighs grammatical filler words.

    “Therefore, the value of key radiology descriptors outweighs grammatical filler words.”

  • The paper relies on this without support.

    The brain lesion’s degree, size, and location are paramount for making precise diagnoses and guiding subsequent clinical decisions.

    “The brain lesion’s degree, size, and location of a brain lesion are paramount for making forming precise diagnoses and guiding subsequent clinical decisions.”

  • The paper relies on this without support.

    There are unmet needs in 3D brain CT report generation.

    “In this study, we assembled a holistic framework to address various unmet needs and achieve robust report generation for 3D brain CT.”

  • The paper relies on this without support.

    The FORTE keyword categories (Degree, Landmark, Feature, Impression) adequately capture the clinical essence of radiology reports.

    “We proposed the feature-oriented radiology task evaluation (FORTE) framework to capture radiology keywords in terms of Degree (size and intensity), Landmark (location and anatomy), Feature (disease traits and primary findings), and Impression (final diagnosis) of the disease.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Specialist agents such as BrainGPT for 3D CT scans, HistoGPT for histopathology images, and scChat for single-cell sequencing data can be easily integrated into the MoMA workflow.

    “For example, specialist agents such as BrainGPT66 for 3D CT scans, HistoGPT67 for histopathology images, and scChat68 for single-cell sequencing data can be easily integrated into the MoMA workflow when these modalities are involved.”

    From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [66]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Turing test used six versus seven report pairs

    Minor

    The Figure 7 caption says physicians evaluated seven report pairs, while the Methods section says the survey contained six caption pairs. The reported 66 evaluations also matches 11 physicians evaluating six pairs, not seven.

1 of 1

Cites

Flagged first, then by citations

6 of the 59 references couldn’t be matched to a known paper.

Research works printed but not linked: 11

  1. Boag, W. et al. Baselines for chest x-ray report generation. In: Machine learning for health workshop) (PMLR, 2020).
  2. Liu, G. et al. Clinically Accurate Chest X-Ray Report Generation. In Proc. 4th Machine Learning for Healthcare Conference. 106, 249–269 (PMLR, 2019).
  3. Liu, H., Li, C., Wu, Q. & Lee, Y. J. Visual instruction tuning. Adv. Neural. Inf. Process. Syst. 36, 34892–34916 (2023).
  4. Wang, Z., Luo, X., Jiang, X., Li, D. & Qiu, L. LLM-RadJudge: achieving radiologist-level evaluation for X-Ray report generation. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240400998W (2024).
  5. Huang, A., Banerjee, O., Wu, K., Pontes Reis, E., Rajpurkar, P. FineRadScore: a radiology report line-by-line evaluation technique generating corrections with severity scores. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240520613H (2024).
  6. Zambrano Chaves, J. M., et al. Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240308002Z (2024).
  7. Bannur, S., et al. MAIRA-2: Grounded radiology report generation. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240604449B (2024).
  8. Liu, J., et al. A Spectrum evaluation benchmark for medical multi-modal large language models. Preprint at https://ui.adsabs.harvard.edu/abs/2024arXiv240211217L (2024).
  9. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:190409675, (2019).
  10. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, Vol 1 (long and short papers) (Association for Computational Linguistics, 2019).
  11. Jackson, V. P. et al. RADPEER™ scoring white paper. J. Am. Coll. Radiol. 6, 21–25 (2009). · DOI

Cited by

Sorted by citations

Related

Authors
Li, C.-Y., Chang, K.-J., Yang, C.-F., Wu, H.-Y., Chen, W., Bansal, H., Chen, L., Yang, Y.-P., Chen, Y.-C., Chen, S.-P., Chen, S.-J., Lirng, J.-F., Chang, K.-W., & Chiou, S.-H.
Affiliations
  • National Yang Ming Chiao Tung University (TW)
  • University of California, Los Angeles (US)
  • City University of Hong Kong (HK)
Corresponding author
  • Chang, Kao-Jung
  • Chang, Kai-Wei
  • Chiou, Shih-Hwa
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2025-03-06
Volume
16
Issue
1
Pages
2258-2258
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-025-57426-0
PMC
PMC11885477
OpenAlex
W4408190450
Semantic Scholar
04e60879e76d682d161cd0365f8db55b5d2c210c
Cited by
75
References
59
Citations by year
2026: 48 · 2025: 26 · 2024: 1
FWCI
25.98
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • CityU Scholars (submittedVersion, cc-by-nc-nd)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
  • U.S. Department of Veterans Affairs
  • Shanghai Educational Development Foundation
  • Taipei Veterans General Hospital
  • National Science and Technology Council (NSTC 112-2321-B-A49-007)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
Keywords
multimodal large language models, radiology report generation, clinical text generation
MeSH terms
Brain, Humans, Tomography, X-Ray Computed, Imaging, Three-Dimensional

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review