The DOI matches its Crossref record; 29 of the 30 references it cites were found in a registry or an index, and 1 are not checked yet.
Performance of an Open-Source Large Language Model in Extracting Information from Free-Text Radiology Reports
Open accessAbstract
“Just Accepted” papers have undergone full peer review and have been accepted for publication in Radiology: Artificial Intelligence. This article will undergo copyediting, layout, and proof review before it is published in its final version. Please note that during production of the final copyedited article, errors may be discovered which could affect the content. Purpose To assess the performance of a local open-source large language model (LLM) on various information extraction tasks from real-life emergency brain MRI reports. Materials and Methods All consecutive emergency brain MRI reports written in 2022 from a French quaternary center were retrospectively reviewed. Two radiologists identified MRIs that were performed for headaches. Four radiologists scored reports' conclusions as normal or abnormal. Abnormalities were labeled as either headache-causing or incidental. Vicuna, an open-source LLM, performed the same tasks. Vicuna’s performance metrics were evaluated using the radiologists’ consensus as the reference standard. Results Among the 2398 reports during the study period, radiologists identified 595 that included headaches in their indication (median age of patients, 35 years [IQR, 26–51], 68% (403/595) female). A positive finding was reported in 227/595 (38%) cases, 136 of which could explain the headache. The LLM had a sensitivity/specificity (95%CI), respectively, of 98% (583/595)(97–99)/99% (1791/1803)(99–100) for detecting the presence of headache in the clinical context, 99% (514/517)(98–100)/99% (68/69)(92–100) for the use of contrast medium injection, 97% (219/227)(93–99)/99% (364/368)(97–100) for study categorization as normal or abnormal and 88% (120/136)(82– 93)/73% (66/91)(62–81) for causal inference between MRI findings and headache. Conclusion An open-source LLM was able to extract information from free-text radiology reports with excellent accuracy without requiring further training. ©RSNA, 2024
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Most studies applying LLM on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data [5, 6, 13, 14, 17–20].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Buderer, N. M. F. (1996) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.1996.tb03538.x · Cited by 912
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
Natural language processing in radiology: a systematic review
Pons, E., Braun, L. M. M., Hunink, M. G. M., & Kors, J. A. (2016) · Radiology · doi:10.1148/radiol.16142770 · Cited by 573
Bhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460
Emergent analogical reasoning in large language models
Webb, T., Holyoak, K. J., & Lu, H. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01659-w · Cited by 404
Adams, L. C., Truhn, D., Busch, F., Kader, A., Niehues, S. M., Makowski, M. R., & Bressem, K. K. (2023) · Radiology · doi:10.1148/radiol.230725 · Cited by 339
Ethics of large language models in medicine and medical research
Li, H., Moon, J. T., Purkayastha, S., Celi, L. A., Trivedi, H., & Gichoya, J. W. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00083-3 · Cited by 286
Flagged first, then by citations
Research works printed but not linked: 1
- Budweg J, 2016, Swiss Med Wkly, 146, w14349
Cited by
Sorted by citations
Items 101–115 of 115
Maline, G. E., Neshatvar, Y., Chandarana, H., Ranganath, R., & Nayan, M. (2026) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2026.100114
Irons, J. L., Cooper, P. S., Bölücü, N., Duenser, A., Timmer, R. C., Yang, H., Lee, C., Jin, B., & Wan, S. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820430
Meşe, İ. (2026) · Journal of Health Sciences and Medicine · doi:10.32322/jhsm.1918471
Guellec, B. L., Bentegeac, R., Tran, V.-T., Homsi, M. E., Amouyel, P., Kuchcinski, G., & HAMROUN, A. (2026) · medRxiv · doi:10.64898/2026.08.05.26359773
Self-citationAnalysis of Large Language Models for Magnetic Resonance Imaging Brain Tumor Segmentation Reporting
Vennila, M., & Kalaiselvi, T. (2026) · Cureus Journal of Computer Science. · doi:10.7759/s44389-026-00225-5
Cavallo, A. U., Midolo, E., D'Anna, G., Fujiwara, M., Arita, Y., Girometti, R., & Cuocolo, R. (2026) · Abdominal Radiology · doi:10.1007/s00261-026-05719-3
Mar, N. B., Duigou, M., Seizeur, R., & Dissaux, B. (2026) · Surgical and Radiologic Anatomy · doi:10.1007/s00276-026-03957-0
Zhang, J., Wang, X., Zhao, Y., Cui, X., Zhao, L., Zhao, X., Zhang, H., & Zhu, Z. (2026) · Journal of Medical Internet Research · doi:10.2196/94904
Erdağı, E. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1919939
Sabu, S., Howard, R., Mahathanthila, N. Y., Jones, K., Turriff, L., Sheldon, E. J., Eze, A., Key, D. A., Patel, S., Chesover, A., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37685-5_42
Haidar, A. I., emam, M., Aljubran, A. A., Alkhudhair, K. M., Alassiri, M. M., Almutairi, B. S., Asulaiman, S. A., Altamimi, F. I., Alkahtani, M. M., Alyami, I. A., et al. (2026) · Diagnostics · doi:10.3390/diagnostics16182951
Ouchi, D., Riudavets, S., Franquet, Á., Fernández-García, S., Llor, C., Moragas, A., Giner‐Soriano, M., Torres, F., & Morros, R. (2026) · medRxiv · doi:10.64898/2026.09.16.26363030
Turgan, A. H., Çolakoğlu, H., Demir, A., & Sakçı, Z. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02427-8
Awawdeh¹, M., & Awawdeh², I. (2026) · The Egyptian Journal of Radiology and Nuclear Medicine · doi:10.1186/s43055-026-01898-7
Large Language Model–Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation
Wang, L., Zhang, Y., Peng, L., Liu, P., Wang, L., Cui, Y., Chen, Y., Li, X., Tao, Y., Gong, X., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/106250
Metadata
- Authors
- Le Guellec, B., Lefèvre, A., Geay, C., Shorten, L., Bruge, C., Hacein-Bey, L., Amouyel, P., Pruvo, J.-P., Kuchcinski, G., & Hamroun, A.
- Affiliations
- Inserm (FR)
- ORCID
- Venue
- Radiology: Artificial Intelligence
- Published
- 2024-07-01
- Volume
- 6
- Issue
- 4
- Pages
- e230364-e230364
- ISSN
- 2638-6100 (ISSN-L)
- PMC
- PMC11294959
- OpenAlex
- W4396722287
- Cited by
- 115
- References
- 28
- Citations by year
- 2026: 42 · 2025: 60 · 2024: 12
- FWCI
- 10.66
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- cc-by
- Locations
- Radiology Artificial Intelligence (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- LillOA (Université de Lille (University Of Lille)) (submittedVersion)
- LillOA (Université de Lille (University Of Lille)) (submittedVersion, other-oa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- open-source large language models, vicuña, headache, contrast medium injection, incidental finding, radiologist consensus, local LLM deployment, radiology education, radiology workforce shortage
- MeSH terms
- Adult, Brain, Emergency Service, Hospital, Female, Headache, Humans, Magnetic Resonance Imaging, Male, Middle Aged, Natural Language Processing, Retrospective Studies, Sensitivity and Specificity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
