The DOI matches its Crossref record; 29 of the 30 references it cites were found in a registry or an index, and 1 are not checked yet.
Performance of an Open-Source Large Language Model in Extracting Information from Free-Text Radiology Reports
Open accessAbstract
“Just Accepted” papers have undergone full peer review and have been accepted for publication in Radiology: Artificial Intelligence. This article will undergo copyediting, layout, and proof review before it is published in its final version. Please note that during production of the final copyedited article, errors may be discovered which could affect the content. Purpose To assess the performance of a local open-source large language model (LLM) on various information extraction tasks from real-life emergency brain MRI reports. Materials and Methods All consecutive emergency brain MRI reports written in 2022 from a French quaternary center were retrospectively reviewed. Two radiologists identified MRIs that were performed for headaches. Four radiologists scored reports' conclusions as normal or abnormal. Abnormalities were labeled as either headache-causing or incidental. Vicuna, an open-source LLM, performed the same tasks. Vicuna’s performance metrics were evaluated using the radiologists’ consensus as the reference standard. Results Among the 2398 reports during the study period, radiologists identified 595 that included headaches in their indication (median age of patients, 35 years [IQR, 26–51], 68% (403/595) female). A positive finding was reported in 227/595 (38%) cases, 136 of which could explain the headache. The LLM had a sensitivity/specificity (95%CI), respectively, of 98% (583/595)(97–99)/99% (1791/1803)(99–100) for detecting the presence of headache in the clinical context, 99% (514/517)(98–100)/99% (68/69)(92–100) for the use of contrast medium injection, 97% (219/227)(93–99)/99% (364/368)(97–100) for study categorization as normal or abnormal and 88% (120/136)(82– 93)/73% (66/91)(62–81) for causal inference between MRI findings and headache. Conclusion An open-source LLM was able to extract information from free-text radiology reports with excellent accuracy without requiring further training. ©RSNA, 2024
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Most studies applying LLM on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data [5, 6, 13, 14, 17–20].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Buderer, N. M. F. (1996) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.1996.tb03538.x · Cited by 912
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
Natural language processing in radiology: a systematic review
Pons, E., Braun, L. M. M., Hunink, M. G. M., & Kors, J. A. (2016) · Radiology · doi:10.1148/radiol.16142770 · Cited by 573
Bhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460
Emergent analogical reasoning in large language models
Webb, T., Holyoak, K. J., & Lu, H. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01659-w · Cited by 404
Adams, L. C., Truhn, D., Busch, F., Kader, A., Niehues, S. M., Makowski, M. R., & Bressem, K. K. (2023) · Radiology · doi:10.1148/radiol.230725 · Cited by 339
Ethics of large language models in medicine and medical research
Li, H., Moon, J. T., Purkayastha, S., Celi, L. A., Trivedi, H., & Gichoya, J. W. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00083-3 · Cited by 286
Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer
Fink, M. A., Bischoff, A., Fink, C. A., Moll, M., Kroschke, J., Dulz, L., Heußel, C. P., Kauczor, H., & Weber, T. F. (2023) · Radiology · doi:10.1148/radiol.231362 · Cited by 284
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177
AI Chatbots, Health Privacy, and Challenges to HIPAA Compliance
Marks, M., & Haupt, C. E. (2023) · JAMA · doi:10.1001/jama.2023.9458 · Cited by 174
Mukherjee, P., Hou, B., Lanfredi, R. B., & Summers, R. M. (2023) · Radiology · doi:10.1148/radiol.231147 · Cited by 117
Artificial Intelligence in Clinical Diagnosis: Opportunities, Challenges, and Hype
Kulkarni, P. A., & Singh, H. (2023) · JAMA · doi:10.1001/jama.2023.11440 · Cited by 112
Why open-source generative AI models are an ethical way forward for science
Spirling, A. (2023) · Nature · doi:10.1038/d41586-023-01295-4 · Cited by 104
Accuracy of ChatGPT, Google Bard, and Microsoft Bing for Simplifying Radiology Reports
Amin, K., Davis, M. A., Doshi, R. H., Haims, A. H., Khosla, P., & Forman, H. P. (2023) · Radiology · doi:10.1148/radiol.232561 · Cited by 104
A Systematic Review of Interventions to Reduce Computed Tomography Usage in the Emergency Department
Dunne, C. L., Elzinga, J. L., Vorobeichik, A., Sudershan, S., Keto‐Lambert, D., Lang, E., & Dowling, S. K. (2022) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.06.001 · Cited by 61
Langlotz, C. P. (2002) · Radiology · doi:10.1148/radiol.2241020415 · Cited by 57
The Future of AI and Informatics in Radiology: 10 Predictions
Langlotz, C. P. (2023) · Radiology · doi:10.1148/radiol.231114 · Cited by 56
Radiology in the era of large language models: the near and the dark side of the moon
Úbeda, P. L., Martín‐Noguerol, T., & Luna, A. (2023) · European Radiology · doi:10.1007/s00330-023-09901-9 · Cited by 27
Diagnostic Neuroimaging in Headache Patients: A Systematic Review and Meta-Analysis
Jang, Y. E., Cho, E. Y., Choi, H. Y., Kim, S. M., & Park, H. Y. (2019) · Psychiatry Investigation · doi:10.30773/pi.2019.04.11 · Cited by 24
Trofimova, A. V., Duszak, R., Kadom, N., & Sadigh, G. (2020) · Headache The Journal of Head and Face Pain · doi:10.1111/head.14020 · Cited by 14
Jupin-Delevaux, É., Djahnine, A., Talbot, F., Richard, A., Gouttard, S., Mansuy, A., Douek, P., Si-Mohamed, S. A., & Boussel, L. (2023) · Research in Diagnostic and Interventional Imaging · doi:10.1016/j.redii.2023.100027 · Cited by 13
Lungren, M. P., Amrhein, T. J., Paxton, B. E., Srinivasan, R. C., Collins, H. R., Eastwood, J. D., & Kilani, R. K. (2013) · Radiology · doi:10.1148/radiol.13130281 · Cited by 13
Evaluating Large Language Models in Extracting Cognitive Exam Dates and ScoresPreprint
Zhang, H., Jethani, N., Jones, S., et al. (2023) · medRxiv · doi:10.1101/2023.07.10.23292373 · Cited by 13
Diagnostic yield of emergency MRI in non-traumatic headache
Happonen, T., Nyman, M. J., Ylikotila, P., Merisaari, H., Mattila, K., & Hirvonen, J. (2022) · Neuroradiology · doi:10.1007/s00234-022-03044-2 · Cited by 11
Factors associated with significant MRI findings in medical walk-in patients with acute headache
Budweg, J., Sprenger, T., Vere-Tyndall, A. D., Hagenkord, A., Stippich, C., & Berger, C. T. (2016) · Swiss Medical Weekly · doi:10.4414/smw.2016.14349 · Cited by 10
Transforming primary healthcare through natural language processing and big data analytics
Uddin, Y., Nair, A., Shariq, S., & Hannan, S. H. (2023) · BMJ · doi:10.1136/bmj.p948 · Cited by 5
Sebro, R., & Kahn, C. E. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad119 · Cited by 2
Flagged first, then by citations
Research works printed but not linked: 1
- Budweg J, 2016, Swiss Med Wkly, 146, w14349
Cited by
Sorted by citations
Large Language Models and Large Multimodal Models in Medical Imaging: A Primer for Physicians
Bradshaw, T., Tie, X., Warner, J., Hu, J., Li, Q., & Li, X. (2025) · Journal of Nuclear Medicine · doi:10.2967/jnumed.124.268072 · Cited by 60
Comprehensive testing of large language models for extraction of structured data in pathology
Grothey, B., Odenkirchen, J., Brkic, A., Schömig-Markiefka, B., Quaas, A., Büttner, R., & Tolkach, Y. (2025) · Communications Medicine · doi:10.1038/s43856-025-00808-8 · Cited by 54
Salam, B., Stüwe, C., Nowak, S., Sprinkart, A. M., Theis, M., Kravchenko, D., Mesropyan, N., Dell, T., Endler, C., Pieper, C. C., et al. (2025) · European Radiology · doi:10.1007/s00330-025-11438-y · Cited by 43
Benchmarking the diagnostic performance of open source LLMs in 1933 Eurorad case reports
Kim, S. H., Schramm, S., Adams, L. C., Braren, R., Bressem, K. K., Keicher, M., Platzek, P., Paprottka, K. J., Zimmer, C., Hedderich, D. M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01488-3 · Cited by 44
Lee, J. E., Park, K. S., Kim, Y., Song, H.-C., Park, B., & Jeong, Y. J. (2024) · American Journal of Roentgenology · doi:10.2214/ajr.24.31696 · Cited by 41
Metadata
- Authors
- Le Guellec, B., Lefèvre, A., Geay, C., Shorten, L., Bruge, C., Hacein-Bey, L., Amouyel, P., Pruvo, J.-P., Kuchcinski, G., & Hamroun, A.
- Affiliations
- Inserm (FR)
- ORCID
- Venue
- Radiology: Artificial Intelligence
- Published
- 2024-07-01
- Volume
- 6
- Issue
- 4
- Pages
- e230364-e230364
- ISSN
- 2638-6100 (ISSN-L)
- PMC
- PMC11294959
- OpenAlex
- W4396722287
- Cited by
- 115
- References
- 28
- Citations by year
- 2026: 42 · 2025: 60 · 2024: 12
- FWCI
- 10.66
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- cc-by
- Locations
- Radiology Artificial Intelligence (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- LillOA (Université de Lille (University Of Lille)) (submittedVersion)
- LillOA (Université de Lille (University Of Lille)) (submittedVersion, other-oa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- open-source large language models, vicuña, headache, contrast medium injection, incidental finding, radiologist consensus, local LLM deployment, radiology education, radiology workforce shortage
- MeSH terms
- Adult, Brain, Emergency Service, Hospital, Female, Headache, Humans, Magnetic Resonance Imaging, Male, Middle Aged, Natural Language Processing, Retrospective Studies, Sensitivity and Specificity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
