Evaluating local open-source large language models for data extraction from unstructured reports on mechanical thrombectomy in patients with ischemic stroke
Open accessAbstract
BACKGROUND: A study was undertaken to assess the effectiveness of open-source large language models (LLMs) in extracting clinical data from unstructured mechanical thrombectomy reports in patients with ischemic stroke caused by a vessel occlusion. METHODS: We deployed local open-source LLMs to extract data points from free-text procedural reports in patients who underwent mechanical thrombectomy between September 2020 and June 2023 in our institution. The external dataset was obtained from a second university hospital and comprised consecutive cases treated between September 2023 and March 2024. Ground truth labeling was facilitated by a human-in-the-loop (HITL) approach, with time metrics recorded for both automated and manual data extractions. We tested three models-Mixtral, Qwen, and BioMistral-assessing their performance on precision, recall, and F1 score across 15 clinical categories such as National Institute of Health Stroke Scale (NIHSS) scores, occluded vessels, and medication details. RESULTS: The study included 1000 consecutive reports from our primary institution and 50 reports from a secondary institution. Mixtral showed the highest precision, achieving 0.99 for first series time extraction and 0.69 for occluded vessel identification within the internal dataset. In the external dataset, precision ranged from 1.00 for NIHSS scores to 0.70 for occluded vessels. Qwen showed moderate precision with a high of 0.85 for NIHSS scores and a low of 0.28 for occluded vessels. BioMistral had the broadest range of precision, from 0.81 for first series times to 0.14 for medication details. The HITL approach yielded an average time savings of 65.6% per case, with variations from 45.95% to 79.56%. CONCLUSION: This study highlights the potential of using LLMs for automated clinical data extraction from medical reports. Incorporating HITL annotations enhances precision and also ensures the reliability of the extracted data. This methodology presents a scalable privacy-preserving option that can significantly support clinical documentation and research endeavors.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs may curtail the resources required for data extraction, which is traditionally a labor- and time-intensive process.
“Moreover, LLMs may curtail the resources required for data extraction, which is traditionally a labor- and time-intensive process [35].”
From Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review (2025) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01925-4 · Cited by 1,120 · see citing works
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855 · see citing works
Structured information extraction from scientific text with large language models
Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A. S., Ceder, G., Persson, K. A., & Jain, A. (2024) · Nature Communications · doi:10.1038/s41467-024-45563-x · Cited by 771 · see citing works
Sarraju, A., Bruemmer, D., Iterson, E. H. V., Cho, L., Rodríguez, F., & Laffin, L. J. (2023) · JAMA · doi:10.1001/jama.2023.1044 · Cited by 455 · see citing works
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452 · see citing works
ChatGPT in Healthcare: A Taxonomy and Systematic ReviewPreprint
Li, J., Dada, A., Puladi, B., et al. (2023) · medRxiv · doi:10.1101/2023.03.30.23287899 · Cited by 411 · see citing works
Adams, L. C., Truhn, D., Busch, F., Kader, A., Niehues, S. M., Makowski, M. R., & Bressem, K. K. (2023) · Radiology · doi:10.1148/radiol.230725 · Cited by 339 · see citing works
Self-citationPMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300 · see citing works
Using AI to write scholarly publications
Hosseini, M., Rasmussen, L. M., & Resnik, D. B. (2024) · Accountability in Research · doi:10.1080/08989621.2023.2168535 · Cited by 286 · see citing works
Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT
Haver, H. L., Ambinder, E. B., Bahl, M., Oluyemi, E. T., Jeudy, J., & Yi, P. H. (2023) · Radiology · doi:10.1148/radiol.230424 · Cited by 247 · see citing works
Potential Use Cases for ChatGPT in Radiology Reporting
Elkassem, A. M. A., & Smith, A. D. (2023) · American Journal of Roentgenology · doi:10.2214/ajr.23.29198 · Cited by 134 · see citing works
Le Guellec, B., Lefèvre, A., Geay, C., Shorten, L., Bruge, C., Hacein-Bey, L., Amouyel, P., Pruvo, J.-P., Kuchcinski, G., & Hamroun, A. (2024) · Radiology: Artificial Intelligence · doi:10.1148/ryai.230364 · Cited by 115 · see citing works
Accuracy of ChatGPT, Google Bard, and Microsoft Bing for Simplifying Radiology Reports
Amin, K., Davis, M. A., Doshi, R. H., Haims, A. H., Khosla, P., & Forman, H. P. (2023) · Radiology · doi:10.1148/radiol.232561 · Cited by 104 · see citing works
Large language models should be used as scientific reasoning engines, not knowledge databases
Truhn, D., Reis-Filho, J. S., & Kather, J. N. (2023) · Nature Medicine · doi:10.1038/s41591-023-02594-z · Cited by 102 · see citing works
LLMs Accelerate Annotation for Medical Information ExtractionPreprint
Goel, A., Gueta, A., Gilon, O., Liu, C., Erell, S., Nguyen, L. H., Hao, X., Jaber, B., Reddy, S., Kartha, R., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.02296 · Cited by 86 · see citing works
Comparative Analysis of Multimodal Large Language Model Performance on Clinical Vignette Questions
Han, T., Adams, L. C., Bressem, K. K., Busch, F., Nebelung, S., & Truhn, D. (2024) · JAMA · doi:10.1001/jama.2023.27861 · Cited by 83 · see citing works
Self-citationCIRSE Clinical Practice Manual
Mahnken, A. H., Seoane, E. B., Cannavale, A., et al. (2021) · CardioVascular and Interventional Radiology · doi:10.1007/s00270-021-02904-3 · Cited by 82 · see citing works
Lehnen, N. C., Dorn, F., Wiest, I. C., Zimmermann, H., Radbruch, A., Kather, J. N., & Paech, D. (2024) · Radiology · doi:10.1148/radiol.232741 · Cited by 64 · see citing works
Multimodal ChatGPT-4V for Electrocardiogram Interpretation: Promise and Limitations
Zhu, L., Mou, W., Wu, K., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/54607 · Cited by 48 · see citing works
Safrai, M., & Orwig, K. E. (2024) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-024-03089-7 · Cited by 27 · see citing works
Yu, A., Liu, Z. A., Pou-Prom, C., et al. (2021) · JMIR Medical Informatics · doi:10.2196/24381 · Cited by 25 · see citing works
Exploring ChatGPT's abilities in medical article writing and peer review
Kadı, G., & Aslaner, M. A. (2024) · Croatian Medical Journal · doi:10.3325/cmj.2024.65.93 · Cited by 23 · see citing works
Xie, S., Zhao, W., Deng, G., He, G., He, N., Lü, Z., Hu, W., Zhao, M., & Du, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae100 · Cited by 20 · see citing works
Rule-based natural language processing for automation of stroke data extraction: a validation study
Gunter, D., Puac, P., Miguel, O., Thornhill, R. E., Yu, A., Liu, Z. A., Mamdani, M., Pou-Prom, C., & Aviv, R. I. (2022) · Neuroradiology · doi:10.1007/s00234-022-03029-1 · Cited by 16 · see citing works
Open‐source large language models in action: A bioinformatics chatbot for PRIDE database
Bai, J.-W., Kamatchinathan, S., Kundu, D. J., Bandla, C., Vizcaíno, J., & Pérez-Riverol, Y. (2024) · Proteomics · doi:10.1002/pmic.202400005 · Cited by 15 · see citing works
Evaluating Large Language Models in Extracting Cognitive Exam Dates and ScoresPreprint
Zhang, H., Jethani, N., Jones, S., et al. (2023) · medRxiv · doi:10.1101/2023.07.10.23292373 · Cited by 13 · see citing works
Open Source Large Language Models in Action: A Bioinformatics Chatbot for PRIDE databasePreprint
Bai, J., Kamatchinathan, S., Kundu, D. J., et al. (2024) · doi:10.22541/au.171025539.92037103/v1 · Cited by 2 · see citing works
Paperwork for the Busy Interventionalist: The Basic Six
Smouse, H. B., & Harty, P. (2006) · Seminars in Interventional Radiology · doi:10.1055/s-2006-957020 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review
Chen, D., Alnassar, S. A., Avison, K. E., Huang, R. S., & Raman, S. (2025) · JMIR Cancer · doi:10.2196/65984 · Cited by 66 · see citing works
“Moreover, LLMs may curtail the resources required for data extraction, which is traditionally a labor-and time-inten-sive process [35].”
- Authors
- Meddeb, A., Ebert, P., Bressem, K. K., Desser, D., Dell'Orco, A., Bohner, G., Kleine, J. F., Siebert, E., Grauhan, N., Brockmann, M. A., Othman, A., Scheel, M., & Nawabi, J.
- Affiliations
- Centre Hospitalier Universitaire de Reims (FR)
- Stiftung Charité (DE)
- Deutsches Herzzentrum München (DE)
- Department of Neuroradiology, Charité Universitätsmedizin Berlin, Berlin, Germany
- University Medical Center of the Johannes Gutenberg University Mainz (DE)
- Department of Neuroradiology, University Medical Center of the Johannes Gutenberg University Mainz, Mainz, Germany
- Corresponding author
- Meddeb, Aymen
- ORCID
- Venue
- Journal of NeuroInterventional Surgery
- Publisher
- BMJ
- Published
- 2024-08-02
- Volume
- 17
- Issue
- 9
- Pages
- 986-991
- ISSN
- 1759-8478 (ISSN-L)
- 1759-8486
- OpenAlex
- W4401248212
- Cited by
- 27
- References
- 33
- Citations by year
- 2026: 12 · 2025: 15
- FWCI
- 3.98
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of NeuroInterventional Surgery (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- Funders
- Berlin Institute of Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, clinical information extraction, mechanical thrombectomy, ischemic stroke, human-in-the-loop annotation, NIHSS score, medication extraction, Mixtral, Qwen
- MeSH terms
- Ischemic Stroke, Large Language Models, Aged, Female, Humans, Information Storage and Retrieval, Thrombectomy, Data Mining, Mechanical Thrombolysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
