LLM-IE: a python package for biomedical generative information extraction with large language models
Open accessAbstract
Objectives: Despite the recent adoption of large language models (LLMs) for biomedical information extraction (IE), challenges in prompt engineering and algorithms persist, with no dedicated software available. To address this, we developed LLM-IE: a Python package for building complete IE pipelines. Materials and Methods: The LLM-IE supports named entity recognition, entity attribute extraction, and relation extraction tasks. We benchmarked it on the i2b2 clinical datasets. Results: The sentence-based prompting algorithm resulted in the best 8-shot performance of over 70% strict F1 for entity extraction and about 60% F1 for entity attribute extraction. Discussion: We developed a Python package, LLM-IE, highlighting (1) an interactive LLM agent to support schema definition and prompt design, (2) state-of-the-art prompting algorithms, and (3) visualization features. Conclusion: The LLM-IE provides essential building blocks for developing robust information extraction pipelines. Future work will aim to expand its features and further optimize computational efficiency.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs can identify relevant literature given mechanistic context, read scientific text, and extract information into structured templates.
“LLMs can identify relevant literature given mechanistic context, read scientific text, and extract information into structured templates [14, 15, 16, 17, 18].”
From Structured Schemas for Provenance‐Rich, <scp>LLM</scp> ‐Assisted <scp>QSP</scp> Model Calibration (2026) · cites this paper as [17]
The Background and significance section (page not provided) describes LLMs extracting entities and relations from documents into a structured format, supporting the information-extraction part of the sentence. It does not address identifying relevant literature from mechanistic context; its examples concern biomedical and clinical text extraction. (LLM, conf=0.85)
Experimental — accuracy not yet measuredLLMs can identify relevant literature given mechanistic context, read scientific text, and extract information into structured templates.
“LLMs can identify relevant literature given mechanistic context, read scientific text, and extract information into structured templates14,15,16,17,18.”
From Structured Schemas for Provenance-Rich, LLM-Assisted QSP Model Calibration (2026) · cites this paper as [17]
MAPLE addresses a gap between general-purpose biomedical extraction frameworks17,18, which do not address QSP-specific downstream requirements (uncertainty quantification, forward model specification, code generation), and QSP-focused tools like QSP-Copilot16, which include literature extraction components but emphasize model structure discovery rather than quantitative parameter calibration with
“MAPLE addresses a gap between general-purpose biomedical extraction frameworks17,18, which do not address QSP-specific downstream requirements (uncertainty quantification, forward model specification, code generation), and QSP-focused tools like QSP-Copilot16”
From Structured Schemas for Provenance-Rich, LLM-Assisted QSP Model Calibration (2026) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Structured information extraction from scientific text with large language models
Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A. S., Ceder, G., Persson, K. A., & Jain, A. (2024) · Nature Communications · doi:10.1038/s41467-024-45563-x · Cited by 771 · see citing works
Evaluating temporal relations in clinical text: 2012 i2b2 Challenge
Sun, W., Rumshisky, A., & Uzuner, Ö. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001628 · Cited by 477 · see citing works
Improving large language models for clinical named entity recognition via prompt engineering
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad259 · Cited by 314 · see citing works
Self-citation2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records
Henry, S., Buchan, K., Filannino, M., Stubbs, A., & Uzuner, Ö. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz166 · Cited by 307 · see citing works
Stubbs, A., Kotfila, C., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.06.007 · Cited by 243 · see citing works
Revisiting Relation Extraction in the era of Large Language Models
Wadhwa, S., Amir, S., & Wallace, B. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.868 · Cited by 207 · see citing works
Leveraging pre-trained language models for mining microbiome-disease relationships
Karkera, N., Acharya, S., & Palaniappan, S. K. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05411-z · Cited by 41 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- [1]Xu D, Chen W, Peng W, et al. Large language models for generative information extraction: a survey. Front Comput Sci. 2024;18:186357.
- [2]Brown T, Mann B, Ryder N, et al Language models are Few-Shot learners. Adv Neural Inf Process Syst. 2020; 33: 1877- 1901.
- [3]Agrawal M, Hegselmann S, Lang H, et al Large language models are few-shot clinical information extractors. 2022, preprint: not peer reviewed.
- [7]Kwon W, Li Z, Zhuang S, et al Efficient memory management for large language model serving with pagedattention. Proceedings of the 29th Symposium on Operating Systems Principles. 2023: 611- 626.
- [11]Renze M, Guven E. Self-reflection in LLM agents: effects on problem-solving performance. arXiv preprint arXiv:240506682. 2024.
- [13]Wang X, Zhou W, Zu C, et al. Instructuie: multi-task instruction tuning for unified information extraction. arXiv preprint arXiv:230408085. 2023.
- [14]Xie T, Li Q, Zhang Y, et al. Self-improving for zero-shot named entity recognition with large language models. arXiv preprint arXiv:231108921. 2023.
- [15]Jahan I, Laskar MTR, Peng C, et al Evaluation of ChatGPT on biomedical tasks: a zero-shot comparison with fine-tuned generative transformers. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. Association for Computational Linguistics; 2023: 326-3 36.
- [20]Meoni S, De la Clergerie E, Ryffel T. Large language models as instructors: a study on multilingual clinical entity extraction. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. Association for Computational Linguistics; 2023: 178-1 90.
- [24]Dubey A, Jauhri A, Pandey A, et al. The llama 3 herd of models. arXiv preprint arXiv:240721783. 2024.
- [25]Hsu E, Roberts K. Leveraging large language models for knowledge-free weak supervision in clinical natural language processing. arXiv preprint arXiv:240606723. 2024. PMC11893743 40064991
Cited by
Sorted by citations
Structured Schemas for Provenance-Rich, LLM-Assisted QSP Model Calibration
Eliason, J., & Popel, A. S. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.05.709623
“LLMs can identify relevant litera-56 ture given mechanistic context, read scientific text, and extract information into structured tem-57 plates 14,15,16,17,18 .”
Structured Schemas for Provenance‐Rich, <scp>LLM</scp> ‐Assisted <scp>QSP</scp> Model Calibration
Eliason, J., & Popel, A. S. (2026) · CPT: Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.70306
- Authors
- Hsu, E., & Roberts, K.
- Affiliations
- The University of Texas MD Anderson Cancer Center (US)
- The University of Texas Health Science Center at Houston (US)
- Corresponding author
- Roberts, Kirk
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-03-06
- Volume
- 8
- Issue
- 2
- Pages
- ooaf012-ooaf012
- ISSN
- 2574-2531 (ISSN-L)
- PMC
- PMC11901043
- OpenAlex
- W4408345962
- Cited by
- 16
- References
- 8
- Citations by year
- 2026: 8 · 2025: 8
- Type
- software paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JAMIA Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- Funders
- National Library of Medicine (R01LM011934)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, biomedical information extraction, named entity recognition, entity attribute extraction, relation extraction, prompt engineering, clinical text mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
