High accuracy information extraction of medication information from clinical notes: 2009 i2b2 medication extraction challenge
Open accessAbstract
OBJECTIVE: Medication information comprises a most valuable source of data in clinical records. This paper describes use of a cascade of machine learners that automatically extract medication information from clinical records. DESIGN: Authors developed a novel supervised learning model that incorporates two machine learning algorithms and several rule-based engines. MEASUREMENTS: Evaluation of each step included precision, recall and F-measure metrics. The final outputs of the system were scored using the i2b2 workshop evaluation metrics, including strict and relaxed matching with a gold standard. RESULTS: Evaluation results showed greater than 90% accuracy on five out of seven entities in the name entity recognition task, and an F-measure greater than 95% on the relationship classification task. The strict micro averaged F-measure for the system output achieved best submitted performance of the competition, at 85.65%. LIMITATIONS: Clinical staff will only use practical processing systems if they have confidence in their reliability. Authors estimate that an acceptable accuracy for a such a working system should be approximately 95%. This leaves a significant performance gap of 5 to 10% from the current processing capabilities. CONCLUSION: A multistage method with mixed computational strategies using a combination of rule-based classifiers and statistical classifiers seems to provide a near-optimal strategy for automated extraction of medication information from clinical records.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Those systems fall well short of the 95% accuracy and F1-scores proposed for clinical use following the original I2B2 challenge.
“Notably, those systems fall well short of the 95% accuracy and F1-scores proposed for clinical use following the original I2B2 challenge.2”
From A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models (2025) · cites this paper as [2]
The Abstract proposes approximately 95% accuracy for a clinically acceptable system and reports a best overall F-measure of 85.65%. But the Conclusions describe the acceptable F-score as minimally above 90 and only perhaps above 95, so the paper does not establish 95% as a firm F-score threshold. (LLM, conf=0.84)
Experimental — accuracy not yet measuredRules-based systems are difficult to develop and can lack real-world robustness.
“However, rules-based systems are difficult to develop and can lack realworld robustness.”
From A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models (2025) · cites this paper as [2]
The Background (page unavailable) reports that the authors’ mixed approach outperformed rule-based approaches used by other challenge teams, offering limited support for a performance concern. The Conclusions (page unavailable) discuss a gap between their own system’s accuracy and what clinical staff would accept, but do not attribute that gap specifically to rule-based systems. The paper does not say rule-based systems are difficult to develop or establish that they lack real-world robustness. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Introduction to Support Vector Machines and Other Kernel-based Learning Methods
Cristianini, N., & Shawe-Taylor, J. (2000) · doi:10.1017/cbo9780511801389 · Cited by 13,855 · see citing works
Extracting medication information from clinical text
Uzuner, Ö., Solti, I., & Cadag, E. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.003947 · Cited by 466 · see citing works
Extracting structured medication event information from discharge summaries.
Gold, S., Elhadad, N., Zhu, X., Cimino, J. J., & Hripcsak, G. M. (2008) · PubMed Central · Cited by 73 · see citing works
Evans, D. A., Brownlow, N. D., Hersh, W., et al. (1996) · PubMed · Cited by 56 · see citing works
An automated system for conversion of clinical notes into SNOMED clinical terminology
Patrick, J. D., Wang, Y., & Budd, P. (2007) · Cited by 56 · see citing works
Self-citationSTUDY OF EFFECT OF DRUG LEXICONS ON MEDICATION EXTRACTION FROM ELECTRONIC MEDICAL RECORDS
SIROHI, E., & Peissig, P. L. (2004) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789812702456_0029 · Cited by 47 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [7]7. Wang Y, Patrick J. Evaluating linguistic features for clinical relation extraction. Proceedings of the Conference of the Pacific Association for Computational Linguistics (PACLING), Sapporo, Japan, September 1e4, 2009:349e55.
Cited by
Sorted by citations
Extracting comprehensive clinical information for breast cancer using deep learning methods
Zhang, X., Zhang, Y., Zhang, Q., Ren, Y., Qiu, T., Ma, J., & Sun, Q. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103985 · Cited by 160 · see citing works
Fraile Navarro, D., Ijaz, K., Rezazadegan, D., Rahimi-Ardabili, H., Dras, M., Coiera, E., & Berkovsky, S. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105122 · Cited by 82 · see citing works
Stuhlmiller, T. J., Rabe, A., Rapp, J., Awawda, A., Kouser, H., Lui, K., Salamon, H., Chuyka, D., Mahoney, W., Furgason, J. M., et al. (2025) · medRxiv · doi:10.1101/2025.02.25.25322898 · Cited by 13 · see citing works
“1,2 However, rules-based systems are difficult to develop and can lack real-world robustness.”
- Authors
- Patrick, J., & Li, M.
- Affiliations
- The University of Sydney (AU)
- Corresponding author
- Patrick, Jon
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-09
- Volume
- 17
- Issue
- 5
- Pages
- 524-527
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC2995676
- OpenAlex
- W2114668172
- Cited by
- 159
- References
- 6
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 1 · 2023: 7 · 2022: 9 · 2021: 9 · 2020: 17 · 2019: 17 · 2018: 10 · 2017: 7 · 2016: 8 · 2015: 14 · 2014: 17 · 2013: 11 · 2012: 12
- FWCI
- 4.43
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- State University of New York
- National Institutes of Health
- U.S. National Library of Medicine (U54LM008748)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- clinical notes, named entity recognition, relationship classification, rule-based classifiers, statistical classifiers, clinical text mining
- MeSH terms
- Artificial Intelligence, Pharmaceutical Preparations, Humans, Natural Language Processing, Information Storage and Retrieval, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
