The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records
Open accessAbstract
Objective: This article summarizes the preparation, organization, evaluation, and results of Track 2 of the 2018 National NLP Clinical Challenges shared task. Track 2 focused on extraction of adverse drug events (ADEs) from clinical records and evaluated 3 tasks: concept extraction, relation classification, and end-to-end systems. We perform an analysis of the results to identify the state of the art in these tasks, learn from it, and build on it. Materials and Methods: For all tasks, teams were given raw text of narrative discharge summaries, and in all the tasks, participants proposed deep learning–based methods with hand-designed features. In the concept extraction task, participants used sequence labelling models (bidirectional long short-term memory being the most popular), whereas in the relation classification task, they also experimented with instance-based classifiers (namely support vector machines and rules). Ensemble methods were also popular. Results: A total of 28 teams participated in task 1, with 21 teams in tasks 2 and 3. The best performing systems set a high performance bar with F1 scores of 0.9418 for concept extraction, 0.9630 for relation classification, and 0.8905 for end-to-end. However, the results were much lower for concepts and relations of Reasons and ADEs. These were often missed because local context is insufficient to identify them. Conclusions: This challenge shows that clinical concept extraction and relation classification systems have a high performance for many concept types, but significant improvement is still required for ADEs and Reasons. Incorporating the larger context or outside knowledge will likely improve the performance of future systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
To Err Is Human: Building a Safer Health System
(2000) · National Academies Press eBooks · doi:10.17226/9728 · Cited by 14,044
To Err is Human: Building a Safer Health System
Hendee, W. R. (2001) · Journal of Vascular and Interventional Radiology · doi:10.1016/s1051-0443(01)70072-3 · Cited by 10,103
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text
Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000203 · Cited by 1,252
Proceedings of the Eighteenth International Conference on Machine Learning
Brodley, C. E., & Danyluk, A. (2001) · Cited by 568
Evaluating temporal relations in clinical text: 2012 i2b2 Challenge
Sun, W., Rumshisky, A., & Uzuner, O. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001628 · Cited by 473
Self-citationExtracting medication information from clinical text
Uzuner, Ö., Solti, I., & Cadag, E. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.003947 · Cited by 466
Evaluating the State-of-the-Art in Automatic De-identification
Uzuner, O., Luo, Y., & Szolovits, P. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2444 · Cited by 452
Self-citationComputer Intensive Methods for Testing Hypotheses.
Hollas, E. A., & Noreen, E. W. (1990) · The Statistician · doi:10.2307/2348203 · Cited by 395
Research works printed but not linked: 6
- Goeuriot, 2017, International Conference of the Cross-Language Evaluation Forume for European Languages, 291
- Goeuriot, 2015, International Conference of the Cross-Language Evaluation Forum for European Languages, 429
- Kelly, 2016, International Conference of the Cross-Language Evaluation Forum for European Languages, 255
- Suominen, 2016, Task 1 of the CLEF eHealth evaluation Lab 2016: handover information extraction. In, CLEF Evaluation Labs and Workshop: Online Working Notes
- Sarker, 2017, Overview of the second social media mining for health (SMM4H) shared tasks at AMIA 2017, Training, 1, 1239
- Mikolov, 2013, NIPS’13 Proceedings of the 26th International Conference on Neural Information Processing Systems – Volume 2, 3111
Cited by
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
Murali, L., Gopakumar, G., Viswanathan, D. M., & Nedungadi, P. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104403 · Cited by 93
Fraile Navarro, D., Ijaz, K., Rezazadegan, D., Rahimi-Ardabili, H., Dras, M., Coiera, E., & Berkovsky, S. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105122 · Cited by 82
Li, J., Zhou, Y., Jiang, X., Natarajan, K., Pakhomov, S. V., Liu, H., & Xu, H. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab112 · Cited by 53
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30205 · Cited by 66
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59
Yang, J., Wang, L., Phadke, N. A., Wickner, P. G., Mancini, C. M., Blumenthal, K. G., & Zhou, L. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.22836 · Cited by 42
Contextualized medication information extraction using Transformer-based deep learning architectures
Chen, A., Yu, Z., Yang, X., Guo, Y., Bian, J., & Wu, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104370 · Cited by 40
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
“…Records [169], Extracting Information from Clinical Narrative Reports [170], Extracting Information From Medical Articles [171] CASI [150], n2c2 [111], i2b2 [93], PubMedQA [108], MedMCQA [118], emrQA [113], BIOASQ [103] quEHRy [172], BiomedBERT [85], PubMedBERT [86], BioGPT [89], Llava-med [90]…”
Yu, Z., Peng, C., Yang, X., Dang, C., Adekkanattu, P., Gopal Patra, B., Peng, Y., Pathak, J., Wilson, D. L., Chang, C.-Y., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104642 · Cited by 37
Yang, X., Zhang, H., He, X., Bian, J., & Wu, Y. (2020) · JMIR Medical Informatics · doi:10.2196/22982 · Cited by 29
A scoping review of publicly available language tasks in clinical natural language processing
Gao, Y., Dligach, D., Christensen, L., Tesch, S., Laffin, R., Xu, D., Miller, T., Uzuner, O., Churpek, M. M., & Afshar, M. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac127 · Cited by 28
Chen, S., Guevara, M., Ramirez, N., Murray, A., Warner, J. L., Aerts, H. J. W. L., Miller, T. A., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2023) · JCO Clinical Cancer Informatics · doi:10.1200/cci.23.00048 · Cited by 20
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Fries, J. A., Weber, L., Seelam, N., Altay, G., Datta, D., Garda, S., Kang, M., Su, R., Kusa, W., Cahyawijaya, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.15076 · Cited by 17
LLM-IE: a python package for biomedical generative information extraction with large language models
Hsu, E., & Roberts, K. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf012 · Cited by 16
Explanatory argumentation in natural language for correct and incorrect medical diagnoses
Molinet, B., Marro, S., Cabrio, E., & Villata, S. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00306-1 · Cited by 4
Metadata
- Authors
- Henry, S., Buchan, K., Filannino, M., Stubbs, A., & Uzuner, O.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-10-04
- Volume
- 27
- Issue
- 1
- Pages
- 3-12
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2979250794
- Cited by
- 303
- References
- 28
- FWCI
- 8.74
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Funders
- National Library of Medicine of the National Institutes of Health (R13LM013127, R13LM011411)
- NIH
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.