The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records
Open accessAbstract
Objective: This article summarizes the preparation, organization, evaluation, and results of Track 2 of the 2018 National NLP Clinical Challenges shared task. Track 2 focused on extraction of adverse drug events (ADEs) from clinical records and evaluated 3 tasks: concept extraction, relation classification, and end-to-end systems. We perform an analysis of the results to identify the state of the art in these tasks, learn from it, and build on it. Materials and Methods: For all tasks, teams were given raw text of narrative discharge summaries, and in all the tasks, participants proposed deep learning–based methods with hand-designed features. In the concept extraction task, participants used sequence labelling models (bidirectional long short-term memory being the most popular), whereas in the relation classification task, they also experimented with instance-based classifiers (namely support vector machines and rules). Ensemble methods were also popular. Results: A total of 28 teams participated in task 1, with 21 teams in tasks 2 and 3. The best performing systems set a high performance bar with F1 scores of 0.9418 for concept extraction, 0.9630 for relation classification, and 0.8905 for end-to-end. However, the results were much lower for concepts and relations of Reasons and ADEs. These were often missed because local context is insufficient to identify them. Conclusions: This challenge shows that clinical concept extraction and relation classification systems have a high performance for many concept types, but significant improvement is still required for ADEs and Reasons. Incorporating the larger context or outside knowledge will likely improve the performance of future systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
To Err Is Human: Building a Safer Health System
(2000) · National Academies Press eBooks · doi:10.17226/9728 · Cited by 14,044
To Err is Human: Building a Safer Health System
Hendee, W. R. (2001) · Journal of Vascular and Interventional Radiology · doi:10.1016/s1051-0443(01)70072-3 · Cited by 10,103
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text
Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000203 · Cited by 1,252
Proceedings of the Eighteenth International Conference on Machine Learning
Brodley, C. E., & Danyluk, A. (2001) · Cited by 568
Evaluating temporal relations in clinical text: 2012 i2b2 Challenge
Sun, W., Rumshisky, A., & Uzuner, O. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001628 · Cited by 473
Self-citationExtracting medication information from clinical text
Uzuner, Ö., Solti, I., & Cadag, E. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.003947 · Cited by 466
Evaluating the State-of-the-Art in Automatic De-identification
Uzuner, O., Luo, Y., & Szolovits, P. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2444 · Cited by 452
Self-citationComputer Intensive Methods for Testing Hypotheses.
Hollas, E. A., & Noreen, E. W. (1990) · The Statistician · doi:10.2307/2348203 · Cited by 395
Identifying Patient Smoking Status from Medical Discharge Records
Uzuner, O., Goldstein, I., Luo, Y., & Kohane, I. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2408 · Cited by 359
Self-citationAutomated Detection of Adverse Events Using Natural Language Processing of Discharge Summaries
Melton, G. B., & Hripcsak, G. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1794 · Cited by 297
Stubbs, A., Kotfila, C., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.06.007 · Cited by 245
Self-citationRecognizing Obesity and Comorbidities in Sparse Data
Uzuner, O. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3115 · Cited by 231
Self-citationComputer-Intensive Methods for Testing Hypotheses.
Buckland, S. T., & Noreen, E. W. (1990) · Biometrics · doi:10.2307/2531461 · Cited by 202
Evaluating the state of the art in coreference resolution for electronic medical records
Uzuner, O., Bodnari, A., Shen, S., Forbush, T., Pestian, J., & South, B. R. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000784 · Cited by 164
Self-citationJagannatha, A., Liu, F., Liu, W., & Yu, H. (2019) · Drug Safety · doi:10.1007/s40264-018-0762-z · Cited by 161
De-identification of psychiatric intake records: Overview of 2016 CEGS N-GRID shared tasks Track 1
Stubbs, A., Filannino, M., & Uzuner, Ö. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.06.011 · Cited by 142
Self-citationStubbs, A., Kotfila, C., Xu, H., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.07.001 · Cited by 130
Self-citationChristopoulou, F., Tran, T. T., Sahu, S. K., Miwa, M., & Ananiadou, S. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz101 · Cited by 113
Annotating risk factors for heart disease in clinical narratives for diabetic patients
Stubbs, A., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.05.009 · Cited by 65
Self-citationEnsemble method–based extraction of medication and related information from clinical texts
Kim, Y., & Meystre, S. M. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz100 · Cited by 56
An ensemble of neural models for nested adverse drug events and medication extraction with subwords
Ju, M., Nguyen, N. T. H., Miwa, M., & Ananiadou, S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz075 · Cited by 55
An end-to-end hybrid algorithm for automated medication discrepancy detection
Li, Q., Spooner, S. A., Kaiser, M., Lingren, N., Robbins, J., Lingren, T., Tang, H., Solti, I., & Ni, Y. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0160-8 · Cited by 52
Dai, H.-J., Su, C.-H., & Wu, C. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz120 · Cited by 43
Filannino, M., Stubbs, A., & Uzuner, Ö. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.04.017 · Cited by 41
Self-citationAdvancing the State of the Art in Clinical Natural Language Processing through Shared Tasks
Filannino, M., & Uzuner, Ö. (2018) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1667079 · Cited by 30
Self-citationExtracting Adverse Drug Event Information with Minimal Engineering
Miller, T. A., Geva, A., & Dligach, D. (2019) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/w19-1903 · Cited by 19
Research works printed but not linked: 6
- Goeuriot, 2017, International Conference of the Cross-Language Evaluation Forume for European Languages, 291
- Goeuriot, 2015, International Conference of the Cross-Language Evaluation Forum for European Languages, 429
- Kelly, 2016, International Conference of the Cross-Language Evaluation Forum for European Languages, 255
- Suominen, 2016, Task 1 of the CLEF eHealth evaluation Lab 2016: handover information extraction. In, CLEF Evaluation Labs and Workshop: Online Working Notes
- Sarker, 2017, Overview of the second social media mining for health (SMM4H) shared tasks at AMIA 2017, Training, 1, 1239
- Mikolov, 2013, NIPS’13 Proceedings of the 26th International Conference on Neural Information Processing Systems – Volume 2, 3111
Cited by
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
Murali, L., Gopakumar, G., Viswanathan, D. M., & Nedungadi, P. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104403 · Cited by 93
Fraile Navarro, D., Ijaz, K., Rezazadegan, D., Rahimi-Ardabili, H., Dras, M., Coiera, E., & Berkovsky, S. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105122 · Cited by 82
Li, J., Zhou, Y., Jiang, X., Natarajan, K., Pakhomov, S. V., Liu, H., & Xu, H. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab112 · Cited by 53
Metadata
- Authors
- Henry, S., Buchan, K., Filannino, M., Stubbs, A., & Uzuner, O.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-10-04
- Volume
- 27
- Issue
- 1
- Pages
- 3-12
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2979250794
- Cited by
- 303
- References
- 28
- FWCI
- 8.74
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Funders
- National Library of Medicine of the National Institutes of Health (R13LM013127, R13LM011411)
- NIH
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.