The 2019 n2c2/UMass Lowell shared task on clinical concept normalization
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Objective: The n2c2/UMass Lowell spin-off shared task focused on medical concept normalization (MCN) in clinical records. This task aimed to assess state-of-the-art methods for matching salient medical concepts from clinical records to a controlled vocabulary. We describe the task and the dataset used, compare the participating systems, and identify the strengths and limitations of the current approaches and directions for future research. Materials and Methods: Participating teams were asked to link preselected text spans in discharge summaries (henceforth referred to as concept mentions) to the corresponding concepts in the SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and RxNorm vocabularies from the Unified Medical Language System. The shared task used the MCN corpus created by the organizers, which maps all mentions of problems, treatments, and tests in the 2010 i2b2/VA challenge data to the Unified Medical Language System concepts. Submitted systems represented 4 broad categories of approaches: cascading dictionary matching, cosine distance, deep learning, and retrieve-and-rank systems. Disambiguation modules were common across all approaches. Results: A total of 33 teams participated in the shared task. The best-performing team achieved an accuracy of 0.8526. The median and mean performances among all teams were 0.7733 and 0.7426, respectively. Conclusions: Overall performance among the top 10 teams was high. However, particularly challenging for all teams were mentions requiring disambiguation of misspelled words, acronyms, abbreviations, and mentions with more than 1 possible semantic type. Complex mentions of long, multiword terms were also challenging and, in the future, will require better methods for learning contextualized representations of concept mentions and better use of domain knowledge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Savova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967
Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016) · Scientific Reports · doi:10.1038/srep26094 · Cited by 1,710
Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688
RxNorm: prescription for electronic drug information exchange
Liu, S., Ma, W., Moore, R., Ganesan, V., & Nelson, S. (2005) · IT Professional · doi:10.1109/mitp.2005.122 · Cited by 274
Ye, C., Fu, T., Hao, S., Zhang, Y., Wang, O., Jin, B., Xia, M., Liu, M., Zhou, X., Wu, Q., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9268 · Cited by 250
Predicting the onset of type 2 diabetes using wide and deep learning with electronic health records
Nguyen, B. P., Pham, H. N., Tran, H., Nghiem, N., Nguyen, Q. H., Do, T. T. T., Tran, C. T., & Simpson, C. R. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.105055 · Cited by 158
MCN: A comprehensive corpus for medical concept normalization
Luo, Y.-F., Sun, W., & Rumshisky, A. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103132 · Cited by 62
Self-citation
Cited by
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358
A scoping review on generative AI and large language models in mitigating medication related harm
Ong, J. C. L., Chen, M. H., Ng, N., Elangovan, K., Tan, N. Y. T., Jin, L., Xie, Q., Ting, D. S. W., Rodriguez-Monguio, R., Bates, D. W., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01565-7 · Cited by 64
A scoping review of publicly available language tasks in clinical natural language processing
Gao, Y., Dligach, D., Christensen, L., Tesch, S., Laffin, R., Xu, D., Miller, T., Uzuner, O., Churpek, M. M., & Afshar, M. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac127 · Cited by 28
Metadata
- Authors
- Luo, Y.-F., Henry, S., Wang, Y., Shen, F., Uzuner, O., & Rumshisky, A.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-09-24
- Volume
- 27
- Issue
- 10
- Pages
- 1529-e1
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W3212714602
- Cited by
- 54
- References
- 9
- FWCI
- 2.07
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The 2019 n2c2/UMass Lowell shared task on clinical concept normalization. https://app.cochat.ai/papers/10.1093/jamia/ocaa106
- The paper
- Luo, Y.-F., Henry, S., Wang, Y., Shen, F., Uzuner, O., & Rumshisky, A. (2020). The 2019 n2c2/UMass Lowell shared task on clinical concept normalization. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaa106
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.