CoChat

The 2019 n2c2/UMass Lowell shared task on clinical concept normalization

Open access
Luo, Yen-Fu, Henry, Sam, Wang, Yanshan, Shen, Feichen, Uzuner, Ozlem, Rumshisky, Anna · 2020 · Journal of the American Medical Informatics Association
Cited by 54

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

Abstract

Objective: The n2c2/UMass Lowell spin-off shared task focused on medical concept normalization (MCN) in clinical records. This task aimed to assess state-of-the-art methods for matching salient medical concepts from clinical records to a controlled vocabulary. We describe the task and the dataset used, compare the participating systems, and identify the strengths and limitations of the current approaches and directions for future research. Materials and Methods: Participating teams were asked to link preselected text spans in discharge summaries (henceforth referred to as concept mentions) to the corresponding concepts in the SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and RxNorm vocabularies from the Unified Medical Language System. The shared task used the MCN corpus created by the organizers, which maps all mentions of problems, treatments, and tests in the 2010 i2b2/VA challenge data to the Unified Medical Language System concepts. Submitted systems represented 4 broad categories of approaches: cascading dictionary matching, cosine distance, deep learning, and retrieve-and-rank systems. Disambiguation modules were common across all approaches. Results: A total of 33 teams participated in the shared task. The best-performing team achieved an accuracy of 0.8526. The median and mean performances among all teams were 0.7733 and 0.7426, respectively. Conclusions: Overall performance among the top 10 teams was high. However, particularly challenging for all teams were mentions requiring disambiguation of misspelled words, acronyms, abbreviations, and mentions with more than 1 possible semantic type. Complex mentions of long, multiword terms were also challenging and, in the future, will require better methods for learning contextualized representations of concept mentions and better use of domain knowledge.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaa106

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 54

Metadata

Authors
Luo, Y.-F., Henry, S., Wang, Y., Shen, F., Uzuner, O., & Rumshisky, A.
Venue
Journal of the American Medical Informatics Association
Published
2020-09-24
Volume
27
Issue
10
Pages
1529-e1
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W3212714602
Cited by
54
References
9
FWCI
2.07
Type
journal article
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.