CoChat

SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection

Open access
Schlechtweg, Dominik, McGillivray, Barbara, Hengchen, Simon, Dubossarsky, Haim, Tahmasebi, Nina · 2020 · Proceedings of the Fourteenth Workshop on Semantic Evaluation
Cited by 172

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

Lexical Semantic Change detection, i.e., the task of identifying words that change meaning over time, is a very active research area, with applications in NLP, lexicography, and linguistics.Evaluation is currently the most pressing problem in Lexical Semantic Change detection, as no gold standards are available to the community, which hinders progress.We present the results of the first shared task that addresses this gap by providing researchers with an evaluation framework and manually annotated, high-quality datasets for English, German, Latin, and Swedish.33 teams submitted 186 systems, which were evaluated on two subtasks. OverviewRecent years have seen an exponentially rising interest in computational Lexical Semantic Change (LSC) detection (Tahmasebi et al., 2018;Kutuzov et al., 2018).However, the field is lacking standard evaluation tasks and data.Almost all papers differ in how the evaluation is performed and what factors are considered in the evaluation.Very few are evaluated on a manually annotated diachronic corpus (McGillivray et al., 2019;Perrone et al., 2019; Schlechtweg et al., 2019, e.g.).This puts a damper on the development of computational models for LSC, and is a barrier for high-quality, comparable results that can be used in follow-up tasks.We report the results of the first SemEval shared task on Unsupervised LSC detection.1 We introduce two related subtasks for computational LSC detection, which aim to identify the change in meaning of words over time using corpus data.We provide a high-quality multilingual (English, German, Latin, Swedish) LSC gold standard relying on approximately 100,000 instances of human judgment.For the first time, it is possible to compare the variety of proposed models on relatively solid grounds and across languages, and to put previously reached conclusions on trial.We may now provide answers to questions concerning the performance of different types of semantic representations (such as token embeddings vs. type embeddings, and topic models vs. vector space models), alignment methods and change measures.We provide a thorough analysis of the submitted results uncovering trends for models and opening perspectives for further improvements.In addition to this, the CodaLab website will remain open to allow any reader to directly and easily compare their results to the participating systems.We expect the long-term impact of the task to be significant, and hope to encourage the study of LSC in more languages than are currently studied, in particular less-resourced languages. SubtasksFor the proposed tasks we rely on the comparison of two time-specific corpora C 1 and C 2 .While this simplifies the LSC detection problem, it has two main advantages: (i) it reduces the number of time periods for which data has to be annotated, so we can annotate larger corpus samples and hence more reliably represent the sense distributions of target words; (ii) it reduces the task complexity, allowing * SH was affiliated with the

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.semeval-1.1

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Cited by

See all 172

Metadata

Authors
Schlechtweg, D., McGillivray, B., Hengchen, S., Dubossarsky, H., & Tahmasebi, N.
Venue
Proceedings of the Fourteenth Workshop on Semantic Evaluation
Published
2020
Pages
1-23
OpenAlex
W3115228514
Cited by
172
References
8
FWCI
13.61
Type
conference paper
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. https://app.cochat.ai/papers/10.18653/v1/2020.semeval-1.1
The paper
Schlechtweg, D., McGillivray, B., Hengchen, S., Dubossarsky, H., & Tahmasebi, N. (2020). SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. Proceedings of the Fourteenth Workshop on Semantic Evaluation. https://doi.org/10.18653/v1/2020.semeval-1.1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.