PreprintCited by 90Topic ModelingNatural Language Processing TechniquesCategorization, perception, and language

HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

Open access
Vulić, Ivan, Gerz, Daniela, Kiela, Douwe, Hill, Felix, Korhonen, Anna·2016·arXiv

Abstract

We introduce HyperLex - a dataset and evaluation resource that quantifies the extent of of the semantic category membership, that is, type-of relation also known as hyponymy-hypernymy or lexical entailment (LE) relation between 2,616 concept pairs. Cognitive psychology research has established that typicality and category/class membership are computed in human semantic memory as a gradual rather than binary relation. Nevertheless, most NLP research, and existing large-scale invetories of concept category membership (WordNet, DBPedia, etc.) treat category membership and LE as binary. To address this, we asked hundreds of native English speakers to indicate typicality and strength of category membership between a diverse range of concept pairs on a crowdsourcing platform. Our results confirm that category membership and LE are indeed more gradual than binary. We then compare these human judgements with the predictions of automatic systems, which reveals a huge gap between human performance and state-of-the-art LE, distributional and representation learning models, and substantial differences between the models themselves. We discuss a pathway for improving semantic models to overcome this discrepancy, and indicate future application areas for improved graded LE systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1608.02117

Cites

173 of 173 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The ranking based on Poincaré embeddings clearly outperforms all state-of-the-art methods evaluated in [32] on HYPERLEX.

    “It can be seen that the ranking based on Poincaré embeddings clearly outperforms all state-of-the-art methods evaluated in [32].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

6 of the 179 references couldn’t be matched to a known paper.

Research works printed but not linked: 14

  1. Al-Rfou, Rami, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of CoNLL, pages 183-192, Sofia.
  2. Baronett, Stan. 2012. Logic, 3rd edition.
  3. Beckwith, Richard, Christiane Fellbaum, Derek Gross, and George A. Miller. 1991. WordNet: A lexical database organized on psycholinguistic principles. Lexical acquisition: Exploiting on-line resources to build a lexicon, pages 211-231.
  4. Fellbaum, Christiane. 1998. WordNet. Cambridge, MA, MIT Press.
  5. Fromkin, Victoria, Robert Rodman, and Nina Hyams. 2013. An Introduction to Language, 10th edition. Wadsworth Cengage Learning.
  6. Levin, Beth. 1993. English Verb Classes and Alternation: A Preliminary Investigation. University of Chicago Press.
  7. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of the 27th Annual Conference on Advances in Neural Information Processing Systems (NIPS), pages 3111-3119, South Lake Tahoe, NV.
  8. Myers, Raymond H. 1990. Classical and Modern Regression with Applications (Volume 2). Duxbury Press.
  9. Pantel, Patrick and Marco Pennacchiotti. 2006. Espresso: Leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of ACL, pages 113-120, Sydney.
  10. Sacaleanu, Bogdan, Constantin Orasan, Christian Spurk, Shiyan Ou, Oscar Ferrandez, Milen Kouylekov, and Matteo Negri. 2008. Entailment-based question answering for structured data. In Proceedings of COLING, pages 173-176, Manchester.
  11. Suchanek, Fabian M., Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In Proceedings of WWW, pages 697-706, Banff.
  12. Taylor, John R. 2003. Linguistic Categorization. Oxford University Press.
  13. Vendrov, Ivan, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016. Order-embeddings of images and language. In Proceedings of ICLR, San Juan, PR.
  14. Wu, Wentao, Hongsong Li, Haixun Wang, and Kenny Q. Zhu. 2012. Probase: A probabilistic taxonomy for text understanding. In Proceedings of SIGMOD, pages 481-492, Scottsdale, AZ.

Cited by

Sorted by citations

Authors
Vulić, I., Gerz, D., Kiela, D., Hill, F., & Korhonen, A.
Affiliations
  • University of Cambridge (GB)
  • Meta (United States) (US)
  • Google DeepMind (United Kingdom) (GB)
Corresponding author
  • Vulić, Ivan
  • Gerz, Daniela
  • Kiela, Douwe
  • Hill, Felix
  • Korhonen, Anna
Venue
arXiv
Publisher
arXiv
Published
2016
Volume
43
Issue
4
Pages
781-835
ISSN
0891-2017 (ISSN-L)
OpenAlex
W2519468206
Cited by
90
References
179
Citations by year
2026: 2 · 2025: 2 · 2024: 7 · 2023: 7 · 2022: 7 · 2021: 9 · 2020: 17 · 2019: 18 · 2018: 11 · 2017: 10
FWCI
9.14
Type
journal article
Language
en
Open access
Locations
  • Computational Linguistics (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
  • Apollo
Funders
  • University of Cambridge
  • H2020 European Research Council (648909)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
Keywords
typicality, human judgment, distributional semantics, representation learning, semantic models, WordNet, DBpedia

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review