CoChat
Sign Up
Cited by 19Topic ModelingNatural Language Processing TechniquesSentiment Analysis and Opinion Mining

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Analogies minus analogy test: measuring regularities in word embeddings

Open access
Fournier, Louis, Dupoux, Emmanuel, Dunbar, Ewan·2020·Proceedings of the 24th Conference on Computational Natural Language Learning
DOI 10.18653/v1/2020.conll-1.29
Open this paper in CoChat — free account

Abstract

Vector space models of words have long been claimed to capture linguistic regularities as simple vector translations, but problems have been raised with this claim. We decompose and empirically analyze the classic arithmetic word analogy test, to motivate two new metrics that address the issues with the standard test, and which distinguish between class-wise offset concentration (similar directions between pairs of words drawn from different broad classes, such as France--London, China--Ottawa, ...) and pairing consistency (the existence of a regular transformation between correctly-matched pairs such as France:Paris::China:Beijing). We show that, while the standard analogy test is flawed, several popular word embeddings do nevertheless encode linguistic regularities.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.conll-1.29

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 : Figure 1: Decomposition of the analogy score in three terms, with only two depending on the word to predict. The largest component is b • b * , proportional to within-pair similarity. b • o a is proportional to the similarity of the offset to the start word b. o b • o a is proportional to the similarity between offsets. All terms are divided by the same overall normalization term (not indicated in the legend).
Figure 1 : Figure 1: Decomposition of the analogy score in three terms, with only two depending on the word to predict. The largest component is b • b * , proportional to within-pair similarity. b • o a is proportional to the similarity of the offset to the start word b. o b • o a is proportional to the similarity between offsets. All terms are divided by the same overall normalization term (not indicated in the legend).CC-BY · From the PDF
Figure 2 : Figure 2: Decomposition of ∆ sim , the difference between the similarity of the analogy to b * and to b. The term proportional to the dissimilarity of b to b * is greater than the one proportional to the offset similarity, which means the common direction of the offset is not enough to bring the analogy closer to b * than to b.
Figure 2 : Figure 2: Decomposition of ∆ sim , the difference between the similarity of the analogy to b * and to b. The term proportional to the dissimilarity of b to b * is greater than the one proportional to the offset similarity, which means the common direction of the offset is not enough to bring the analogy closer to b * than to b.CC-BY · From the PDF
Figure 3 : Figure3: Schematic illustration of an analogy underlining the role of within-pair similarity in the arithmetic analogy test. At left, China + (Paris -France) predicts Beijing in part for the "right" reason (offsets are parallel). At right, the offsets are not parallel, but Beijing is likely to be predicted, since its cosine similarity to China is high. The problem is worse than the 2D schematic would suggests, since it can occur even when the word vectors are normalized, and when the offsets are quite long. China will be predicted if it is not excluded, since it is also in the neighbourhood.
Figure 3 : Figure3: Schematic illustration of an analogy underlining the role of within-pair similarity in the arithmetic analogy test. At left, China + (Paris -France) predicts Beijing in part for the "right" reason (offsets are parallel). At right, the offsets are not parallel, but Beijing is likely to be predicted, since its cosine similarity to China is high. The problem is worse than the 2D schematic would suggests, since it can occur even when the word vectors are normalized, and when the offsets are quite long. China will be predicted if it is not excluded, since it is also in the neighbourhood.CC-BY · From the PDF
Figure 4 : Figure 4: Schematic illustration of an analogy underlining the pitfalls of offset concentration. At left, Paris -France appears nearly orthogonal to Beijing -China. At right, however, the offsets have much greater cosine similarity, simply because Paris and Beijing are far away. The similarity between Paris -China and Beijing -France (dotted lines) is almost as high. Unlike what might be suggested by the 2D schema, this can arise easily even for normalized word vectors.
Figure 4 : Figure 4: Schematic illustration of an analogy underlining the pitfalls of offset concentration. At left, Paris -France appears nearly orthogonal to Beijing -China. At right, however, the offsets have much greater cosine similarity, simply because Paris and Beijing are far away. The similarity between Paris -China and Beijing -France (dotted lines) is almost as high. Unlike what might be suggested by the 2D schema, this can arise easily even for normalized word vectors.CC-BY · From the PDF
Figure 5 : Figure 5: Distribution of the cosine similarities between vector offsets for word2vec embeddings, grouped by broad relation type. The Offset Concentration Score is the mean offset similarity for all pairs within a single relation (e.g., country-capital).
Figure 5 : Figure 5: Distribution of the cosine similarities between vector offsets for word2vec embeddings, grouped by broad relation type. The Offset Concentration Score is the mean offset similarity for all pairs within a single relation (e.g., country-capital).CC-BY · From the PDF
Figure 6 : Figure 6: Left: accuracy on the standard arithmetic analogy test, across embeddings, averaged across BATS relations (within the four broad types). Right: the same figure for the Pairing Consistency Score, starting at .5 (chance-level).
Figure 6 : Figure 6: Left: accuracy on the standard arithmetic analogy test, across embeddings, averaged across BATS relations (within the four broad types). Right: the same figure for the Pairing Consistency Score, starting at .5 (chance-level).CC-BY · From the PDF
Figure 7 : Figure7: Distribution of the cosine similarity of vector offsets to the mean direction for the default word2vec embeddings for each category, then grouped by broad type. The mean similarity improves the similarity even for anisotropic offsets, as discussed. Variance changes seem category dependent.
Figure 7 : Figure7: Distribution of the cosine similarity of vector offsets to the mean direction for the default word2vec embeddings for each category, then grouped by broad type. The mean similarity improves the similarity even for anisotropic offsets, as discussed. Variance changes seem category dependent.CC-BY · From the PDF
Figure 8 : Figure 8: Decomposition of the analogy score if o a = o b (the score being therefore equal to 1). The similar values of b • (b + o a ) and o a • o a indicates that the length of o a and b is similar, and thus that the offset length does not explain the low value of o a •o b in the real decomposition.
Figure 8 : Figure 8: Decomposition of the analogy score if o a = o b (the score being therefore equal to 1). The similar values of b • (b + o a ) and o a • o a indicates that the length of o a and b is similar, and thus that the offset length does not explain the low value of o a •o b in the real decomposition.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 20

  1. Carl Allen and Timothy Hospedales. 2019. Analo- gies Explained: Towards Understanding Word Em- beddings. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Pro- ceedings of Machine Learning Research, pages 223- 231, Long Beach, California, USA. PMLR.
  2. Zied Bouraoui, Shoaib Jameel, and Steven Schock- aert. 2018. Relation Induction in Word Embed- dings Revisited. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, pages 1627-1637, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
  3. Dawn Chen, Joshua C. Peterson, and Thomas L. Grif- fiths. 2017. Evaluating vector-space models of anal- ogy. CoRR, abs/1705.04416. · arXiv
  4. Aleksandr Drozd, Anna Gladkova, and Satoshi Mat- suoka. 2016. Word Embeddings, Analogies, and Machine Learning: Beyond King -Man + Woman = Queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguis- tics: Technical Papers, pages 3519-3530, Osaka, Japan, December 11-17.
  5. Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Towards Understanding Linear Word Analo- gies. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3253-3262, Florence, Italy. Association for Computational Linguistics. · DOI
  6. Gregory Finley, Stephanie Farmer, and Serguei Pakho- mov. 2017. What Analogies Reveal about Word Vec- tors and their Compositionality. In Proceedings of the 6th Joint Conference on Lexical and Computa- tional Semantics (*SEM 2017), pages 1-11, Vancou- ver, Canada. Association for Computational Linguis- tics. · DOI
  7. Anna Gladkova, Aleksandr Drozd, and Satoshi Mat- suoka. 2016. Analogy-Based Detection of Morpho- logical and Semantic Relations with Word Embed- dings: What Works and What Doesn't. In Proceed- ings of the NAACL-HLT SRW, pages 47-54, San Diego, California, June 12-17, 2016. ACL. · DOI
  8. Omer Levy and Yoav Goldberg. 2014. Linguistic Reg- ularities in Sparse and Explicit Word Representa- tions. In Proceedings of the Eighteenth Confer- ence on Computational Natural Language Learning, pages 171-180, Ann Arbor, Michigan. Association for Computational Linguistics. · DOI
  9. Tal Linzen. 2016. Issues in Evaluating Semantic Spaces Using Word Analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computa- tional Linguistics. · DOI
  10. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Repre- sentations in Vector Space. In Proceedings of Inter- national Conference on Learning Representations (ICLR).
  11. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL- HLT 2013, pages 746-751, Atlanta, Georgia, 9-14 June 2013.
  12. Denis Newman-Griffis, Albert Lai, and Eric Fosler- Lussier. 2017. Insights into Analogy Completion from the Biomedical Domain. In BioNLP 2017, pages 19-28, Vancouver, Canada,. Association for Computational Linguistics. · DOI
  13. Malvina Nissim, Rik van Noord, and Rob van der Goot. 2020. Fair Is Better than Sensational: Man Is to Doc- tor as Woman Is to Doctor. Computational Linguis- tics, 46(2):487-497. · DOI
  14. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics. · DOI · Paper page
  15. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Ope- nAI Blog, 1(8):9.
  16. Anna Rogers, Aleksandr Drozd, and Bofang Li. 2017. The (Too Many) Problems of Analogical Reasoning with Word Vectors. In Proceedings of the 6th Joint Conference on Lexical and Computational Seman- tics (* SEM 2017), pages 135-148.
  17. Natalie Schluter. 2018. The Word Analogy Testing Caveat. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 242-246, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI
  18. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, AAAI'17, page 4444-4451. AAAI Press.
  19. Julien Tissier, Christophe Gravier, and Amaury Habrard. 2017. Dict2vec : Learning Word Em- beddings using Lexical Dictionaries. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 254-263, Copenhagen, Denmark. Association for Computa- tional Linguistics. · DOI
  20. Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. 2016. Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1671-1682, Berlin, Germany. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

See all 19

Metadata

Authors
Fournier, L., Dupoux, E., & Dunbar, E.
Affiliations
Laboratoire de Sciences Cognitives et Psycholinguistique (FR)
ORCID
Dunbar, Ewan
Venue
Proceedings of the 24th Conference on Computational Natural Language Learning
Publisher
Association for Computational Linguistics
Published
2020
Pages
365-375
DOI
10.18653/v1/2020.conll-1.29
OpenAlex
W3092565990
Cited by
19
References
1
Citations by year
2026: 4 · 2025: 2 · 2024: 1 · 2023: 3 · 2022: 4 · 2021: 5
FWCI
0.96
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Computational Natural Language Learning (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Microsoft Research
  • Canadian Institute for Advanced Research (ANR-17-EURE-0017)
  • Agence Nationale de la Recherche (ANR-10-LABX-0083, ANR-10-IDEX-0001, ANR-17-EURE-0017, ANR-17-CE28-0009, ANR-19-P3IA-0001, IDEX-0001-02, ANR-18-IDEX-0001)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
word embeddings, vector space model, linguistic regularities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review