Improving Vector Space Word Representations Using Multilingual Correlation
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
The distributional hypothesis of Harris (1954), according to which the meaning of words is evidenced by the contexts they occur in, has motivated several effective techniques for obtaining vector space semantic representations of words using unannotated text corpora. This paper argues that lexico-semantic content should additionally be invariant across languages and proposes a simple technique based on canonical correlation analysis (CCA) for incorporating multilingual evidence into vectors generated monolingually. We evaluate the resulting word representations on standard lexical semantic evaluation tasks and show that our method produces substantially better semantic representations than monolingual techniques.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Besides, Hewitt and Manning [125], Jawahar et al. [126], Kim et al. [127] proposed several methods to extract dependency trees and constituency trees from BERT, which proved the BERT's ability to encode syntax structure.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [125]
Citations
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 6×
“This dataset contains word pairs that are different syntactic forms of a given word and was prepared by Mikolov et al. (2013a).”
BackgroundMethodInfluentialIndexing by latent semantic analysis
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6<391::aid-asi1>3.0.co;2-9 · Cited by 12,751
Golub, G. H., & Loan, C. F. V. (1996) · Johns Hopkins University Press eBooks · Cited by 6,027
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“Our method can also be seen as an application of multi-view learning (Chang et al., 2013; Collobert and Weston, 2008), where one of the views can be used to capture cross-lingual information.”
BackgroundMethodInfluentialHarris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840 · cited 2×
“01) according to the McNemar’s test (Dietterich, 1998).”
MethodInfluentialWord association norms, mutual information, and lexicography
Church, K., & Hanks, P. (1990) · doi:10.5555/89086.89095 · Cited by 3,590
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
From Frequency to Meaning: Vector Space Models of Semantics
Turney, P. D., & Pantel, P. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2934 · Cited by 2,866 · cited 1×
“Using cooccurrence statistics from a large corpus of text (Deerwester et al., 1990; Turney and Pantel, 2010), 1 it is possible to construct high-quality semantic vectors — as judged by both correlations with human judgements of semantic relatedness (Turney, 2006; Agirre et al., 2009) and as…”
Background(2010) · Cited by 1,906
A Synopsis of Linguistic Theory, 1930-1955
Firth, J. R. (1957) · Medical Entomology and Zoology · Cited by 1,675
Contextual correlates of semantic similarity
Miller, G. A., & Charles, W. G. (1991) · Language and Cognitive Processes · doi:10.1080/01690969108406936 · Cited by 1,594 · cited 3×
“The second and third benchmarks are the RG-65 (Rubenstein and Goodenough, 1965) and the MC-30 (Miller and Charles, 1991) datasets that contain 65 and 30 pairs of nouns respectively and have been given similarity rankings by humans.”
BackgroundContextual correlates of synonymy
Rubenstein, H., & Goodenough, J. B. (1965) · Communications of the ACM · doi:10.1145/365628.365657 · Cited by 1,425 · cited 3×
“The second and third benchmarks are the RG-65 (Rubenstein and Goodenough, 1965) and the MC-30 (Miller and Charles, 1991) datasets that contain 65 and 30 pairs of nouns respectively and have been given similarity rankings by humans.”
BackgroundMining the Web for Synonyms: PMI-IR versus LSA on TOEFL
Turney, P. D. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44795-4_42 · Cited by 1,285 · cited 1×
“We then replace every entry in the sparse frequency matrix F by its pointwise mutual information (PMI) (Church and Hanks, 1990; Turney, 2001) resulting in X .”
MethodImproving Word Representations via Global Context and Multiple Word Prototypes
Huang, E., Socher, R., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,073
A study on similarity and relatedness using distributional and WordNet-based approaches
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., & Soroa, A. (2009) · Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics on - NAACL '09 · doi:10.3115/1620754.1620758 · Cited by 860 · cited 2×
“This data was further divided into two fragments by Agirre et al. (2009) who claimed that similarity (WS-SIM) and relatedness (WS-REL) are two different kinds of relations and should be dealt with separately.”
BackgroundPlacing search in context: the concept revisited
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., & Ruppin, E. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372094 · Cited by 839 · cited 1×
“The first one is the WS-353 dataset (Finkelstein et al., 2001) containing 353 pairs of English words that have been assigned similarity ratings by humans.”
BackgroundA Scalable Hierarchical Distributed Language Model
Mnih, A., & Hinton, G. E. (2008) · Cited by 838
(2005) · Cited by 823
Paraphrasing with bilingual parallel corpora
Bannard, C., & Callison-Burch, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219914 · Cited by 533
Bilingual Word Embeddings for Phrase-Based Machine Translation
Zou, W. Y., Socher, R., Cer, D., & Manning, C. D. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1141 · Cited by 518 · cited 1×
“Our method of learning multilingual word vectors is most closely associated to Zou et al. (2013) who learn bilingual word embeddings and show their utility in machine translation.”
MethodResearch Design & Statistical Analysis
Myers, J. L., & Well, A. D. (2003) · Psychology Press eBooks · doi:10.4324/9781410607034 · Cited by 407 · cited 1×
“We then report Spearman’s rank correlation coefficient (Myers and Well, 1995) between the rankings produced by our model against the human rankings.”
MethodA word at a time: computing word relatedness using temporal semantic analysis
Radinsky, K., Agichtein, E., Gabrilovich, E., & Markovitch, S. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963455 · Cited by 405 · cited 1×
“The fourth benchmark is the MTurk-287 (Radinsky et al., 2011) dataset that constitutes of 287 pairs of words and is different from the above two benchmarks in that it has been constructed by crowdsourcing the human similarity ratings using Amazon Mechanical Turk.”
MethodInducing Crosslingual Distributed Representations of Words
Klementiev, A., Titov, I., & Bhattarai, B. (2012) · International Conference on Computational Linguistics · Cited by 389
Similarity of Semantic RelationsPreprint
Turney, P. D. (2006) · CogPrints (University of Southampton) · Cited by 359
Bootstrapping parsers via syntactic projection across parallel texts
HWA, R., RESNIK, P., WEINBERG, A., CABEZAS, C., & KOLAK, O. (2005) · Natural Language Engineering · doi:10.1017/s1351324905003840 · Cited by 340 · cited 1×
“…pairs can be treated as evidence that semantics or information at a conceptual level (since both of them basically model word cooccurrence counts) transfers well across languages (Dyvik, 2004) although syntax has been projected across languages as well (Hwa et al., 2005; Yarowsky and Ngai, 2001).”
BackgroundLearning Bilingual Lexicons from Monolingual Corpora
Haghighi, A., Liang, P., Berg-Kirkpatrick, T., & Klein, D. (2008) · Cited by 299
Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora
Yarowsky, D., & Ngai, G. (2001) · Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001 - NAACL '01 · doi:10.3115/1073336.1073362 · Cited by 242 · cited 1×
“…pairs can be treated as evidence that semantics or information at a conceptual level (since both of them basically model word cooccurrence counts) transfers well across languages (Dyvik, 2004) although syntax has been projected across languages as well (Hwa et al., 2005; Yarowsky and Ngai, 2001).”
BackgroundMulti-View Learning of Word Embeddings via CCA
Dhillon, P. S., Foster, D. P., & Ungar, L. (2011) · ScholarlyCommons (University of Pennsylvania) · Cited by 241
RESNIK, P., & YAROWSKY, D. (1999) · Natural Language Engineering · doi:10.1017/s1351324999002211 · Cited by 231 · cited 1×
“…information and should lead to better VSMs. Parallel corpora have long been recognized as valuable for lexical semantic applications, including identifying word senses (Diab, 2003; Resnik and Yarowsky, 1999) and paraphrase and synonymy relationships (Bannard and Callison-Burch, 2005).”
BackgroundDyer, C., Lopez, A., Ganitkevitch, J., Weese, J., Setiawan, H., Türe, F., Eidelman, V., Blunsom, P., & Resnik, P. (2010) · Cited by 224
Self-citationCross-lingual Word Clusters for Direct Transfer of Linguistic Structure
Täckström, O., McDonald, R., & Uszkoreit, J. (2012) · Cited by 217
Multilingual Topic Models for Unaligned TextPreprint
Boyd-Graber, J. L., & Blei, D. M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2657 · Cited by 148 · cited 6×
“The untransformed LSA vectors are in the upper part of Fig.”
MethodInfluentialBiTAM: bilingual topic AdMixture models for word alignment
Zhao, B., & Xing, E. P. (2006) · International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics (COLING/ACL) · doi:10.3115/1273073.1273197 · Cited by 91 · cited 2×
“Bilingual representations have previously been explored with manually designed vector space models (Peirsman and Padó, 2010; Sumita, 2000) and with unsupervised algorithms like LDA and LSA (Boyd-Graber and Blei, 2012; Zhao and Xing, 2006).”
MethodTranslations as semantic mirrors: from parallel corpus to wordnet
Dyvik, H. (2004) · BRILL eBooks · doi:10.1163/9789004333710_019 · Cited by 90 · cited 1×
“…pairs can be treated as evidence that semantics or information at a conceptual level (since both of them basically model word cooccurrence counts) transfers well across languages (Dyvik, 2004) although syntax has been projected across languages as well (Hwa et al., 2005; Yarowsky and Ngai, 2001).”
BackgroundMulti-Relational Latent Semantic Analysis
Chang, K.-W., Yih, W.-T., & Meek, C. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1167 · Cited by 61 · cited 2×
“Our method can also be seen as an application of multi-view learning (Chang et al., 2013; Collobert and Weston, 2008), where one of the views can be used to capture cross-lingual information.”
MethodA Study on Bootstrapping Bilingual Vector Spaces from Non-Parallel Data (and Nothing Else)
Vulić, I., & Moens, M. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1168 · Cited by 43 · cited 1×
“Vulić and Moens (2013) learn bilingual vector spaces from non parallel data induced using a seed lexicon.”
BackgroundWORD SENSE DISAMBIGUATION WITHIN A MULTILINGUAL FRAMEWORK
Diab, M. (2003) · University Libraries (University of Maryland) · doi:10.13016/ny9e-yqwo · Cited by 34 · cited 1×
“Parallel corpora have long been recognized as valuable for lexical semantic applications, including identifying word senses (Diab, 2003; Resnik and Yarowsky, 1999) and paraphrase and synonymy relationships (Bannard and CallisonBurch, 2005).”
BackgroundCross-lingual Induction of Selectional Preferences with Bilingual Vector Spaces
Peirsman, Y., & Padó, S. (2010) · Lirias · Cited by 29
An Information Theoretic Approach to Bilingual Word Clustering
Faruqui, M., & Dyer, C. (2013) · Cited by 11
Self-citationBilingual word spectral clustering for statistical machine translation
Zhao, B., Xing, E. P., & Waibel, A. (2005) · doi:10.3115/1654449.1654454 · Cited by 10 · cited 2×
“Bilingual evidence has also been exploited for word clustering which is yet another form of representation learning, using both spectral methods (Zhao et al., 2005) and structured prediction approaches (Täckström et al., 2012; Faruqui and Dyer, 2013).”
MethodLexical transfer using a vector-space model
Sumita, E. (2000) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1075218.1075272 · Cited by 8 · cited 1×
“Bilingual representations have previously been explored with manually designed vector space models (Peirsman and Padó, 2010; Sumita, 2000) and with unsupervised algorithms like LDA and LSA (Boyd-Graber and Blei, 2012; Zhao and Xing, 2006).”
Method
Cited by
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“tions shared across languages plays an important role in many cross-lingual NLP tasks. Cross-Lingual Language Understanding (XLU) Most of the early works focus on learning multilingual word embedding [45, 123, 158], which represents text from multiple languages in a single semantic space. However, these methods usuall…”
BackgroundCross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
“A series of follow-up studies show that cross-lingual representations can be used to improve the quality of monolingual representations [16], that orthogonal transformations are sufficient to align these word distributions [43], and that all these techniques can be applied to an arbitrary number of languages [2].”
BackgroundWord Translation Without Parallel Data
Conneau, A., Lample, G., Ranzato, M., Denoyer, L., & Jeǵou, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.04087 · Cited by 219
“Since then, several works aimed at improving these cross-lingual word embeddings (Faruqui & Dyer (2014); Xing et al. (2015); Baroni et al. (2015); Ammar et al. (2016); Artetxe et al. (2016); Smith et al. (2017)), but they all rely on bilingual word lexicons.”
BackgroundOn Deep Multi-View Representation Learning
Wang, W., Livescu, K., & Bilmes, J. (2015) · Cited by 871
Retrofitting Word Vectors to Semantic Lexicons
Faruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E., & Smith, N. A. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1184 · Cited by 721
Self-citation
Metadata
- Authors
- Faruqui, M., & Dyer, C.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics
- Published
- 2014
- Pages
- 462-471
- OpenAlex
- W342285082
- Cited by
- 620
- References
- 43
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 33 · 2023: 19 · 2022: 28 · 2021: 57 · 2020: 106 · 2019: 100 · 2018: 91 · 2017: 61 · 2016: 67 · 2015: 33 · 2014: 14
- FWCI
- 34.35
- Type
- conference paper
- Language
- en
- Open access
- green
- figshare.com
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion)
- KiltHub Repository (submittedVersion)
- Figshare (submittedVersion, cc-by)
- KiltHub Repository
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- word embeddings, multilingual word representation, distributional semantics, canonical correlation analysis, word meaning, word sense disambiguation, word similarity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.