The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
Open accessAbstract
We present SimLex-999, a gold standard resource for evaluating distributional semantic models that improves on existing resources in several important ways. First, in contrast to gold standards such as WordSim-353 and MEN, it explicitly quantifies similarity rather than association or relatedness so that pairs of entities that are associated but not actually similar (Freud, psychology) have a low rating. We show that, via this focus on similarity, SimLex-999 incentivizes the development of models with a different, and arguably wider, range of applications than those which reflect conceptual association. Second, SimLex-999 contains a range of concrete and abstract adjective, noun, and verb pairs, together with an independent rating of concreteness and (free) association strength for each pair. This diversity enables fine-grained analyses of the performance of models on concepts of different types, and consequently greater insight into how architectures can be improved. Further, unlike existing gold standard evaluations, for which automatic approaches have reached or surpassed the inter-annotator agreement ceiling, state-of-the-art models perform well below this ceiling on SimLex-999. There is therefore plenty of scope for SimLex-999 to quantify future improvements to distributional semantic models, guiding the development of the next generation of representation-learning architectures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Tversky, A. (1977) · Psychological Review · doi:10.1037/0033-295x.84.4.327 · Cited by 7,414
Landauer, T. K., & Dumais, S. T. (1997) · Psychological Review · doi:10.1037/0033-295x.104.2.211 · Cited by 6,282
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
NLTK: the Natural Language Toolkit
Loper, E., & Bird, S. (2002) · Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - · doi:10.3115/1118108.1118117 · Cited by 3,218
Using Information Content to Evaluate Semantic Similarity in a Taxonomy
Resnik, P. (1995) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9511007 · Cited by 3,193
Verbs semantics and lexical selection
Wu, Z., & Palmer, M. (1994) · Proceedings of the 32nd annual meeting on Association for Computational Linguistics - · doi:10.3115/981732.981751 · Cited by 3,136
Singular value decomposition and least squares solutions
Golub, G. H., & Reinsch, C. (1970) · Numerische Mathematik · doi:10.1007/bf02163027 · Cited by 2,929 · cited 2×
“As described by Landauer and Dumais (1997), we applied Singular Value Decomposition (SVD) (Golub and Reinsch, 1970) to the PMI-weighted VSM matrix, reducing the dimension of each concept representation to 300 (which yielded best re-sults after experimenting, as before, with 100-500 dimension vectors).”
MethodFrom Frequency to Meaning: Vector Space Models of Semantics
Turney, P. D., & Pantel, P. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2934 · Cited by 2,866
RCV1: A New Benchmark Collection for Text Categorization Research
Lewis, D., Yang, Y., Rose, T. G., & Li, F. (2004) · Goldsmiths (University of London) · doi:10.5555/1005332.1005345 · Cited by 2,503
Dual coding theory: Retrospect and current status.
Paivio, A. (1991) · Canadian Journal of Psychology / Revue canadienne de psychologie · doi:10.1037/h0084295 · Cited by 2,175
The University of South Florida free association, rhyme, and word fragment norms
Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195588 · Cited by 1,929
(2010) · Cited by 1,906
Word sense disambiguation: A survey
Navigli, R. (2009) · ACM Computing Surveys · doi:10.1145/1459352.1459355 · Cited by 1,819
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303322533223 · Cited by 1,566
Evaluating WordNet-based Measures of Lexical Semantic Relatedness
Budanitsky, A., & Hirst, G. (2006) · Computational Linguistics · doi:10.1162/089120106776173093 · Cited by 1,448
Contextual correlates of synonymy
Rubenstein, H., & Goodenough, J. B. (1965) · Communications of the ACM · doi:10.1145/365628.365657 · Cited by 1,425
Baroni, M., Dinu, G., & Kruszewski, G. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1023 · Cited by 1,364
Grounding conceptual knowledge in modality-specific systems
Barsalou, L. W., Kyle Simmons, W., Barbey, A. K., & Wilson, C. D. (2003) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(02)00029-3 · Cited by 1,256
Topics in semantic representation
Griffiths, T. L., Steyvers, M., & Tenenbaum, J. B. (2007) · Psychological Review · doi:10.1037/0033-295x.114.2.211 · Cited by 1,122
Improving Word Representations via Global Context and Multiple Word Prototypes
Huang, E., Socher, R., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,073
Dependency-Based Word Embeddings
Levy, O., & Goldberg, Y. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2050 · Cited by 1,047
Firth, J. R. (1957) · Medical Entomology and Zoology · Cited by 886
A study on similarity and relatedness using distributional and WordNet-based approaches
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., & Soroa, A. (2009) · Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics on - NAACL '09 · doi:10.3115/1620754.1620758 · Cited by 860
NLTK: The Natural Language Toolkit
Bird, S. (2006) · Proceedings of the COLING/ACL on Interactive presentation sessions - · doi:10.3115/1225403.1225421 · Cited by 841
Placing search in context: the concept revisited
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., & Ruppin, E. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372094 · Cited by 839
WordNet::Similarity: measuring the relatedness of concepts
Pedersen, T., Patwardhan, S., & Michelizzi, J. (2004) · Demonstration Papers at HLT-NAACL 2004 on XX - HLT-NAACL '04 · doi:10.3115/1614025.1614037 · Cited by 826
Better Word Representations with Recursive Neural Networks for Morphology
Luong, M.-T., Socher, R., & Manning, C. D. (2013) · Conference on Computational Natural Language Learning · Cited by 786
Phan, X.-H., Nguyen, L.-M., & Horiguchi, S. (2008) · Proceedings of the 17th international conference on World Wide Web · doi:10.1145/1367497.1367510 · Cited by 740
Distributional Memory: A General Framework for Corpus-Based Semantics
Baroni, M., & Lenci, A. (2010) · Computational Linguistics · doi:10.1162/coli_a_00016 · Cited by 659
On Relational Meaning: The Acquisition of Verb Meaning
Gentner, D. (1978) · Child Development · doi:10.1111/j.1467-8624.1978.tb04067.x · Cited by 552
Learning Subjective Adjectives from Corpora
Wiebe, J. (2000) · Cited by 551
Learning Concept Hierarchies from Text Corpora using Formal Concept Analysis
Cimiano, P., Hotho, A., & Staab, S. (2005) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1648 · Cited by 522
Structural bases of typicality effects.
Rosch, E. H., Simpson, C., & Miller, R. S. (1976) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.2.4.491 · Cited by 507
Integrating experiential and distributional data to learn semantic representations.
Andrews, M., Vigliocco, G., & Vinson, D. (2009) · Psychological Review · doi:10.1037/a0016261 · Cited by 454
Distributional Semantics in Technicolor
Bruni, E., Boleda, G., Baroni, M., & Tran, N. K. (2012) · Iris (University of Trento) · Cited by 415
Medelyan, O., Milne, D., Legg, C., & Witten, I. H. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.05.004 · Cited by 357
Multi-Prototype Vector-Space Models of Word Meaning
Reisinger, J., & Mooney, R. J. (2010) · Cited by 347
Learning Bilingual Lexicons from Monolingual Corpora
Haghighi, A., Liang, P., Berg-Kirkpatrick, T., & Klein, D. (2008) · Cited by 299
Tailoring Continuous Word Representations for Dependency Parsing
Bansal, M., Gimpel, K., & Livescu, K. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2131 · Cited by 290
Gentner, D. (2006) · Action Meets Word · doi:10.1093/acprof:oso/9780195170009.003.0022 · Cited by 260
The Reuters Corpus Volume 1 -from Yesterday’s News to Tomorrow’s Language Resources
Rose, T. G., Stevenson, M., & Whitehead, M. (2002) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5aeuc8zmytsy · Cited by 250
Metaphor Detection with Cross-Lingual Model Transfer
Tsvetkov, Y., Boytsov, L., Gershman, A., Nyberg, E., & Dyer, C. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1024 · Cited by 227
Learning Grounded Meaning Representations with Autoencoders
Silberer, C., & Lapata, M. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1068 · Cited by 217
Do Supervised Distributional Methods Really Learn Lexical Inference Relations?
Levy, O., Remus, S., Biemann, C., & Dagan, I. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1098 · Cited by 207
Semantic and Associative Priming in a Distributed Attractor Network
Plaut, D. C. (1995) · Cited by 203
Similar and different: The differentiation of basic-level categories.
Markman, A. B., & Wisniewski, E. J. (1997) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.23.1.54 · Cited by 195
A Systematic Study of Semantic Vector Space Model Parameters
Kiela, D., & Clark, S. (2014) · Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC) · doi:10.3115/v1/w14-1503 · Cited by 175
Domain and Function: A Dual-Space Model of Semantic Relations and Compositions
Turney, P. D. (2012) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3640 · Cited by 172
CLAWS4: the tagging of the British National Corpus
Leech, G., Garside, R., & Bryant, M. (1994) · Proceedings of the 15th conference on Computational linguistics - · doi:10.3115/991886.991996 · Cited by 169
Ontology Learning from Text: A Survey of Methods
Biemann, C. (2005) · Journal for Language Technology and Computational Linguistics · doi:10.21248/jlcl.20.2005.76 · Cited by 161
Recchia, G. L., & Jones, M. N. (2009) · Behavior Research Methods · doi:10.3758/brm.41.3.647 · Cited by 160
Improved statistical machine translation using monolingually-derived paraphrases
Marton, Y., Callison-Burch, C., & Resnik, P. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699510.1699560 · Cited by 148
Information Extraction, Automatic
Cunningham, H. (2006) · Elsevier eBooks · doi:10.1016/b0-08-044854-2/00960-3 · Cited by 146
Semantic Relatedness Using Salient Semantic Analysis
Hassan, S., & Mihalcea, R. F. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7971 · Cited by 142
Domain and function: A dual-space model of semantic relations and compositions
Turney, P. D. (2012) · Cited by 138
Exploring distributional similarity based models for query spelling correction
Li, M., Zhang, Y., Zhu, M., & Zhou, M. (2006) · International Conference on Computational Linguistics · doi:10.3115/1220175.1220304 · Cited by 116
Extending a Lexical Ontology by a Combination of Distributional Semantics Signatures
Alfonseca, E., & Manandhar, S. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45810-7_1 · Cited by 92
Improving Multi-Modal Representations Using Image Dispersion: Why Less is Sometimes More
Kiela, D., Hill, F., Korhonen, A., & Clark, S. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2135 · Cited by 87
Self-citationResnik, P., & Lin, J. (2010) · doi:10.1002/9781444324044.ch11 · Cited by 83
Multi-Modal Models for Concrete and Abstract Concept Meaning
Hill, F., Reichart, R., & Korhonen, A. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00183 · Cited by 75
Self-citationA Case Study on Inter-Annotator Agreement for Word Sense Disambiguation
Ng, H. T., Lim, C. Y., & Foo, S. K. (1999) · Cited by 66
Semantic and associative relations in adolescents and young adults: Examining a tenuous dichotomy.
McRae, K., Khalkhali, S., & Hare, M. (2012) · American Psychological Association eBooks · doi:10.1037/13493-002 · Cited by 55
A Quantitative Empirical Analysis of the Abstract/Concrete Distinction
Hill, F., Korhonen, A., & Bentz, C. (2013) · Cognitive Science · doi:10.1111/cogs.12076 · Cited by 48
Self-citationVerb Class Discovery from Rich Syntactic Data
Sun, L., Korhonen, A., & Krymolowski, Y. (2008) · Springer eBooks · doi:10.1007/978-3-540-78135-6_2 · Cited by 40
Self-citationA Mixture Model with Sharing for Lexical Semantics
Reisinger, J., & Mooney, R. J. (2010) · Cited by 39
Bruni, E., Uijlings, J., Baroni, M., & Sebe, N. (2012) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2393347.2396422 · Cited by 36
Flexible, Corpus-Based Modelling of Human Plausibility Judgements
Padó, S., Padó, U., & Erk, K. (2007) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 35
Semantic Parsing using Distributional Semantics and Probabilistic Logic
Beltagy, I., Erk, K., & Mooney, R. J. (2014) · doi:10.3115/v1/w14-2402 · Cited by 19
Concreteness and Corpora: A Theoretical and Practical Analysis
Hill, F., Kiela, D., & Korhonen, A. (2013) · Cited by 14
Self-citationObtaining Better Word Representations via Language Transfer
Li, C., Xu, B., Wu, G., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-642-54906-9_11 · Cited by 3
Cited by
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Improving Distributional Similarity with Lessons Learned from Word Embeddings
Levy, O., Goldberg, Y., & Dagan, I. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00134 · Cited by 1,308
Jin, D., Jin, Z., Zhou, J. T., & Szolovits, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6311 · Cited by 918
Ethayarajh, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1006 · Cited by 841
Metadata
- Authors
- Hill, F., Reichart, R., & Korhonen, A.
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2015-12
- Volume
- 41
- Issue
- 4
- Pages
- 665-695
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W1854884267
- Cited by
- 1,175
- References
- 72
- FWCI
- 136.70
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.