The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
Open accessAbstract
Context representations are central to various NLP tasks, such as word sense disambiguation, named entity recognition, coreference resolution, and many more.In this work we present a neural model for efficiently learning a generic context embedding function from large corpora, using bidirectional LSTM.With a very simple application of our context representations, we manage to surpass or nearly reach state-of-the-art results on sentence completion, lexical substitution and word sense disambiguation tasks, while substantially outperforming the popular context representation of averaged word embeddings.We release our code and pretrained models, suggesting they could be useful in a wide variety of NLP tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401
(2010) · Cited by 1,906
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
Socher, R., Pennington, J., Huang, E., Ng, A. Y., & Manning, C. D. (2011) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 1,431
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · cited 1×
“Finally, there is considerable work on using recurrent neural networks to represent word sequences, such as phrases or sentences (Socher et al., 2011; Kiros et al., 2015).”
BackgroundImproving Word Representations via Global Context and Multiple Word Prototypes
Huang, E., Socher, R., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,073
Dependency-Based Word Embeddings
Levy, O., & Goldberg, Y. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2050 · Cited by 1,047
Exploring the Limits of Language Modeling
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02410 · Cited by 823
Two/Too Simple Adaptations of Word2Vec for Syntax Problems
Ling, W., Dyer, C., Black, A. W., & Trancoso, I. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1142 · Cited by 384
End-to-end learning of semantic role labeling using recurrent neural networks
Zhou, J., & Xu, W. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1109 · Cited by 333
Tailoring Continuous Word Representations for Dependency Parsing
Bansal, M., Gimpel, K., & Livescu, K. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2131 · Cited by 290
A Unified Model for Word Sense Representation and Disambiguation
Chen, X., Liu, Z., & Sun, M. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1110 · Cited by 265
The SENSEVAL-3 English Lexical Sample Task
Mihalcea, R. F., Chklovski, T. A., & Kilgarriff, A. (2004) · University of North Texas Digital Library (University of North Texas) · Cited by 221
AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes
Rothe, S., & Schütze, H. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1173 · Cited by 203
Not All Contexts Are Created Equal: Better Word Representations with Variable Attention
Wang, L., Tsvetkov, Y., Amir, S., Fermandez, R., Dyer, C., Black, A. W., Trancoso, I. M., & Lin, C. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1161 · Cited by 147
Learning Semantic Word Embeddings based on Ordinal Knowledge Constraints
Liu, Q., Jiang, H., Wei, S., Ling, Z.-H., & Hu, Y. H. (2015) · doi:10.3115/v1/p15-1145 · Cited by 128
The Role of Context Types and Dimensionality in Learning Word Embeddings
Melamud, O., McClosky, D., Patwardhan, S., & Bansal, M. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1118 · Cited by 110
Self-citationWhat Substitutes Tell Us - Analysis of an "All-Words" Lexical Substitution Corpus
Kremer, G., Erk, K., Padó, S., & Thater, S. (2014) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1057 · Cited by 100
A Simple Word Embedding Model for Lexical Substitution
Melamud, O., Levy, O., & Dagan, I. (2015) · doi:10.3115/v1/w15-1501 · Cited by 97
Self-citationThe Microsoft Research Sentence Completion Challenge
Zweig, G., & Burges, C. J. C. (2011) · Cited by 71
Dependency Recurrent Neural Language Models for Sentence Completion
Mirowski, P., & Vlachos, A. (2015) · doi:10.3115/v1/p15-2084 · Cited by 61
Learning Syntactic Categories Using Paradigmatic Representations of Word Context
Yatbaz, M. A., Sert, E., & Yüret, D. (2012) · Empirical Methods in Natural Language Processing · Cited by 37
Learning to Represent Words in Context with Multilingual SupervisionPreprint
Kawakami, K., & Dyer, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04623 · Cited by 13
Cited by
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827
“Pre-trained contextualized word representations that can be trained from unlabeled text (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and…”
Method
Metadata
- Authors
- Melamud, O., Goldberger, J., & Dagan, I.
- Venue
- Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning
- Published
- 2016
- Pages
- 51-61
- OpenAlex
- W2507974895
- Cited by
- 513
- References
- 27
- FWCI
- 35.21
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.