The DOI matches its DataCite record; 63 of the 72 references it cites were found in a registry or an index.
Learned in Translation: Contextualized Word Vectors
Open accessAbstract
Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The increase of 4.7% with ELMo is also significantly larger then the 1.8% improvement from adding CoVe to a baseline model (McCann et al., 2017).”
From Deep Contextualized Word Representations (2018) · cites this paper as McCann et al., 2017
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For tasks where direct comparisons are possible, ELMo outperforms CoVe (McCann et al., 2017), which computes contextualized representations using a neural machine translation encoder.”
From Deep Contextualized Word Representations (2018) · cites this paper as McCann et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“[2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”
BackgroundMethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Qi et al. [2016] propose a CNN-based object tracking framework, which uses hierarchical features from a pretrained CNN (VGG-19 by Simonyan and Zisserman [2014]).”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“In computer vision, convolutional neural networks (CNNs) pretrained on ImageNet [Krizhevsky et al., 2012, Deng et al., 2009] have become the de facto initialization for more complex and deeper models.”
MethodSelf-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 14×
“The MT-LSTM is used to compute a sequence of hidden states h = MT-LSTM ( GloVe ( w x )) .”
BackgroundMethodInfluentialSelf-citationRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · cited 1×
“Girshick et al. [2014] show that using a pretrained CNN to extract features from region proposals improves object detection and semantic segmentation models.”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“…representation for all inputs. x pool = max( X | y ); mean( X | y ); min( X | y ); x self (15) We feed this joined representation through a three-layer, batch-normalized [Ioffe and Szegedy, 2015] maxout network [Goodfellow et al., 2013] to produce a probability distribution over possible classes.”
MethodEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Sutskever et al. [2014] investigate sequence-to-sequence models that consist of a neural network encoder and decoder for machine translation.”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“Bahdanau et al. [2015] propose the augmenting sequence to sequence models with an attention mechanism that gives the decoder access to the encoder representations of the input sequence at each step of sequence generation.”
Background
Research works printed but not linked: 6
- Q. Chen, X.-D. Zhu, Z.-H. Ling, S. Wei, and H. Jiang. Enhancing and combining sequential and tree LSTM for natural language inference. CoRR , abs/1609.06038, 2016. · arXiv
- A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P11-1015 .
- H. T. Madabushi and M. Lee. High accuracy rule-based question classification using question syntax and semantics. In COLING , 2016.
- V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML , 2010.
- L. Sha, B. Chang, Z. Sui, and S. Li. Reading and thinking: Re-read LSTM unit for textual entailment recognition. In COLING , 2016.
- Y. Yu, W. Zhang, K. Hasan, M. Yu, B. Xiang, and B. Zhou. End-to-end reading comprehension with dynamic answer chunk ranking. ICLR , 2017.
Cited by
Sorted by citations
Items 501–571 of 571
A Review of Machine Reading Comprehension
殷, 明. (2020) · Computer Science and Application · doi:10.12677/csa.2020.1012261 · Cited by 1
Multi-level spatial attention network for image data segmentation
Guo, J., Jiang, Z.-X., & Jiang, D. (2021) · International Journal of Embedded Systems · doi:10.1504/ijes.2021.116134 · Cited by 1
Stance Prediction of Tweets on Farmers Protests in India
Kamble, A., Badgujar, P., Kadam, A., Shah, D., & Kadam, A. (2022) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2022.43070 · Cited by 1
Towards Controlled Transformation of Sentiment in SentencesPreprint
Leeftink, W., & Spanakis, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11467 · Cited by 2
Contextual Salience for Fast and Accurate Sentence VectorsPreprint
Zelikman, E., & Socher, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08493 · Cited by 1
Self-citationGENERALIZED OPTIMAL LINEAR ORDERSPreprint
Bommasani, R. (2020) · arXiv (Cornell University) · doi:10.7298/5x0j-me63 · Cited by 2
Using BERT to identify drug-target interactions from whole PubMedPreprint
Aldahdooh, J., Vähä-Koskela, M. J. V., Tang, J. P., & Tanoli, Z. (2021) · Research Square · doi:10.21203/rs.3.rs-1015236/v1 · Cited by 1
Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization
Barba, J. Á. G. (2021) · doi:10.4995/thesis/10251/172245 · Cited by 1
Enhance Long Text Understanding via Distilled Gist Detector from Abstractive SummarizationPreprint
Liu, Y., & Yang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04741 · Cited by 5
Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems ResearchPreprint
Gruetzemacher, R., & Paradice, D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3944942 · Cited by 1
Portfolio Construction with News Sentiment using a Large Language ModelPreprint
Zhang, Q. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4454949 · Cited by 1
Epilogue: Frontiers of NLP in the Deep Learning Era
Deng, L., & Liu, Y. (2018) · Springer eBooks · doi:10.1007/978-981-10-5209-5_11 · Cited by 2
Transfer Pretrained Sentence Encoder to Sentiment Classification
Bai, M., Han, X., Jia, H., Wang, C., & Sun, Y. (2018) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc.2018.00068
Learning to Compose over Tree Structures via POS TagsPreprint
Shen, G., Deng, Z., Huang, T., & Chen, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.06075
A Multiple Source based Transfer Learning Framework for Marketing Campaigns
Brownlow, J., Chu, C., Xu, G., Culbert, B., Fu, B., & Meng, Q. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489772
Text Classification with Keywords and Co-occurred Words in Two-stream Neural Network
Deng, J., & Ren, F. (2018) · IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS) · doi:10.1109/ccis.2018.8691302
What does a Car-ssette tape tell?Preprint
Xu, X., Dinkel, H., Wu, M., & Yu, K. (2019) · arXiv (Cornell University)
Towards Language Agnostic Universal RepresentationsPreprint
Aghajanyan, A., Song, X., & Tiwary, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.08510
Kim, T., Choi, J., & Lee, S. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1182 · Cited by 2
A Dynamic Word Representation Model Based on Deep Context
Xiao, Y., Xiong, X., Ju, S., Xie, Z., & Wang, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32236-6_60
Neural architectures for natural language understanding
Tay, Y. (2019) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/83291
Ruminating Word Representations with Random Noise MaskingPreprint
Jo, H., & Zhang, B. (2021) · arXiv (Cornell University)
Using Feature Filtering Metrics as Meta-dimensions in Constructing Distributional Representations
Yang, D., Yin, Y., Han, T., & Ma, H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-35288-2_27
Binarized PMI Matrix: Bridging Word Embeddings and Hyperbolic Spaces.Preprint
Assylbekov, Z., & Jangeldin, A. (2020) · arXiv (Cornell University)
Adaptação lexical automática em textos informativos para o Ensino Fundamental
Hartmann, N. S. (2020) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2020.tde-29072020-161751
Clue Extraction for Fine-Grained Emotion Analysis
Bi, H., & Liu, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60450-9_61
Improving Image Similarity Learning by Adding External Memory
Gao, X., Mu, T., Goulermas, J. Y., Song, J., & Wang, M. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3047104
Automated Graph Generation at Sentence Level for Reading Comprehension Based on Conceptual Graphs
Lin, W.-H., & Lu, C.-S. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.240
Semantic Dependency Word Representation
Fu, C., Hangzhao, Zhang, S., & Zhou, L. (2020) · International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · doi:10.1109/iccwamtip51612.2020.9317476
Architecture of machine reading comprehension models
Zhu, C. (2021) · Elsevier eBooks · doi:10.1016/b978-0-323-90118-5.00004-7
Customer Satisfaction Research based on Customer Service Dialogue Corpus
Chai, J., Wang, S., Zhu, J., Mao, X.-L., & Huang, H. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1924/1/012013
Combining Weakly Supervised ML Techniques for Low-Resource NLU
Soto, V., & Arkoudas, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-industry.36
Multi-level spatial attention network for image data segmentation
Jiang, Z.-X., Jiang, D., & Guo, J. (2021) · International Journal of Embedded Systems · doi:10.1504/ijes.2021.10039166
LV-BERT: Exploiting Layer Variety for BERTPreprint
Yu, W., Jiang, Z., Chen, F., Hou, Q., & Feng, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11740
LV-BERT: Exploiting Layer Variety for BERT
Yu, W., Jiang, Z., Chen, F., Hou, Q., & Feng, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.2 · Cited by 1
BIPRE: Bi-directional Inter-Personal Relationship Extraction Task
Tian, Y., Liu, P., & Du, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-81197-6_54
Development of Digital Storytelling Platform for Children based on Emotions
Vijayakumaran, T., Thavachelvam, T., Gnaeswaran, A., Sandanayake, T., Sumathipala, K., & Bandara, S. (2021) · International Conference on Information Technology Research (ICITR) · doi:10.1109/icitr54349.2021.9657431 · Cited by 1
Multitask Learning as Question Answering with BERT
Roy, S., Ehtesham, N., Islam, M. S., & Ismail, S. (2021) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit54785.2021.9689900
Abrechnung medizinischer Leistungen mit künstlicher Intelligenz
Landgrebe, J. (2022) · Springer eBooks · doi:10.1007/978-3-658-33597-7_34
Couto, F. M. (2021) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.736581786.793583871
Qi, X., Gao, Y., Wang, R., Zhao, M., Cui, S., & Mortazavi, M. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/8407713 · Cited by 1
Research on text classification model based on ERNIE2.0-DICNN
Gu, Y., Yao, X., & Luo, S. (2022) · International Conference on Advanced Algorithms and Signal Image Processing (AASIP) · doi:10.1117/12.2659644
Impact of Grammar on Language Model Comprehension
Ameri, K., Hempel, M., Sharif, H., Lopez, J., & Perumalla, K. S. (2023) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/icnc57223.2023.10074239
Representation Learning for Texts and Graphs
Galke, L. P. A. (2023) · doi:10.21941/kcss/2023/1
Dore, G. M. D., McCarthy, A. D., & Scharf, J. A. (2023) · SpringerBriefs in political science · doi:10.1007/978-3-031-27584-5_4
Data-efficient domain adaptation for pretrained language models
Guo, X. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/167965
Ahmed, S. (2023) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · doi:10.21248/gups.74395
Arquitetura da Informação Multimodal: contribuições no desenvolvimento de Inteligência Artificial
Jùnior, G. H. K., & Duque, C. G. (2023) · Transinformação · doi:10.1590/2318-0889202335e226729 · Cited by 3
Research on quality estimation of Chinese-English translation based on machine learning
Han, Y., & Zhao, C. (2023) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1117/12.3003922
On Enhancing Fine-Tuning for Pre-trained Language Models
Betka, A., Ferhat, Z., Barka, R., Boutiba, S., Kahhoul, Z. S., Tiar, M. L., Abdelalí, A., & Dahmani, H. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.33 · Cited by 1
Disentangling the Linguistic Competence of Privacy-Preserving BERT
Arnold, S., Kemmerzell, N., & Schreiner, A. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.5
Modal Identification of Aircraft Time-varying Structure Based on Fiber Grating Measurement
Wang, Y., Yue, Z., & Liu, L. (2023) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2658/1/012014
MQDD: Pre-training of Multimodal Question Duplicity Detection for Software Engineering Domain
Pašek, J., Sido, J., Konopík, M., & Pražák, O. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_089 · Cited by 3
Orphan Entity Allocation Using Entity Chunking, Context Mining, and Semantic Networks for CV-Parsing
Chitale, J., Kamthe, S., & Khatavkar, V. K. (2023) · Asia Meeting on Environment and Electrical Engineering (EEE-AM) · doi:10.1109/eee-am58328.2023.10395421
Advancing COVID-19 Inquiry Responses Through Transfer Learning-Based Question Entailment Methodology
Godslove, J. F., & Nayak, A. K. (2023) · International Conference on Ambient Intelligence in Health Care (ICAIHC) · doi:10.1109/icaihc59020.2023.10431471
Heterogeneous Graphs for Text Representation: An Integrated Approach with Language Models
dos Santos, B. N. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2023.tde-25032024-112903
FUZZY INTERACTIVE CLUSTERING METHOD
Steblyanko, B., Ni, O., Kuchuk, H., et al. (2024) · Системи управління навігації та зв’язку Збірник наукових праць · doi:10.26906/sunz.2024.2.152
Filho, I. J. D. R. (2024) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2024.tde-13062024-143337
ConSentiment: Análise de Sentimentos baseada em Aspectos a partir de Consenso de Modelos em Grafos
Marcon, G. D. S. (2024) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.55.2024.tde-17062024-085235
Shi, B., Liu, M., Zhou, J., et al. (2024) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd61410.2024.10580254
Universal detection method for mitigating adversarial text attacks through token loss information
Chen, Y., Du, X., Wang, D., et al. (2024) · Journal of Image and Graphics · doi:10.11834/jig.230432
Exploring Social Media Sentiment Polarity: A Case Study Utilizing Language Models
Yang, B., & Johnson-Shapoval, S. (2024) · IEEE International Conference on Software Engineering and Artificial Intelligence (SEAI) · doi:10.1109/seai62072.2024.10674171
-, D. B., -, K. S., -, A. B., & -, S. M. A. H. (2024) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2024.v06i06.32271
Improving long text classification based on Selective State Space model (Mamba)
Sarem, M., Jurdi, T., Albshlawy, L., & Massrie, E. (2024) · IEEE International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC) · doi:10.1109/mcsoc64144.2024.00015
Zheng, J. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100409
RetractedParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270
“To encode context in these representations, features are extracted from internal representations of sequence models, such as MT systems (McCann et al., 2017), and BiLSTM language models, as used in ELMo (Peters et al.”
MethodThe Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448
“Intermediate representations from supervised machine translation models improve performance on question answering, sentiment analysis, and natural language inference (McCann et al., 2017).”
BackgroundMethodInfluentialSelf-citationBackdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation
Liao, C., Zhong, H., Squicciarini, A., Zhu, S., & Miller, D. (2018) · arXiv · doi:10.48550/arxiv.1808.10307 · Cited by 110
“ral language processing (NLP) [35], automatic speech recognition (ASR) [1], etc.”
Background
Metadata
- Authors
- McCann, B., Bradbury, J., Xiong, C., & Socher, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 30
- Pages
- 6297-6308
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1708.00107
- OpenAlex
- W2740711318
- Semantic Scholar
- bc8fa64625d9189f5801837e7b133e7fe3c581f7
- Cited by
- 572
- References
- 72
- Citations by year
- 2025: 14 · 2024: 39 · 2023: 65 · 2022: 57 · 2021: 98 · 2020: 94 · 2019: 122 · 2018: 80 · 2017: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- coves, pre-trained word embeddings, machine translation, LSTM encoder, sentiment analysis, logical consequence, question answering, question classification, transfer learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
