CoChat
Sign Up
PreprintCited by 572Topic ModelingSentiment Analysis and Opinion MiningNatural Language Processing Techniques

The DOI matches its DataCite record; 63 of the 72 references it cites were found in a registry or an index.

Learned in Translation: Contextualized Word Vectors

Open access
McCann, Bryan, Bradbury, James, Xiong, Caiming, Socher, Richard·2017·arXiv
DOI 10.48550/arxiv.1708.00107arXiv 1708.00107
Open this paper in CoChat — free accountOpen PDF

Abstract

Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1708.00107

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Deep Residual Learning for Image Recognition

    He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×

    “[2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”
    BackgroundMethod

  • Very Deep Convolutional Networks for Large-Scale Image Recognition

    Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×

    “Qi et al. [2016] propose a CNN-based object tracking framework, which uses hierarchical features from a pretrained CNN (VGG-19 by Simonyan and Zisserman [2014]).”
    Method

  • ImageNet classification with deep convolutional neural networks

    Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155

  • ImageNet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×

    “In computer vision, convolutional neural networks (CNNs) pretrained on ImageNet [Krizhevsky et al., 2012, Deng et al., 2009] have become the de facto initialization for more complex and deeper models.”
    MethodSelf-citation

  • Glove: Global Vectors for Word Representation

    Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 14×

    “The MT-LSTM is used to compute a sequence of hidden states h = MT-LSTM ( GloVe ( w x )) .”
    BackgroundMethodInfluentialSelf-citation

  • Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

    Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · cited 1×

    “Girshick et al. [2014] show that using a pretrained CNN to extract features from region proposals improves object detection and semantic segmentation models.”
    Method

  • Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

    Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×

    “…representation for all inputs. x pool = max( X | y ); mean( X | y ); min( X | y ); x self (15) We feed this joined representation through a three-layer, batch-normalized [Ioffe and Szegedy, 2015] maxout network [Goodfellow et al., 2013] to produce a probability distribution over possible classes.”
    Method

  • Efficient Estimation of Word Representations in Vector Space

    Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568

  • Sequence to Sequence Learning with Neural Networks

    Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×

    “Sutskever et al. [2014] investigate sequence-to-sequence models that consist of a neural network encoder and decoder for machine translation.”
    Background

  • Neural Machine Translation by Jointly Learning to Align and Translate

    Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×

    “Bahdanau et al. [2015] propose the augmenting sequence to sequence models with an attention mechanism that gives the decoder access to the encoder representations of the input sequence at each step of sequence generation.”
    Background

  • Effective Approaches to Attention-based Neural Machine Translation

    Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1×

    “Luong et al. [2015] further study the effectiveness of various attention mechanisms with respect to machine translation.”
    Background

  • Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 2×

    “Recent work in NLP has continued in this direction by using pretrained word representations to improve models for entailment [Bowman et al., 2014], sentiment analysis [Socher et al., 2013], summarization [Nallapati et al., 2016], and question answering [Seo et al., 2017, Xiong et al., 2017].”
    MethodSelf-citation

  • SQuAD: 100,000+ Questions for Machine Comprehension of Text

    Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208

  • Framewise phoneme classification with bidirectional LSTM and other neural network architectures

    Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 6,152 · cited 1×

    “We feed GloVe( w x ) to a standard, two-layer, bidirectional, long short-term memory network 1 [Graves and Schmidhuber, 2005] that we refer to as an MT-LSTM to indicate that it is this same two-layer BiLSTM that we later transfer as a pretrained encoder.”
    Method

  • Moses: open source toolkit for statistical machine translation

    Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735 · cited 1×

    “Each is tokenized using the Moses Toolkit [Koehn et al., 2007].”
    Method

  • A large annotated corpus for learning natural language inference

    Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 4×

    “For entailment, we use the Stanford Natural Language Inference Corpus (SNLI) [Bow-man et al., 2015], which has 550,152 training, 10,000 validation, and 10,000 testing examples.”
    MethodInfluential

  • Adapting visual category models to new domains

    Saenko, K., Kulis, B., Fritz, M., & Darrell, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_16 · Cited by 2,847 · cited 1×

    “Saenko et al. [2010] adapt object recognition models developed for one visual domain to new imaging conditions by learning a transformation that minimizes domain-induced changes in the feature distribution.”
    Background

  • Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

    Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361 · cited 4×

    “Recent work in NLP has continued in this direction by using pretrained word representations to improve models for entailment [Bowman et al., 2014], sentiment analysis [Socher et al., 2013], summarization [Nallapati et al., 2016], and question answering [Seo et al., 2017, Xiong et al., 2017].”
    BackgroundInfluential

  • Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

    Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1070 · Cited by 2,049 · cited 2×

    “Attention mechanisms have also been successfully applied to NLP tasks like entailment [Conneau et al., 2017], summarization [Nallapati et al.”
    Background

  • OpenNMT: Open-Source Toolkit for Neural Machine Translation

    Klein, G., Kim, Y., Deng, Y., Sénellart, J., & Rush, A. M. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4012 · Cited by 1,916 · cited 1×

    “We begin by training an attentional sequence-to-sequence model for English-to-German translation based on Klein et al. [2017] with the goal of transferring the encoder to other tasks.”
    Method

  • Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning

    Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.345 · Cited by 1,724 · cited 2×

    “For image captioning, Lu et al. [2016] train a visual sentinel with a pretrained CNN and fine-tune the model with a smaller learning rate.”
    MethodSelf-citation

  • Bidirectional Attention Flow for Machine ComprehensionPreprint

    Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 3×

    “Recent work in NLP has continued in this direction by using pretrained word representations to improve models for entailment [Bowman et al., 2014], sentiment analysis [Socher et al., 2013], summarization [Nallapati et al., 2016], and question answering [Seo et al., 2017, Xiong et al., 2017].”
    BackgroundMethod

  • Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

    Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · cited 1×

    “For VQA, Fukui et al. [2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”
    Method

  • A Decomposable Attention Model for Natural Language Inference

    Parikh, A. P., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1244 · Cited by 1,351

  • Skip-Thought Vectors

    Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293

  • Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

    Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., & Socher, R. (2015) · arXiv · doi:10.48550/arxiv.1506.07285 · Cited by 1,020 · cited 1×

    “MVN [Guo et al., 2017] 51.5 T R E C - 50 SVM [Loni et al., 2011] 89.0 DMN [Kumar et al., 2016] 52.1 SNoW [Li and Roth, 2006] 89.3 LSTM-CNN [Zhou et al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53…”
    InfluentialSelf-citation

  • Semi-supervised Sequence Learning

    Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · cited 1×

    “…al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53 [Dai and Le, 2015] 92.8 NTI [Munkhdalai and Yu, 2016b] 87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et…”
    Influential

  • Maxout Networks

    Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1302.4389 · Cited by 935 · cited 1×

    “…representation for all inputs. x pool = max( X | y ); mean( X | y ); min( X | y ); x self (15) We feed this joined representation through a three-layer, batch-normalized [Ioffe and Szegedy, 2015] maxout network [Goodfellow et al., 2013] to produce a probability distribution over possible classes.”
    Method

  • Grounded Compositional Semantics for Finding and Describing Images with Sentences

    Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 794 · cited 1×

    “This initialization improves accuracy on other related tasks such as visual question answering [Xiong et al., 2016] or image captioning [Lu et al., 2016, Socher et al., 2014].”
    MethodSelf-citation

  • Dynamic Memory Networks for Visual and Textual Question AnsweringPreprint

    Xiong, C., Merity, S., & Socher, R. (2016) · International Conference on Machine Learning · doi:10.48550/arxiv.1603.01417 · Cited by 770 · cited 4×

    “We did not submit the SQuAD model for testing, but the addition of CoVe was enough to push the validation performance of the original DCN, which already used character n-gram embeddings, above the validation performance of the published version of the R-NET.”
    MethodInfluentialSelf-citation

  • Hedged Deep Tracking

    Qi, Y., Zhang, S., Qin, L., Yao, H., Huang, Q., Lim, J., & Yang, M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.466 · Cited by 735 · cited 2×

    “Qi et al. [2016] propose a CNN-based object tracking framework, which uses hierarchical features from a pretrained CNN (VGG-19 by Simonyan and Zisserman [2014]).”
    MethodInfluential

  • Adversarial Training Methods for Semi-Supervised Text ClassificationPreprint

    Miyato, T., Dai, A. M., & Goodfellow, I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07725 · Cited by 713 · cited 1×

    “…87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et al., 2016] 93.8 btree-LSTM [Paria et al., 2016] 87.6 oh-LSTM [Johnson and Zhang, 2016] 94.1 600D ESIM [Chen et al., 2016] 88.0 Virtual [Miyato et al., 2017] 94.1 BCN+Char+CoVe (Ours) 88.1 Test Performance.”
    Influential

  • Language to Logical Form with Neural Attention

    Dong, L., & Lapata, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1004 · Cited by 700 · cited 1×

    “Attention mechanisms have also been successfully applied to NLP tasks like entailment [Conneau et al., 2017], summarization [Nallapati et al., 2016], question answering [Seo et al., 2017, Xiong et al., 2017, Min et al., 2017], and semantic parsing [Dong and Lapata, 2016].”
    Background

  • Gated Self-Matching Networks for Reading Comprehension and Question Answering

    Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1018 · Cited by 674

  • Learning Distributed Representations of Sentences from Unlabelled Data

    Hill, F., Cho, K., & Korhonen, A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1162 · Cited by 566 · cited 1×

    “Hill et al. [2016] study the effect of transferring from a variety of source domains to the semantic similarity tasks in Agirre et al. [2014].”
    Background

  • A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

    Hashimoto, K., xiong, C., Tsuruoka, Y., & Socher, R. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1206 · Cited by 535 · cited 1×

    “For VQA, Fukui et al. [2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”
    MethodSelf-citation

  • Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max PoolingPreprint

    Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., & Xu, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06639 · Cited by 534 · cited 2×

    “…89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8 LSTM-CNN [Zhou et al., 2016] 96.1”
    Influential

  • SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

    Agirre, E., Banea, C., Cardie, C., Cer, D. M., Diab, M., González-Agirre, A., Guo, W., Mihalcea, R. F., Rigau, G., & Wiebe, J. (2014) · International Workshop on Semantic Evaluation (SemEval) · doi:10.3115/v1/s14-2010 · Cited by 511 · cited 1×

    “Hill et al. [2016] study the effect of transferring from a variety of source domains to the semantic similarity tasks in Agirre et al. [2014].”
    Background

  • Dynamic Coattention Networks For Question AnsweringPreprint

    Xiong, C., Zhong, V., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01604 · Cited by 504 · cited 7×

    “In order to test the transferability of these encoders, we develop a common architecture for a variety of classification tasks, and we modify the Dynamic Coattention Network for question answering [Xiong et al., 2017].”
    BackgroundMethodInfluentialSelf-citation

  • Machine Comprehension Using Match-LSTM and Answer PointerPreprint

    Wang, S., & Jiang, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07905 · Cited by 469

  • Towards Universal Paraphrastic Sentence EmbeddingsPreprint

    Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08198 · Cited by 446

  • Heterogeneous Transfer Learning for Image Classification

    Zhu, Y., Chen, Y., Lu, Z., Pan, S. J., Xue, G.-R., Yu, Y. H., & Yang, Q. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.8090 · Cited by 384 · cited 1×

    “Zhu et al. [2011] use matrix factorization to incorporate textual information into tagged images to enhance image classification.”
    Background

  • Learning to Generate Reviews and Discovering Sentiment

    Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 2×

    “…89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8 LSTM-CNN [Zhou et al., 2016] 96.1”
    Influential

  • Learning question classifiers: the role of semantic information

    Li, X., & Roth, D. (2005) · Natural Language Engineering · doi:10.1017/s1351324905003955 · Cited by 327 · cited 1×

    “MVN [Guo et al., 2017] 51.5 T R E C - 50 SVM [Loni et al., 2011] 89.0 DMN [Kumar et al., 2016] 52.1 SNoW [Li and Roth, 2006] 89.3 LSTM-CNN [Zhou et al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53…”
    Influential

  • Natural Language Processing (Almost) from Scratch

    CollobertRonan, WestonJason, BottouLéon, KarlenMichael, KavukcuogluKoray, & KuksaPavel (2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078186 · Cited by 309 · cited 2×

    “part-of-speech tagging [Collobert et al., 2011], named entity recognition [Pennington et al.”
    Background

  • Unsupervised Pretraining for Sequence to Sequence Learning

    Ramachandran, P., Liu, P., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1039 · Cited by 251 · cited 1×

    “Ramachandran et al. [2016] propose initializing sequence-to-sequence models with pretrained language models and fine-tuning for a specific task.”
    Background

  • Supervised and Semi-Supervised Text Categorization using LSTM for Region EmbeddingsPreprint

    Johnson, R., & Zhang, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02373 · Cited by 248 · cited 1×

    “…87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et al., 2016] 93.8 btree-LSTM [Paria et al., 2016] 87.6 oh-LSTM [Johnson and Zhang, 2016] 94.1 600D ESIM [Chen et al., 2016] 88.0 Virtual [Miyato et al., 2017] 94.1 BCN+Char+CoVe (Ours) 88.1 Test Performance.”
    Influential

  • TopicRNN: A Recurrent Neural Network with Long-Range Semantic DependencyPreprint

    Dieng, A. B., Wang, C., Gao, J., & Paisley, J. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01702 · Cited by 233 · cited 1×

    “…2016b] 53 [Dai and Le, 2015] 92.8 NTI [Munkhdalai and Yu, 2016b] 87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et al., 2016] 93.8 btree-LSTM [Paria et al., 2016] 87.6 oh-LSTM [Johnson and Zhang, 2016] 94.1 600D ESIM [Chen et al., 2016] 88.0 Virtual…”
    Influential

  • A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

    Specia, L., Frank, S., Sima’an, K., & Elliott, D. (2016) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w16-2346 · Cited by 228 · cited 1×

    “Our smallest MT dataset comes from the WMT 2016 multi-modal translation shared task [Specia et al., 2016].”
    Method

  • From symbolic to sub-symbolic information in question classification

    Silva, J. N., Coheur, L., Mendes, A., et al. (2011) · Artificial Intelligence Review · doi:10.1007/s10462-010-9188-4 · Cited by 192 · cited 2×

    “…2016] 52.1 SNoW [Li and Roth, 2006] 89.3 LSTM-CNN [Zhou et al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53 [Dai and Le, 2015] 92.8 NTI [Munkhdalai and Yu, 2016b] 87.3 bmLSTM [Radford et al.,…”
    Influential

  • Question Answering through Transfer Learning from Large Fine-grained Supervision Data

    Min, S., Seo, M., & Hajishirzi, H. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2081 · Cited by 139 · cited 1×

    “Attention mechanisms have also been successfully applied to NLP tasks like entailment [Conneau et al., 2017], summarization [Nallapati et al., 2016], question answering [Seo et al., 2017, Xiong et al., 2017, Min et al., 2017], and semantic parsing [Dong and Lapata, 2016].”
    Background

  • Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents

    Zhang, R., Lee, H., & Radev, D. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1177 · Cited by 136 · cited 1×

    “…resource for transfer learning in NLP. [Looks et al., 2017] 89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM…”
    Influential

  • Discriminative Neural Sentence Modeling by Tree-Based Convolution

    Mou, L., Peng, H., Li, G., Xu, Y., Zhang, L., & Jin, Z. G. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1279 · Cited by 133 · cited 1×

    “…89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8 LSTM-CNN [Zhou et al., 2016] 96.1”
    Influential

  • Encoding Syntactic Knowledge in Neural Networks for Sentiment Classification

    Huang, M., Qian, Q., & Zhu, X. (2017) · ACM Transactions on Information Systems · doi:10.1145/3052770 · Cited by 118 · cited 2×

    “…et al., 2011] 89.0 DMN [Kumar et al., 2016] 52.1 SNoW [Li and Roth, 2006] 89.3 LSTM-CNN [Zhou et al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53 [Dai and Le, 2015] 92.8 NTI [Munkhdalai and Yu,…”
    Influential

  • Neural Tree Indexers for Text Understanding

    Munkhdalai, T., & Yu, H. Q. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1002 · Cited by 93 · cited 3×

    “…in NLP. [Looks et al., 2017] 89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8…”
    Influential

  • Deep Learning with Dynamic Computation GraphsPreprint

    Looks, M., Herreshoff, M., Hutchins, D. S., & Norvig, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02181 · Cited by 88 · cited 1×

    “This is evidence for our hypothesis that MT data has potential as a large resource for transfer learning in NLP. [Looks et al., 2017] 89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe…”
    BackgroundInfluential

  • Neural Semantic Encoders

    Munkhdalai, T., & Yu, H. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1038 · Cited by 73 · cited 3×

    “…in NLP. [Looks et al., 2017] 89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8…”
    Influential

  • The representational geometry of word meanings acquired by neural machine translation models

    Hill, F., Cho, K., Jean, S., & Bengio, Y. (2017) · Machine Translation · doi:10.1007/s10590-017-9194-2 · Cited by 49 · cited 1×

    “Hill et al. [2017] further demonstrate that fixed-length representations obtained from NMT encoders outperform those obtained from monolingual (e.g. language modeling) encoders on semantic similarity tasks.”

  • Improving Question Classification by Feature Extraction and Selection

    Nguyen, V., & Anh-Cuong, L. (2016) · doi:10.17485/ijst/2016/v9i17/93160 · Cited by 46 · cited 1×

    “This is evidence for our hypothesis that MT data has potential as a large resource for transfer learning in NLP. [Looks et al., 2017] 89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe…”
    BackgroundInfluential

  • Question Classification by Weighted Combination of Lexical, Syntactic and Semantic Features

    Loni, B., van Tulder, G., Wiggers, P., et al. (2011) · International Conference on Text, Speech and Dialogue · doi:10.1007/978-3-642-23538-2_31 · Cited by 36 · cited 1×

    “MVN [Guo et al., 2017] 51.5 T R E C - 50 SVM [Loni et al., 2011] 89.0 DMN [Kumar et al., 2016] 52.1 SNoW [Li and Roth, 2006] 89.3 LSTM-CNN [Zhou et al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53…”
    Influential

  • Recursive Neural Networks for Learning Logical Semantics

    Bowman, S. R., Potts, C., & Manning, C. D. (2014) · arXiv.org · doi:10.48550/arxiv.1406.1827 · Cited by 33 · cited 2×

    “Recent work in NLP has continued in this direction by using pretrained word representations to improve models for entailment [Bowman et al., 2014], sentiment analysis [Socher et al.”
    Method

  • End-to-End Multi-View Networks for Text Classification

    Guo, H., Cherry, C., & Su, J. (2017) · arXiv.org · doi:10.48550/arxiv.1704.05907 · Cited by 12 · cited 1×

    “MVN [Guo et al., 2017] 51.5 T R E C - 50 SVM [Loni et al., 2011] 89.0 DMN [Kumar et al., 2016] 52.1 SNoW [Li and Roth, 2006] 89.3 LSTM-CNN [Zhou et al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53…”
    Influential

  • A Neural Architecture Mimicking Humans End-to-End for Natural Language InferencePreprint

    Paria, B., Annervaz, K. M., Dukkipati, A., Chatterjee, A., & Podder, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04741 · Cited by 10 · cited 1×

    “…[Munkhdalai and Yu, 2016b] 87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et al., 2016] 93.8 btree-LSTM [Paria et al., 2016] 87.6 oh-LSTM [Johnson and Zhang, 2016] 94.1 600D ESIM [Chen et al., 2016] 88.0 Virtual [Miyato et al., 2017] 94.1 BCN+Char+CoVe…”
    Influential

Research works printed but not linked: 6

  1. Q. Chen, X.-D. Zhu, Z.-H. Ling, S. Wei, and H. Jiang. Enhancing and combining sequential and tree LSTM for natural language inference. CoRR , abs/1609.06038, 2016. · arXiv
  2. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P11-1015 .
  3. H. T. Madabushi and M. Lee. High accuracy rule-based question classification using question syntax and semantics. In COLING , 2016.
  4. V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML , 2010.
  5. L. Sha, B. Chang, Z. Sui, and S. Li. Reading and thinking: Re-read LSTM unit for textual entailment recognition. In COLING , 2016.
  6. Y. Yu, W. Zhang, K. Hasan, M. Yu, B. Xiang, and B. Zhou. End-to-end reading comprehension with dynamic answer chunk ranking. ICLR , 2017.

Cited by

Sorted by citations

See all 572

Related

See all

Metadata

Authors
McCann, B., Bradbury, J., Xiong, C., & Socher, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Volume
30
Pages
6297-6308
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1708.00107
arXiv
1708.00107
OpenAlex
W2740711318
Semantic Scholar
bc8fa64625d9189f5801837e7b133e7fe3c581f7
Cited by
572
References
72
Citations by year
2025: 14 · 2024: 39 · 2023: 65 · 2022: 57 · 2021: 98 · 2020: 94 · 2019: 122 · 2018: 80 · 2017: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
coves, pre-trained word embeddings, machine translation, LSTM encoder, sentiment analysis, logical consequence, question answering, question classification, transfer learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review