CoChat
Sign Up
PreprintCited by 1,293Topic ModelingSentiment Analysis and Opinion MiningNatural Language Processing Techniques

Skip-Thought Vectors

Open access
Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan, Zemel, Richard S., Torralba, Antonio, Urtasun, Raquel, Fidler, Sanja·2015·arXiv
DOI 10.48550/arxiv.1506.06726arXiv 1506.06726
Open this paper in CoChat — free accountOpen PDF

Abstract

We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1506.06726

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Early results on transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier fed sentence embeddings from a fixed pre-trained model.

    “Early results on transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model (Subramanian et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Hill et al., 2016; Conneau et al., 2017).”

    From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Kiros et al., 2015

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 30

  1. [1][1] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , 2013. -
  2. [3][3] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. ACL , 2014. -
  3. [4][4] Yoon Kim. Convolutional neural networks for sentence classification. EMNLP , 2014. -
  4. [6][6] Han Zhao, Zhengdong Lu, and Pascal Poupart. Self-adaptive hierarchical sentence model. IJCAI , 2015. -
  5. [7][7] Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. ICML , 2014. -
  6. [10][10] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In EMNLP , pages 17001709, 2013. -
  7. [11][11] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. EMNLP , 2014. -
  8. [12][12] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In NIPS , 2014. -
  9. [13][13] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR , 2015. -
  10. [14][14] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS Deep Learning Workshop , 2014. -
  11. [15][15] Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168 , 2013. - · arXiv
  12. [16][16] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR , 2014. -
  13. [17][17] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR , 2015. -
  14. [22][22] Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. ACL , 2015. -
  15. [23][23] Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. Grounded compositional semantics for finding and describing images with sentences. TACL , 2014. -
  16. [24][24] Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS , 2011. -
  17. [26][26] Dipanjan Das and Noah A Smith. Paraphrase identification as probabilistic quasi-synchronous recognition. In ACL , 2009. -
  18. [27][27] Stephen Wan, Mark Dras, Robert Dale, and Cécile Paris. Using dependency-based features to take the â ˘ AIJpara-farceâ ˘ A˙ I out of paraphrase. In Proceedings of the Australasian Language Technology Workshop , 2006. -
  19. [28][28] Nitin Madnani, Joel Tetreault, and Martin Chodorow. Re-examining machine translation metrics for paraphrase identification. In NAACL , 2012. -
  20. [30][30] Bill Dolan, Chris Quirk, and Chris Brockett. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th international conference on Computational Linguistics , 2004. -
  21. [32][32] Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Associating neural word embeddings with deep image representations using fisher vectors. In CVPR , 2015. -
  22. [33][33] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). ICLR , 2015. -
  23. [34][34] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755. 2014. -
  24. [35][35] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR , 2015. -
  25. [36][36] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL , pages 115-124, 2005. -
  26. [37][37] Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining , pages 168-177, 2004. -
  27. [38][38] Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL , 2004. -
  28. [40][40] Xin Li and Dan Roth. Learning question classifiers. In Proceedings of the 19th international conference on Computational linguistics , 2002. -
  29. [41][41] Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL , 2012. -
  30. [42][42] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR , 2008.

Cited by

Sorted by citations

Authors
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S.
Affiliations
  • University of Toronto (CA)
  • Canadian Institute for Advanced Research (CA)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2015
Volume
28
Pages
3294-3302
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1506.06726
arXiv
1506.06726
OpenAlex
W1486649854
Cited by
1,293
References
4
Citations by year
2025: 26 · 2024: 47 · 2023: 73 · 2022: 77 · 2021: 187 · 2020: 236 · 2019: 292 · 2018: 186 · 2017: 113 · 2016: 52 · 2015: 3 · 2012: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
  • Canadian Institute for Advanced Research
  • Samsung
  • Natural Sciences and Engineering Research Council of Canada
  • Office of Naval Research (N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
sentence embeddings, encoder-decoder architecture, sentence reconstruction, vocabulary enrichment, semantic relatedness, paraphrase detection, question classification, sentiment analysis, subjectivity, word representation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review