CoChat
Sign Up
PreprintCited by 1,267Topic ModelingDomain Adaptation and Few-Shot LearningSentiment Analysis and Opinion Mining

Universal Sentence Encoder

Open access
Cer, Daniel, Yang, Yinfei, Kong, Sheng-yi, Hua, Nan, Limtiaco, Nicole, John, Rhomni St., Constant, Noah, Guajardo-Cespedes, Mario +5 more·2018·arXiv
DOI 10.48550/arxiv.1803.11175arXiv 1803.11175
Open this paper in CoChat — free accountOpen PDF

Abstract

We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate performance on diverse transfer tasks. Two variants of the encoding models allow for trade-offs between accuracy and compute resources. For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance. Comparisons are made with baselines that use word level transfer learning via pretrained word embeddings as well as baselines do not use any transfer learning. We find that transfer learning using sentence embeddings tends to outperform word level transfer. With transfer learning via sentence embeddings, we observe surprisingly good performance with minimal amounts of supervised training data for a transfer task. We obtain encouraging results on Word Embedding Association Tests (WEAT) targeted at detecting model bias. Our pre-trained sentence encoding models are made freely available for download and on TF Hub.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1803.11175

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • InferSent and Universal Sentence Encoder both use (u, v, |u-v|, u*v) as input for a softmax classifier.

    “InferSent (Conneau et al., 2017) and Universal Sentence Encoder (Cer et al., 2018) both use ( u, v, | u -v | , u ∗ v ) as input for a softmax classifier.”

    From Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) · cites this paper as Cer et al., 2018

    Section 5, p. 3, says Universal Sentence Encoder computes pairwise semantic similarity directly from sentence embeddings using cosine similarity and then angular distance, rather than feeding (u, v, |u−v|, u*v) to a softmax classifier. It describes a task-specific DNN for sentence classification but does not give the claimed four-part input. (LLM, conf=0.83)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Previous work found that the SNLI datasets are suitable for training sentence embeddings.

    “Previous work (Conneau et al., 2017; Cer et al., 2018) found that the SNLI datasets are suitable for training sentence embeddings.”

    From Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) · cites this paper as Cer et al., 2018

  • Universal Sentence Encoder trains a transformer network and augments unsupervised learning with training on SNLI.

    “Universal Sentence Encoder (Cer et al., 2018) trains a transformer network and augments unsupervised learning with training on SNLI.”

    From Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) · cites this paper as Cer et al., 2018

  • Fine-tuning SBERT on NLI data creates sentence embeddings that significantly outperform other state-of-the-art sentence embedding methods like InferSent and Universal Sentence Encoder.

    “We fine-tune SBERT on NLI data, which creates sentence embeddings that significantly outperform other state-of-the-art sentence embedding methods like InferSent (Conneau et al., 2017) and Universal Sentence Encoder (Cer et al., 2018).”

    From Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) · cites this paper as Cer et al., 2018

  • Universal Sentence Encoder was pre-trained on question-answering data.

    “Universal Sentence Encoder was pre-trained on question-answering data”

    From Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) · cites this paper as Cer et al., 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. Mart´ ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of USENIX OSDI'16 .
  2. Marianne Bertrand and Sendhil Mullainathan. 2004. Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination. The American Economic Review , 94(4).
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP .
  4. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science , 356(6334):183-186.
  5. Daniel Cer, Mona Diab, Eneko Agirre, Inigo LopezGazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of SemEval-2017 .
  6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364 . · arXiv
  7. Anthony G. Greenwald, Debbie E. McGhee, and Jordan L. K. Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology , 74(6).
  8. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, L´ aszl´ o Luk´ acs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. CoRR , abs/1705.00652. · arXiv
  9. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of KDD '04 .
  10. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daum´ e III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of ACL/IJCNLP .
  11. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP .
  12. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In In Proceedings of NIPS .
  13. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of COLING '02 .
  14. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS'13 .
  15. Lindsey L. Monteith and Jeremy W. Pettit. 2011. Implicit and explicit stigmatizing attitudes and stereotypes about depression. Journal of Social and Clinical Psychology , 30(5).
  16. Brian A. Nosek, Mahzarin R. Banaji, and Anthony G Greenwald. 2002b. Math = male, me = female, therefore math me. Journal of Personality and Social Psychology, , 83(1).
  17. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL'04), Main Volume .
  18. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL'05 .
  19. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceeding of EMNLP .
  20. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP .
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS .
  22. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation , 39(2):165-210.

Cited by

Sorted by citations

Authors
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Sung, Y.-H., Strope, B., & Kurzweil, R.
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1803.11175
arXiv
1803.11175
OpenAlex
W2794557536
Cited by
1,267
References
1
Citations by year
2026: 8 · 2025: 44 · 2024: 104 · 2023: 165 · 2022: 172 · 2021: 276 · 2020: 253 · 2019: 199 · 2018: 46
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
sentence embeddings, transfer learning, pre-trained embeddings, word embeddings, model complexity, resource efficiency, algorithmic bias detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review