CoChat
Sign Up
PreprintCited by 477Topic ModelingSentiment Analysis and Opinion MiningDomain Adaptation and Few-Shot Learning

SentEval: An Evaluation Toolkit for Universal Sentence Representations

Open access
Conneau, Alexis, Kiela, Douwe·2018·arXiv
DOI 10.48550/arxiv.1803.05449arXiv 1803.05449
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. The aim is to provide a fairer, less cumbersome and more centralized way for evaluating sentence representations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1803.05449

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 37 · Cited by 477

Research works printed but not linked: 37

  1. Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y. (2017). Fine-grained analysis of sentence embed- dings using auxiliary prediction tasks. In Proceedings of ICLR Conference Track, Toulon, France. Published online: https://openreview.net/group?id= ICLR.cc/2017/conference.
  2. Agirre, E., Diab, M., Cer, D., and Gonzalez-Agirre, A. (2012). Semeval-2012 task 6: A pilot on semantic tex- tual similarity. In Proceedings of Semeval-2012, pages 385-393.
  3. Agirre, E., Cer, D., Diab, M., Gonzalez-agirre, A., and Guo, W. (2013). sem 2013 shared task: Semantic tex- tual similarity, including a pilot on typed-similarity. In In *SEM 2013: The Second Joint Conference on Lexical and Computational Semantics. Association for Compu- tational Linguistics.
  4. Agirre, E., Banea, C., Cardie, C., Cer, D. M., Diab, M. T., Gonzalez-Agirre, A., Guo, W., Lopez-Gazpio, I., Mar- itxalar, M., Mihalcea, R., et al. (2015). Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In SemEval@ NAACL-HLT, pages 252-263.
  5. Agirre, E., Baneab, C., Cerd, D., Diabe, M., Gonzalez- Agirre, A., Mihalceab, R., Rigaua, G., Wiebef, J., and Donostia, B. C. (2016). Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evalua- tion. Proceedings of SemEval, pages 497-511.
  6. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer nor- malization. Advances in neural information processing systems (NIPS).
  7. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. In Proceedings of EMNLP.
  8. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Spe- cia, L. (2017). Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evalua- tion. arXiv preprint arXiv:1708.00055. · arXiv
  9. Chiu, B., Korhonen, A., and Pyysalo, S. (2016). Intrinsic evaluation of word vectors fails to predict extrinsic per- formance. In First Workshop on Evaluating Vector Space Representations for NLP (RepEval).
  10. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). Supervised learning of universal sentence representations from natural language inference data. In Proceedings of EMNLP, Copenhagen, Denmark.
  11. Dolan, B., Quirk, C., and Brockett, C. (2004). Unsuper- vised construction of large paraphrase corpora: Exploit- ing massively parallel news sources. In Proceedings of ACL, page 350.
  12. Ettinger, A., Elgohary, A., and Resnik, P. (2016). Probing for semantic evidence of composition by means of sim- ple classification tasks. In First Workshop on Evaluating Vector Space Representations for NLP (RepEval), page 134.
  13. Faruqui, M., Tsvetkov, Y., Rastogi, P., and Dyer, C. (2016). Problems with evaluation of word embeddings using word similarity tasks. arXiv preprint arXiv:1605.02276. · arXiv
  14. Hill, F., Cho, K., and Korhonen, A. (2016a). Learning distributed representations of sentences from unlabelled data. In Proceedings of NAACL.
  15. Hu, M. and Liu, B. (2004). Mining and summarizing cus- tomer reviews. In Proceedings of SIGKDD, pages 168- 177.
  16. Ji, Y. and Eisenstein, J. (2013). Discriminative improve- ments to distributional sentence similarity. In Proceed- ings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  17. Karpathy, A. and Fei-Fei, L. (2015). Deep visual-semantic alignments for generating image descriptions. In Pro- ceedings of CVPR, pages 3128-3137.
  18. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urta- sun, R., Torralba, A., and Fidler, S. (2015). Skip-thought vectors. In Advances in neural information processing systems, pages 3294-3302.
  19. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Fed- erico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demon- stration Sessions, ACL '07, pages 177-180, Strouds- burg, PA, USA. Association for Computational Linguis- tics.
  20. Lai, A. and Hockenmaier, J. (2014). Illinois-lh: A deno- tational and distributional approach to semantics. Proc. SemEval, 2:5.
  21. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In Euro- pean Conference on Computer Vision, pages 740-755. Springer International Publishing.
  22. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., and Zamparelli, R. (2014). A SICK cure for the evaluation of compositional distributional seman- tic models. In Proceedings of LREC.
  23. Nayak, N., Angeli, G., and Manning, C. D. (2016). Eval- uating word embeddings using a representative suite of practical tasks. In First Workshop on Evaluating Vector Space Representations for NLP (RepEval), page 19.
  24. Pang, B. and Lee, L. (2004). A sentimental educa- tion: Sentiment analysis using subjectivity summariza- tion based on minimum cuts. In Proceedings of ACL, page 271.
  25. Pang, B. and Lee, L. (2005). Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL, pages 115-124.
  26. Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In Pro- ceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), volume 14, pages 1532-1543.
  27. Shi, X., Padhi, I., and Knight, K. (2016). Does string- based neural MT learn source syntax? In Proceedings of EMNLP, pages 1526-1534, Austin, Texas.
  28. Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., Potts, C., et al. (2013). Recursive deep models for semantic compositionality over a senti- ment treebank. In Proceedings of EMNLP, pages 1631- 1642.
  29. Tai, K. S., Socher, R., and Manning, C. D. (2015a). Improved semantic representations from tree-structured long short-term memory networks. Proceedings of ACL.
  30. Tai, K. S., Socher, R., and Manning, C. D. (2015b). Improved semantic representations from tree-structured long short-term memory networks. Proceedings of the 53rd Annual Meeting of the Association for Computa- tional Linguistics (ACL).
  31. Voorhees, E. M. and Tice, D. M. (2000). Building a ques- tion answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Re- search and development in information retrieval, pages 200-207. ACM.
  32. Vulić, I., Kiela, D., and Korhonen, A. (2017). Evalua- tion by association: A systematic study of quantitative word association evaluation. In Proceedings of EACL, volume 1, pages 163-175.
  33. Wiebe, J., Wilson, T., and Cardie, C. (2005). Annotating expressions of opinions and emotions in language. Lan- guage resources and evaluation, 39(2):165-210.
  34. Wieting, J., Bansal, M., Gimpel, K., and Livescu, K. (2015). Towards universal paraphrastic sentence embed- dings. Proceedings of the 4th International Conference on Learning Representations (ICLR).
  35. Wieting, J., Bansal, M., Gimpel, K., and Livescu, K. (2016). Charagram: Embedding words and sentences via character n-grams. Proceedings of the 2016 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP).
  36. Zhao, H., Lu, Z., and Poupart, P. (2015). Self-adaptive hi- erarchical sentence model. In Proceedings of the 24th International Conference on Artificial Intelligence, IJ- CAI'15, pages 4069-4076. AAAI Press.
  37. Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., and Xu, B. (2016). Text classification improved by integrating bidi- rectional lstm with two-dimensional max pooling. Pro- ceedings of COLING 2016, the 26th International Con- ference on Computational Linguistics.

Authors
Conneau, A., & Kiela, D.
Affiliations
Meta (Israel) (IL)
Venue
arXiv
Publisher
arXiv
Published
2018
DOI
10.48550/arxiv.1803.05449
arXiv
1803.05449
OpenAlex
W2964165804
Cited by
477
References
37
Citations by year
2026: 10 · 2025: 6 · 2024: 25 · 2023: 81 · 2022: 67 · 2021: 97 · 2020: 82 · 2019: 84 · 2018: 24 · 2017: 1
FWCI
29.18
Type
conference paper
Language
en
Open access
Licence
Locations
  • Language Resources and Evaluation Conference (LREC) (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
sentence embeddings, evaluation toolkit, natural language inference, sentence similarity, text classification, benchmark dataset, sentence encoders

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review