CoChat
Sign Up
Cited by 1,562Topic ModelingNatural Language Processing TechniquesSentiment Analysis and Opinion Mining

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

Open access
Cer, Daniel, Diab, Mona, Agirre, Eneko, Lopez-Gazpio, Inigo, Specia, Lucia·2017·Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)
DOI 10.18653/v1/s17-2001
Open this paper in CoChat — free accountOpen PDF

Abstract

Semantic Textual Similarity (STS) measures the meaning similarity of sentences.Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems.The STS shared task is a venue for assessing the current state-of-the-art.The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data.The task obtained strong participation from 31 teams, with 17 participating in all language tracks.We summarize performance and review a selection of well performing methods.Analysis highlights common errors, providing insight into the limitations of existing models.To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012)(2013)(2014)(2015)(2016)(2017).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/s17-2001

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Using this setup, BERT set a new state-of-the-art performance on the Semantic Textual Similarity (STS) benchmark.

    “Using this setup, BERT set a new state-of-the-art performance on the Semantic Textual Semilarity (STS) benchmark (Cer et al., 2017).”

    From Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) · cites this paper as Cer et al., 2017

    Section 8, “STS Benchmark” (page not provided), introduces the benchmark and explains that it enables comparisons and tracking of the state of the art. It reports results for 2017 systems, but does not report BERT results or establish that BERT set a new state of the art. The citation supports the benchmark’s identification, not the performance claim. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Bias-STS-B adapts STS-B to generate pairs of sentences differing only in gendered terms but otherwise maintaining the same meaning.

    “Bias-STS-B (Webster et al. 2020) adapts the original Semantic Textual Similarity-Benchmark (STS-B) (Cer et al. 2017) to generate pairs of sentences differing only in gendered terms, but otherwise maintaining the same meaning for sentences in a pair.”

    From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Cer et al. 2017

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 83

  1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, Ger- man Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 Task 2: Semantic Textual Similarity, En- glish, Spanish and Pilot on Interpretability. In Proceedings of SemEval 2015. http://www.aclweb.org/anthology/S15- 2045.
  2. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 Task 10: Multilingual semantic tex- tual similarity. In Proceedings of SemEval 2014. http://www.aclweb.org/anthology/S14-2010.
  3. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. Semeval-2016 task 1: Se- mantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the SemEval-2016. http://www.aclweb.org/anthology/S16-1081.
  4. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez- Agirre. 2012. SemEval-2012 Task 6: A pilot on semantic textual similarity. In Proceedings of *SEM 2012/SemEval 2012. http://www.aclweb.org/anthology/S12-1051.
  5. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Se- mantic Textual Similarity. In Proceedings of *SEM 2013. http://www.aclweb.org/anthology/S13-1004.
  6. Hussein T. Al-Natsheh, Lucie Martinet, Fabrice Muhlen- bach, and Djamel Abdelkader ZIGHED. 2017. UdL at SemEval-2017 Task 1: Semantic textual similarity esti- mation of english sentence pairs using regression model over pairwise features. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2013.
  7. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sen- tence embeddings. In Proceedings of ICLR 2017. https://openreview.net/pdf?id=SyK00v5xx.
  8. Ignacio Arroyo-Fernández and Ivan Vladimir Meza Ruiz. 2017. LIPN-IIMAS at SemEval-2017 Task 1: Sub- word embeddings, attention recurrent neural net- works and cross word alignment for semantic tex- tual similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2031.
  9. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet Project. In Proceedings of COLING '98. http://aclweb.org/anthology/P/P98/P98- 1013.pdf.
  10. Daniel Bär, Chris Biemann, Iryna Gurevych, and Torsten Zesch. 2012. Ukp: Computing semantic textual sim- ilarity by combining multiple content similarity mea- sures. In Proceedings of *SEM 2012/SemEval 2012. http://www.aclweb.org/anthology/S12-1059.
  11. Joe Barrow and Denis Peskov. 2017. UMDeep at SemEval- 2017 Task 1: End-to-end shared weight LSTM model for semantic textual similarity. In Proceedings of SemEval- 2017. http://www.aclweb.org/anthology/S17-2026.
  12. Luisa Bentivogli, Raffaella Bernardi, Marco Marelli, Ste- fano Menini, Marco Baroni, and Roberto Zamparelli. 2016. SICK through the SemEval glasses. lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. Lang Resour Eval 50(1):95-124. https://doi.org/10.1007/s10579-015-9332-5. · DOI
  13. Alexandre Berard, Christophe Servan, Olivier Pietquin, and Laurent Besacier. 2016. MultiVec: a multilin- gual and multilevel representation learning toolkit for NLP. In Proceedings of LREC 2016. http://www.lrec- conf.org/proceedings/lrec2016/pdf/666 Paper.pdf.
  14. Ergun Bic ¸ici. 2017. RTM at SemEval-2017 Task 1: Referential translation machines for predicting seman- tic similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2030.
  15. Johannes Bjerva and Robert Östling. 2017. ResSim at SemEval-2017 Task 1: Multilingual word representations for semantic textual similarity. In Proceedings of SemEval- 2017. http://www.aclweb.org/anthology/S17-2021.
  16. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statisti- cal machine translation. In Proceedings of WMT 2014. http://www.aclweb.org/anthology/W/W14/W14- 3302.pdf.
  17. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Chris- tian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of WMT 2013. http://www.aclweb.org/anthology/W13-2201.
  18. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP 2015. http://aclweb.org/anthology/D/D15/D15- 1075.pdf.
  19. Tomas Brychcin and Lukas Svoboda. 2016. UWB at SemEval-2016 Task 1: Semantic textual sim- ilarity using lexical, syntactic, and semantic in- formation. In Proceedings of SemEval 2016. https://www.aclweb.org/anthology/S/S16/S16-1089.pdf.
  20. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, and Hui Jiang. 2016. Enhancing and combining sequential and tree LSTM for natural language inference. CoRR abs/1609.06038. http://arxiv.org/abs/1609.06038. · arXiv
  21. Yun-Nung Chen, Dilek Hakkani-Tür, and Xiaodong He. 2015. Learning bidirectional intent embeddings by convolutional deep structured semantic models for spoken language understanding. In Proceedings of NIPS-SLU, 2015. https://www.microsoft.com/en- us/research/publication/learning-bidirectional-intent- embeddings-by-convolutional-deep-structured-semantic- models-for-spoken-language-understanding/.
  22. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Bar- rault, and Antoine Bordes. 2017. Supervised learn- ing of universal sentence representations from natu- ral language inference data. CoRR abs/1705.02364. http://arxiv.org/abs/1705.02364. · arXiv
  23. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rational, evalua- tion and approaches. J. Nat. Language Eng. 16:105-105. https://doi.org/10.1017/S1351324909990234. · DOI
  24. Birk Diedenhofen and Jochen Musch. 2015. co- cor: A comprehensive solution for the statisti- cal comparison of correlations. PLoS ONE 10(4). http://dx.doi.org/10.1371/journal.pone.0121945. · DOI · Paper page
  25. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsu- pervised construction of large paraphrase corpora: Ex- ploiting massively parallel news sources. In Proceedings of COLING 04. http://aclweb.org/anthology/C/C04/C04- 1051.pdf.
  26. Mirela-Stefania Duma and Wolfgang Menzel. 2017. SEF@UHH at SemEval-2017 Task 1: Unsuper- vised knowledge-free semantic textual similarity via paragraph vector. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2024.
  27. Cristina España Bonet and Alberto Barrón-Cedeño. 2017. Lump at SemEval-2017 Task 1: Towards an interlingua semantic similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2019.
  28. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. https://books.google.com/books?id=Rehu8OOzMIMC.
  29. Jérémy Ferrero, Laurent Besacier, Didier Schwab, and Frédéric Agnès. 2017. CompiLIG at SemEval-2017 Task 1: Cross-language plagiarism detection methods for seman- tic textual similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2012.
  30. Pedro Fialho, Hugo Patinho Rodrigues, Luísa Coheur, and Paulo Quaresma. 2017. L2f/inesc-id at semeval-2017 tasks 1 and 2: Lexical and semantic features in word and textual similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2032.
  31. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of NAACL/HLT 2013. http://cs.jhu.edu/ ccb/publications/ppdb.pdf.
  32. Basma Hassan, Samir AbdelRahman, Reem Bahgat, and Ibrahim Farag. 2017. FCICU at SemEval-2017 Task 1: Sense-based language independent semantic textual similarity approach. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2015.
  33. Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multi- perspective sentence similarity modeling with convolu- tional neural networks. In Proceedings of EMNLP. pages 1576-1586. http://aclweb.org/anthology/D15-1181.
  34. Hua He and Jimmy Lin. 2016. Pairwise word interaction modeling with deep neural networks for semantic sim- ilarity measurement. In Proceedings of NAACL/HLT. http://www.aclweb.org/anthology/N16-1108.
  35. Hua He, John Wieting, Kevin Gimpel, Jinfeng Rao, and Jimmy Lin. 2016. UMD-TTIC-UW at SemEval- 2016 Task 1: Attention-based multi-perspective convolutional neural networks for textual similar- ity measurement. In Proceedings of SemEval 2016. http://www.anthology.aclweb.org/S/S16/S16-1170.pdf.
  36. John Henderson, Elizabeth Merkhofer, Laura Strickhart, and Guido Zarrella. 2017. MITRE at SemEval-2017 Task 1: Simple semantic similarity. In Proceedings of SemEval- 2017. http://www.aclweb.org/anthology/S17-2027.
  37. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proceedings of NAACL/HLT. http://www.aclweb.org/anthology/N16-1162.
  38. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of NAACL/HLT 2006. http://aclweb.org/anthology/N/N06/N06-2015.pdf.
  39. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of CIKM. https://www.microsoft.com/en- us/research/publication/learning-deep-structured- semantic-models-for-web-search-using-clickthrough- data/.
  40. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of ACL/IJCNLP. http://www.aclweb.org/anthology/P15- 1162.
  41. Sergio Jimenez, Claudia Becerra, and Alexander Gelbukh. 2012a. Soft cardinality: A parameterized similarity func- tion for text comparison. In Proceedings of *SEM 2012/Se- mEval 2012. http://www.aclweb.org/anthology/S12-1061.
  42. Sergio Jimenez, Claudia Becerra, and Alexander Gelbukh. 2012b. Soft Cardinality: A parameterized similarity func- tion for text comparison. In Proceedings of *SEM 2012/Se- mEval 2012. http://aclweb.org/anthology/S/S12/S12- 1061.pdf.
  43. Tom Kenter, Alexey Borisov, and Maarten de Rijke. 2016. Siamese cbow: Optimizing word embeddings for sentence representations. In Proceedings of ACL. http://www.aclweb.org/anthology/P16-1089.
  44. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fi- dler. 2015. Skip-thought vectors. CoRR abs/1506.06726. http://arxiv.org/abs/1506.06726. · arXiv
  45. Sarah Kohail, Amr Rekaby Salama, and Chris Biemann. 2017. STS-UHH at SemEval-2017 Task 1: Scoring semantic textual similarity using supervised and unsu- pervised ensemble. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2025.
  46. Jey Han Lau and Timothy Baldwin. 2016. An em- pirical evaluation of doc2vec with practical insights into document embedding generation. In Proceed- ings of ACL Workshop on Representation Learning for NLP. http://www.aclweb.org/anthology/W/W16/W16- 1609.pdf.
  47. Quoc V. Le and Tomas Mikolov. 2014. Distributed represen- tations of sentences and documents. CoRR abs/1405.4053. http://arxiv.org/abs/1405.4053. · arXiv
  48. I-Ta Lee, Mahak Goindani, Chang Li, Di Jin, Kristen Marie Johnson, Xiao Zhang, Maria Leonor Pacheco, and Dan Goldwasser. 2017. PurdueNLP at SemEval-2017 Task 1: Predicting semantic textual similarity with paraphrase and event embeddings. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2029.
  49. Wenjie Liu, Chengjie Sun, Lei Lin, and Bingquan Liu. 2017. ITNLP-AiKF at SemEval-2017 Task 1: Rich features based svr for semantic textual simi- larity computing. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2022.
  50. Nabin Maharjan, Rajendra Banjade, Dipesh Gautam, Lasang J. Tamang, and Vasile Rus. 2017. Dt team at semeval-2017 task 1: Semantic similarity using align- ments, sentence-level embeddings and gaussian mix- ture model output. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2014.
  51. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language process- ing toolkit. In Proceedings of ACL 2014 Demonstrations. http://www.aclweb.org/anthology/P/P14/P14-5010.
  52. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC 14. http://www.lrec- conf.org/proceedings/lrec2014/pdf/363 Paper.pdf.
  53. Yuan Cao Gaurav Kumar Matt Post. 2015. Joshua 6: A phrase-based and hierarchical statistical machine transla- tion. The Prague Bulletin of Mathematical Linguistics 104:516. https://ufal.mff.cuni.cz/pbml/104/art-post-cao- kumar.pdf.
  54. Fanqing Meng, Wenpeng Lu, Yuteng Zhang, Jinyong Cheng, Yuehan Du, and Shuwang Han. 2017. QLUT at SemEval- 2017 Task 1: Semantic textual similarity based on word embeddings. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2020.
  55. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Pro- ceedings of NIPS 2013. http://papers.nips.cc/paper/5021- distributed-representations-of-words-and-phrases-and- their-compositionality.pdf.
  56. Alessandro Moschitti. 2006. Efficient convolution kernels for dependency and constituent syntactic trees. In Proceed- ings of ECML'06. http://dx.doi.org/10.1007/11871842 · DOI
  57. Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2017. Rep- resenting sentences as low-rank subspaces. CoRR abs/1704.05358. http://arxiv.org/abs/1704.05358. · arXiv
  58. El Moatez Billah Nagoudi, Jérémy Ferrero, and Didier Schwab. 2017. LIM-LIG at SemEval-2017 Task1: En- hancing the semantic similarity for arabic sentences with vectors weighting. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2017.
  59. Roberto Navigli and Simone Paolo Ponzetto. 2010. BabelNet: Building a very large multilingual se- mantic network. In Proceedings of ACL 2010. http://aclweb.org/anthology/P/P10/P10-1023.pdf.
  60. Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2017. Unsupervised Learning of Sentence Embed- dings using Compositional n-Gram Features. arXiv https://arxiv.org/pdf/1703.02507.pdf. · arXiv
  61. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of EMNLP 2014. http://www.aclweb.org/anthology/D14-1162.
  62. Nghia The Pham, Germán Kruszewski, Angeliki Lazari- dou, and Marco Baroni. 2015. Jointly optimizing word representations for lexical and sentential tasks with the c-phrase model. In Proceedings of ACL/IJCNLP. http://www.aclweb.org/anthology/P15-1094.
  63. Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016. Task-oriented intrinsic evaluation of semantic tex- tual similarity. In Proceedings of COLING 2016. http://aclweb.org/anthology/C16-1009.
  64. Barbara Rychalska, Katarzyna Pakulska, Krystyna Chodor- owska, Wojciech Walczak, and Piotr Andruszkiewicz. 2016. Samsung Poland NLP Team at SemEval-2016 Task 1: Necessity for diversity; combining recursive au- toencoders, wordnet and ensemble methods to measure semantic similarity. In Proceedings of SemEval-2016. http://www.aclweb.org/anthology/S16-1091.
  65. Alexandre Salle, Marco Idiart, and Aline Villavicencio. 2016a. Enhancing the lexvec distributed word representation model using positional contexts and external memory. CoRR abs/1606.01283. http://arxiv.org/abs/1606.01283. · arXiv
  66. Alexandre Salle, Marco Idiart, and Aline Villavicencio. 2016b. Matrix factorization using window sampling and negative sampling for improved word representations. In Proceed- ings of ACL. http://aclweb.org/anthology/P16-2068.
  67. Gilles Serasset. 2015. DBnary: Wiktionary as a lemon- based multilingual lexical resource in RDF. Semantic Web Journal (special issue on Multilingual Linked Open Data) 6:355-361. https://doi.org/10.3233/SW-140147. · DOI
  68. Yang Shao. 2017. HCTI at SemEval-2017 Task 1: Use convolutional neural network to evaluate seman- tic textual similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2016.
  69. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gregoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for in- formation retrieval. In Proceedings of CIKM '14. https://www.microsoft.com/en-us/research/publication/a- latent-semantic-model-with-convolutional-pooling- structure-for-information-retrieval/.
  70. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Mic- ciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceed- ings of AMTA 2006. http://mt-archive.info/AMTA-2006- Snover.pdf.
  71. Martyna Śpiewak, Piotr Sobecki, and Daniel Karaś. 2017. OPI-JSA at SemEval-2017 Task 1: Appli- cation of ensemble learning for computing semantic textual similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2018.
  72. Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2015. DLS@CU: Sentence similarity from word alignment and semantic vector composition. In Proceedings of SemEval 2015. http://aclweb.org/anthology/S/S15/S15-2027.pdf.
  73. Junfeng Tian, Zhiheng Zhou, Man Lan, and Yuanbin Wu. 2017. ECNU at SemEval-2017 Task 1: Leverage kernel- based traditional nlp features and neural networks to build a universal model for multilingual and cross-lingual seman- tic textual similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2028.
  74. Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012a. Take- lab: Systems for measuring semantic text similar- ity. In Proceedings of *SEM 2012/SemEval 2012. http://www.aclweb.org/anthology/S12-1060.
  75. Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012b. TakeLab: Systems for mea- suring semantic text similarity. In Proceedings of SemEval 2012. http://www.aclweb.org/anthology/S12-1060.
  76. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the ACL (TACL) 3:345-358. http://aclweb.org/anthology/Q/Q15/Q15-1025.pdf.
  77. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016a. Charagram: Embedding words and sen- tences via character n-grams. In Proceedings of EMNLP. https://aclweb.org/anthology/D16-1157.
  78. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016b. Towards universal paraphrastic sen- tence embeddings. In Proceedings of ICLR 2016. http://arxiv.org/abs/1511.08198. · arXiv
  79. John Wieting and Kevin Gimpel. 2017. Revisiting recurrent networks for paraphrastic sentence embeddings. CoRR abs/1705.00364. http://arxiv.org/abs/1705.00364. · arXiv
  80. Hao Wu, Heyan Huang, Ping Jian, Yuhang Guo, and Chao Su. 2017. BIT at SemEval-2017 Task 1: Us- ing semantic information space to evaluate semantic textual similarity. In Proceedings of SemEval-2017. http://www.aclweb.org/anthology/S17-2007.
  81. Wei Xu, Chris Callison-Burch, and William B. Dolan. 2015. SemEval-2015 Task 1: Paraphrase and semantic similar- ity in Twitter (PIT). In Proceedings of SemEval 2015. http://www.aclweb.org/anthology/S15-2001.
  82. Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to vi- sual denotations: New similarity metrics for seman- tic inference over event descriptions. TACL 2:67-78. http://aclweb.org/anthology/Q14-1006.
  83. WenLi Zhuang and Ernie Chang. 2017. Neobility at SemEval-2017 Task 1: An attention-based sen- tence similarity model. In Proceedings SemEval-2017. http://www.aclweb.org/anthology/S17-2023.

Cited by

Sorted by citations

Authors
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L.
Affiliations
  • Google (United States) (US)
  • George Washington University (US)
  • University of the Basque Country (ES)
  • University of Sheffield (GB)
Venue
Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)
Publisher
Association for Computational Linguistics
Published
2017
Pages
1-14
DOI
10.18653/v1/s17-2001
OpenAlex
W2739351760
Cited by
1,562
References
4
Citations by year
2026: 72 · 2025: 75 · 2024: 140 · 2023: 266 · 2022: 253 · 2021: 303 · 2020: 220 · 2019: 143 · 2018: 64 · 2017: 26
FWCI
56.50
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • International Workshop on Semantic Evaluation (SemEval) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • Communities in ADDI (University of the Basque Country) (submittedVersion, cc-by)
  • Hispana (submittedVersion, cc-by)
  • Spiral (Imperial College London) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • Ministerio de Educación, Cultura y Deporte
  • Qatar National Research Fund
  • Horizon 2020 Framework Programme
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
semantic textual similarity, machine translation quality estimation, shared task evaluation, semantic representation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review