CoChat
Sign Up
PreprintCited by 1,051Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Open access
Wang, Alex, Pruksachatkun, Yada, Nangia, Nikita, Singh, Amanpreet, Michael, Julian, Hill, Felix, Levy, Omer, Bowman, Samuel R.·2019·arXiv
DOI 10.48550/arxiv.1905.00537arXiv 1905.00537
Open this paper in CoChat — free accountOpen PDF

Abstract

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1905.00537

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks.

    “The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks”

    From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Wang et al., 2019

    The Introduction (p. 1) calls GLUE a prominent evaluation framework built from existing datasets, and Related Work (p. 3) says it was used to evaluate GPT and BERT. These passages support the use of fixed-task benchmarks, but the paper does not establish the broader historical claim that large-language-model evaluation relied on non-interactive tasks rather than interactive ones. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Well-crafted benchmarks have accelerated progress in new algorithms, evaluation, and analysis in their respective research areas.

    “These well-crafted benchmarks have accelerated progress in new algorithms, evaluation, and analysis in their respective research areas.”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [163]

    The Introduction (pp. 1–2) describes GLUE as a prominent evaluation framework and says SuperGLUE is designed to measure progress and encourage future innovations. The paper does not establish that these benchmarks have already accelerated new algorithms, evaluation, and analysis across their respective research areas. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Summary vectors outperform few-shot ICL with a comparable number of in-context tokens on 8 out of 11 tasks.

    “We consider 11 classification tasks, including 7 SuperGLUE tasks (Wang et al., 2019), and we find that summary vectors outperform few-shot ICL with a comparable number of in-context tokens on 8 out of 11 tasks.”

    From Adapting Language Models to Compress Contexts (2023) · cites this paper as Wang et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works

  • WordNet: a lexical database for English

    Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 3× · see citing works

    “Systems also may not use the unlabeled test data for the tasks in system development in any way, may not use the structured source data that was used to collect the WiC labels (sense-annotated example sentences from WordNet, VerbNet, and Wiktionary) in any way, and may not build systems that share information across s…”
    Method

  • Deep Contextualized Word Representations

    Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works

  • QuAC: Question Answering in Context

    Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714 · cited 9× · see citing works

    “It was challenging to train annotators to do well on Quora Insincere Questions 10 , Empa-thetic Reactions (Buechel et al., 2018), and a recast version of Ultra-Fine Entity Typing (Choi et al., 2018b, see Appendix C.3 for details), leading to low human performance.”
    BackgroundMethodInfluential

Flagged first, then by citations

Research works printed but not linked: 68

  1. Stephen H. Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alexander Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Ré, and Rob Malkin. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In SIGMOD . ACM, 2018.
  2. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment , 2006. URL http://u.cs.biu.ac.il/~nlp/RTE2/Proceedings/01.pdf .
  3. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. In Textual Analysis Conference (TAC) , 2009. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.232.1231 .
  4. Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc. Modeling empathy and distress in reaction to news stories. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018.
  5. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluation the role of bleu in machine translation research. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL) . Association for Computational Linguistics, 2006. URL https://www.aclweb.org/anthology/E06-1032 .
  6. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) . Association for Computational Linguistics, 2017. doi: 10.18653/v1/S17-2001. URL https://www.aclweb. org/anthology/S17-2001 . · DOI · Paper page
  7. Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. Ultra-fine entity typing. In Proceedings of the Association for Computational Linguistics (ACL) . Association for Computational Linguistics, 2018b. URL https://www.aclweb.org/anthology/P18-1009 .
  8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2924-2936, 2019a.
  9. Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. BAM! Born-again multi-task networks for natural language understanding. In Proceedings of the Association of Computational Linguistics (ACL) . Association for Computational Linguistics, 2019b. URL https://arxiv.org/pdf/1907.04829.pdf . · arXiv
  10. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning (ICML) . Association for Computing Machinery, 2008. URL https://dl.acm. org/citation.cfm?id=1390177 .
  11. Alexis Conneau and Douwe Kiela. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the 11th Language Resources and Evaluation Conference . European Language Resource Association, 2018. URL https://www.aclweb.org/anthology/L18-1269 .
  12. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
  13. (EMNLP) . Association for Computational Linguistics, 2017. doi: 10.18653/v1/D17-1070. URL https://www.aclweb.org/anthology/D17-1070 . · DOI · Paper page
  14. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment . Springer, 2006. URL https: //link.springer.com/chapter/10.1007/11736790\_9 . · DOI
  15. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (NeurIPS) . Curran Associates, Inc., 2015. URL http://papers. nips.cc/paper/5949-semi-supervised-sequence-learning.pdf .
  16. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. 2019. To appear in Proceedings of Sinn und Bedeutung 23 . Data can be found at https://github.com/mcdm/CommitmentBank/ .
  17. William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of IWP , 2005.
  18. Manaal Faruqui and Dipanjan Das. Identifying well-formed natural language questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2018. URL https://www.aclweb.org/anthology/ D18-1091 .
  19. Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. International Conference on Machine Learning (ICML) , 2018. URL http://proceedings.mlr.press/v80/furlanello18a.html .
  20. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software , 2017. URL https://www.aclweb.org/anthology/W18-2501 .
  21. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing . Association for Computational Linguistics, 2007.
  22. Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 609-614, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. URL https: //www.aclweb.org/anthology/N19-1061 .
  23. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) . Association for Computational Linguistics, 2016. doi: 10.18653/v1/N16-1162. URL https: //www.aclweb.org/anthology/N16-1162 . · DOI · Paper page
  24. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1503.02531 , 2015. URL https://arxiv.org/abs/1503.02531 . · arXiv
  25. Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2017. doi: 10.18653/v1/D17-1215. URL https: //www.aclweb.org/anthology/D17-1215 . · DOI · Paper page
  26. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) . Association for Computational Linguistics, 2018. URL https://www.aclweb.org/anthology/papers/N/N18/N18-1023/ .
  27. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6980 , 2014. URL https://arxiv.org/abs/1412.6980 . · arXiv
  28. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems , 2015.
  29. Nikita Kitaev, Steven Cao, and Dan Klein. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3499-3505, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1340. URL https://www.aclweb.org/anthology/P19-1340 . · DOI
  30. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for the Winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4837-4842, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1478. URL https://www.aclweb.org/anthology/P19-1478 . · DOI · Paper page
  31. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, September 2017. doi: 10.18653/v1/ D17-1018. URL https://www.aclweb.org/anthology/D17-1018 . · DOI
  32. Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning , 2012. URL http://dl.acm.org/citation.cfm?id=3031843.3031909 .
  33. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 2016. doi: 10.18653/ v1/D16-1230. URL https://www.aclweb.org/anthology/D16-1230 .
  34. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) . Association for Computational Linguistics, 2019a. URL https: //arxiv.org/abs/1903.08855 .
  35. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACLHLT) . Association for Computational Linguistics, 2019b. URL https://arxiv.org/abs/1904. 02668 .
  36. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint 1904.09482 , 2019c. URL http://arxiv.org/abs/1904.09482 . · arXiv
  37. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint 1901.11504 , 2019d. · arXiv
  38. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. arXiv preprint 1807.11714 , 2018. URL http://arxiv.org/ abs/1807.11714 . · arXiv
  39. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems (NeurIPS) . Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/ 7209-learned-in-translation-contextualized-word-vectors.pdf .
  40. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint 1806.08730 , 2018. URL https://arxiv.org/abs/1806.08730 . · arXiv
  41. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the Association for Computational Linguistics (ACL) . Association for Computational Linguistics, 2019. URL https://arxiv.org/ abs/1902.01007 . · arXiv
  42. Richard T. McCoy and Tal Linzen. Non-entailed subsequences as a challenge for natural language inference. In Proceedings of the Society for Computational in Linguistics (SCiL) 2019 , 2019. URL https://scholarworks.umass.edu/scil/vol2/iss1/46/ .
  43. Aakanksha Naik, Abhilasha Ravichander, Norman M. Sadeh, Carolyn Penstein Rosé, and Graham Neubig. Stress test evaluation for natural language inference. In International Conference on Computational Linguistics (COLING) , 2018.
  44. Nikita Nangia and Samuel R. Bowman. Human vs. Muppet: A conservative estimate of human performance on the GLUE benchmark. In Proceedings of the Association of Computational Linguistics (ACL) . Association for Computational Linguistics, 2019. URL https: //woollysocks.github.io/assets/GLUE\_Human\_Baseline.pdf .
  45. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Advances in Neural Information Processing Systems (NeurIPS) . Curran Associates, Inc., 2017. URL https://openreview.net/pdf?id=BJJsrmfCZ .
  46. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2014. doi: 10.3115/v1/D14-1162. URL https://www.aclweb.org/anthology/D14-1162 . · DOI · Paper page
  47. Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint 1811.01088 , 2018. URL https: //arxiv.org/abs/1811.01088 . · arXiv
  48. Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) . Association for Computational Linguistics, 2019. URL https: //arxiv.org/abs/1808.09121 .
  49. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 2018. URL https: //www.aclweb.org/anthology/D18-1007 .
  50. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1264. URL http://aclweb.org/anthology/D16-1264 . · DOI · Paper page
  51. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series , 2011.
  52. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, 2018. doi: 10.18653/v1/N18-2002. URL https://www.aclweb. org/anthology/N18-2002 . · DOI · Paper page
  53. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. SocialIQa: Commonsense reasoning about social interactions. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2019. URL https://arxiv.org/abs/1904.09728 . · arXiv
  54. Nathan Schneider and Noah A Smith. A corpus and model integrating multiword expressions and supersenses. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) . Association for Computational Linguistics, 2015. URL https://www.aclweb.org/anthology/N15-1177 .
  55. Karin Kipper Schuler. Verbnet: A Broad-coverage, Comprehensive Verb Lexicon . PhD thesis, 2005. URL http://verbs.colorado.edu/~kipper/Papers/dissertation.pdf .
  56. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2013. URL https://www.aclweb.org/ anthology/D13-1170 .
  57. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. International Conference on Learning Representations (ICLR) , 2019. URL https://openreview.net/forum? id=SJzSgnRcKX .
  58. Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubramanian. Repurposing entailment for multi-hop question answering tasks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2948-2958, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1302. URL https://www.aclweb.org/anthology/N19-1302 . · DOI
  59. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations , 2019a. URL https://openreview. net/forum?id=rJ4km2R5t7 .
  60. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association of Computational Linguists , 2019. URL https://arxiv.org/ abs/1805.12471 . · arXiv
  61. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics (TACL) , 2018. URL https://www.aclweb.org/anthology/Q18-1042 .
  62. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACLHLT) . Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/ N18-1101 .
  63. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems (NeurIPS) , 2019.
  64. Fabio Massimo Zanzotto and Lorenzo Ferrone. Have you lost the thread? discovering ongoing conversations in scattered dialog blocks. ACM Transactions on Interactive Intelligent Systems (TiiS) , 2017.
  65. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. 2018. URL https://www.aclweb.org/anthology/ D18-1009 .
  66. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885 , 2018. · arXiv
  67. Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , 2019. URL https://arxiv.org/ abs/1904.01130 . · arXiv
  68. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, 2018. doi: 10.18653/v1/N18-2003. URL https://www.aclweb.org/anthology/N18-2003 . · DOI · Paper page

Cited by

Sorted by citations

Authors
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R.
Affiliations
  • New York University (US)
  • Meta (United States) (US)
  • University of Washington (US)
  • Google DeepMind (United Kingdom) (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1905.00537
arXiv
1905.00537
OpenAlex
W2943552823
Cited by
1,051
References
4
Citations by year
2026: 8 · 2025: 69 · 2024: 142 · 2023: 362 · 2022: 219 · 2021: 157 · 2020: 78 · 2019: 16
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
SuperGLUE, GLUE benchmark, language comprehension, benchmarking, natural language processing evaluation, transfer learning, pre-training, software toolkit

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review