CoChat
Sign Up
PreprintCited by 672Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Open access
Srivastava, Aarohi, Rastogi, Abhinav, Rao, Abhishek, Shoeb, Abu Awal Md, Abid, Abubakar, Fisch, Adam, Brown, Adam R., Santoro, Adam +443 more·2022·arXiv
DOI 10.48550/arxiv.2206.04615arXiv 2206.04615
Open this paper in CoChat — free accountOpen PDF

Abstract

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.04615

Cites

1 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Pretrained LLMs perform some traditionally symbolic tasks well with only a few training examples.

    “even performing some traditionally symbolic tasks well with only a few training examples [61].”

    From Symbols and grounding in large language models (2023) · cites this paper as [61]

    Section 2 (“What is in BIG-bench?”, page not provided) says the benchmark evaluates pretrained models without task-specific fine-tuning in zero- and few-shot settings. Section 6.1 reports an improving ability to propose legal chess moves, a symbolic-type skill, but it does not establish that models perform symbolic tasks well in the few-shot setting. The paper also says overall BIG-bench performance remains poor relative to expert humans. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks.

    “The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks”

    From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Srivastava et al., 2022

    Section 1.2 (page unavailable) says existing language-model benchmarks often have restricted scope, citing language understanding, summarization, and trivia-question answering. Section 2.1 contrasts fixed input-and-target tasks with programmatic tasks that can interact with a model over multiple rounds. These passages support the claim's general direction, but do not establish that benchmarks historically relied on static, non-interactive tasks. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • Many benchmark sets do not allow for arbitrary generation of novel questions, or draw data from publicly available sources that LLM trainers have access to.

    “Many benchmark sets do not allow for arbitrary generation of novel questions, or worse, draw data from publicly available sources-the same sources LLM trainers have access to [51, 39].”

    From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [39]

    Section 2.3.1 says many BIG-bench tasks use text available on the internet, making indirect training-data leakage possible. Section 2.1 says about 80% of BIG-bench tasks use fixed lists of inputs and targets, which supports the concern about generating novel questions. The paper does not establish that many benchmark sets generally lack that capability. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Using training FLOPs leads to reasonable predictions of language model performance with and without chain-of-thought on BigBench-Hard tasks.

    “Surprisingly, we observe that using training FLOPs leads to reasonable predictions of LM performance with and without CoT on BBH tasks, possibly due to the denoising effect of aggregation over all tasks.”

    From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]

    Section 3.1 (page unavailable) reports that aggregate BIG-bench performance improves with compute scale. It does not establish that training FLOPs yield reasonable predictions on BigBench-Hard tasks, either with or without chain-of-thought prompting; Section 6.1 instead cautions that newer models can outperform naive extrapolations. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The paper uses the same prompt for all models evaluated to standardize evaluation.

    “In order to standardize our evaluation, we use the same prompt for all models evaluated, following the convention established by [83–85].”

    From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [85]

    Section 2.1 (page unavailable) describes BIG-bench tasks as shared input-and-target examples evaluated through a common model interface, which supports a standardized evaluation setup. It does not say that every model received the same prompt or establish that as an explicit convention; the paper also examines different prompt formats in Section 3.5 (page unavailable). (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 141

  1. Scott Alexander. A very unlikely chess game, 2020. URL https://slatestarcodex.com/2020/01/06/a-very-unlikely-chessgame/ . (cited on pp. 29 and 30)
  2. Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 245-256. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/alon20a.html . (cited on p. 37)
  3. Miriam Amin and Manuel Burghardt. A survey on approaches to computational humor generation. In Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature , pp. 29-41, Online, December 2020. International Committee on Computational Linguistics. URL https://aclanthology. org/2020.latechclfl-1.4 . (cited on p. 38)
  4. Brandon Amos and J. Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks, 2017. URL https: //arxiv.org/abs/1703.00443 . (cited on p. 38)
  5. Salvatore Attardo. Humor in language. In Oxford Research Encyclopedia of Linguistics . Oxford University Press, 2017. doi: 10. 1093/acrefore/9780199384655.013.342. URL https://oxfordre.com/linguistics/view/10.1093/acrefore/9780199384655. 001.0001/acrefore-9780199384655-e-342 . (cited on p. 38) · DOI
  6. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: //arxiv.org/abs/2108.07732 . (cited on p. 4)
  7. Satanjeev Banerjee and Ted Pedersen. Extended gloss overlaps as a measure of semantic relatedness. In IJCAI'03: Proceedings of the 18th International Joint Conference on Artificial Intelligence , pp. 805-810, San Francisco, 2003. Morgan Kaufmann. doi: 10.5555/1630659.1630775. URL https://dl.acm.org/doi/10.5555/1630659.1630775 . (cited on p. 36) · DOI
  8. Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review , 104(3):671-732, 2016. URL http://www.jstor.org/stable/24758720 . (cited on p. 31)
  9. Gregor Betz, Christian Voigt, and Kyle Richardson. Critical thinking for language models, 2020. URL https://arxiv.org/abs/ 2009.07185 . (cited on p. 33)
  10. Stella Biderman and Edward Raff. Neural language models are effective plagiarists, 2022. URL https://arxiv.org/abs/2201. 07406 . (cited on p. 4)
  11. Julia Birke and Anoop Sarkar. A clustering approach for nearly unsupervised recognition of nonliteral language. In 11th Conference of the European Chapter of the Association for Computational Linguistics , pp. 329-336, Trento, Italy, April 2006. Association for Computational Linguistics. URL https://aclanthology.org/E06-1042 . (cited on p. 35)
  12. Nicholas Boillot. Vector forms as a foreign language, 24 June 2019. URL https://www.fluate.net/en/travaux/vectoglyph . (cited on pp. 31 and 33)
  13. Paul F. Boller, Jr. and John George. They Never Said It: A Book of Fake Quotes, Misquotes, and Misleading Attributions . Oxford University Press, Oxford, 1989. (cited on p. 35)
  14. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings. neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html . (cited on pp. 17, 29, 31, and 33)
  15. Bozhidar Bozhanov and Ivan Derzhanski. Rosetta stone linguistic problems. In Proceedings of the Fourth Workshop on Teaching NLP and CL , pp. 1-8, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https: //aclanthology.org/W13-3401 . (cited on p. 34)
  16. Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable Forth interpreter. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 547-556, 2017. URL https: //arxiv.org/abs/1605.06640 . (cited on p. 34)
  17. Gwern Branwen. GPT-3 creative fiction. Gwern.net , June 2020. URL https://www.gwern.net/GPT-3 . (cited on pp. 19, 38, and 39)
  18. Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review , 78(1):1-3, 1950. (cited on pp. 8 and 11)
  19. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html . (cited on pp. 4, 7, 8, 10, 11, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40, and 41)
  20. Corrado Böhm. On a family of Turing machines and the related programming language. ICC Bulletin , 3:187-194, 1964. (cited on p. 38)
  21. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pp. 2633-2650. USENIX Association, August 2021. URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting . (cited on p. 19)
  22. Nathanael Chambers. Labeling documents with timestamps: Learning from their time expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 98-106, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1011 . (cited on p. 31)
  23. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 1691-1703. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/chen20s.html . (cited on p. 29)
  24. Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. https://openreview.net/pdf?id= H1gfOiAqYm , 2019b. (cited on p. 34)
  25. Kate Crawford. The trouble with bias. https://www.youtube.com/watch?v=fMym\_BKWQzk , 2017. Keynote address, NIPS 2017, Long Beach CA. Dec. 5, 2017. (cited on p. 29)
  26. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 933-941, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305478. URL https://dl.acm.org/doi/10.5555/3305381. 3305478 . (cited on p. 10) · DOI
  27. Wayne Davis. Implicature. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Fall 2019 edition, 2019. (cited on p. 33)
  28. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. Finding contradictions in text. In Proceedings of ACL-08: HLT , pp. 1039-1047, Columbus, Ohio, June 2008. Association for Computational Linguistics. URL https: //aclanthology.org/P08-1118 . (cited on p. 32)
  29. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. RobustFill: Neural program learning under noisy I/O. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 990-998, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305484. URL https://dl.acm.org/doi/10.5555/3305381.3305484 . (cited on p. 34) · DOI
  30. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. Beyond English-centric multilingual machine translation. Journal of Machine Learning Research , 22(107): 1-48, 2021. URL http://jmlr.org/papers/v22/20-1307.html . (cited on p. 20)
  31. Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database . MIT Press, Cambridge, MA, 1998. (cited on p. 33)
  32. Jerry A. Fodor. The Language of Thought . Harvard University Press, Cambridge, MA, 1975. (cited on p. 30)
  33. Evgeniy Gabrilovich and Shaul Markovitch. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In IJCAI'07: Proceedings of the 20th International Joint Conference on Artifical Intelligence , pp. 1606-1611, San Francisco, 2007. Morgan Kaufmann. doi: 10.5555/1625275.1625535. URL https://dl.acm.org/doi/10.5555/1625275.1625535 . (cited on p. 36) · DOI
  34. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics , 9: 346-361, 04 2021. doi: 10.1162/tacl\_a\_00370. URL https://doi.org/10.1162/tacl\_a\_00370 . (cited on pp. 29 and 39) · DOI
  35. Arthur S. Goldberger. Structural equation methods in the social sciences. Econometrica , 40(6):979-1001, 1972. URL http://www.jstor.org/stable/1913851 . (cited on p. 32)
  36. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pp. 581-586, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https://aclanthology.org/P11-2102 . (cited on p. 39)
  37. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pp. 1321-1330. PMLR, 06-11 Aug. 2017. URL https://proceedings.mlr.press/v70/guo17a. html . (cited on p. 11)
  38. Samuel Gyasi Obeng. The proverb as a mitigating and politeness strategy in Akan discourse. Anthropological Linguistics , 38(3): 521-549, 1996. URL http://www.jstor.org/stable/30028601 . (cited on p. 32)
  39. Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171, Dec. 2020. doi: 10.1162/tacl\_a\_00306. URL https://doi.org/10.1162/tacl\_a\_00306 . (cited on p. 31) · DOI
  40. Joseph Y. Halpern. Actual causality . MIT Press, Cambridge, MA, 2016. (cited on p. 29)
  41. Maria Hanzén. When in Rome, do as the Romans do: Proverbs as a part of EFL teaching. Master's thesis, Jönköping University, School of Education and Communication, Jönköping, 2007. URL http://www.diva-portal.org/smash/get/diva2: 3499/fulltext01.pdf . (cited on p. 40)
  42. Irene Heim. On the projection problem for presuppositions. In Paul Portner and Barbara H. Partee (eds.), Formal Semantics The Essential Readings , pp. 249-260. Blackwell, Oxford, 1983. (cited on p. 37)
  43. Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Hkg4TI9xl . (cited on pp. 11 and 12)
  44. Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 2712-2721. PMLR, 09-15 June 2019. URL https: //proceedings.mlr.press/v97/hendrycks19a.html . (cited on p. 11)
  45. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS, 2021a. URL https: //arxiv.org/abs/2105.09938 . (cited on pp. 4, 37, and 41)
  46. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021b. URL https: //openreview.net/forum?id=d7KBjmI3GmQ . (cited on pp. 25, 31, 35, 38, and 40)
  47. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips. cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html . (cited on pp. 5 and 32)
  48. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URL https: //arxiv.org/abs/2102.01293 . (cited on p. 41)
  49. Keith J. Holyoak. Analogy and relational reasoning. In Keith J. Holyoak and Robert G. Morrison (eds.), The Oxford Handbook of Thinking and Reasoning . Oxford University Press, Oxford, 2012. URL https://www.oxfordhandbooks.com/view/10.1093/ oxfordhb/9780199734689.001.0001/oxfordhb-9780199734689-e-13 . (cited on p. 28)
  50. Yufang Hou. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 1428-1438, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.132. URL https://aclanthology.org/2020.acl-main.132 . (cited on p. 29)
  51. Yufang Hou, Katja Markert, and Michael Strube. Global inference for bridging anaphora resolution. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 907-917, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology.org/N131111 . (cited on p. 29)
  52. Hrisztalina Hrisztova-Gotthardt and Melita Aleksa Varga. Introduction to Paremiology: A Comprehensive Guide to Proverb Studies . De Gruyter Open, Warsaw, 2015. URL https://www.degruyter.com/document/doi/10.2478/9783110410167/html . (cited on p. 32) · DOI
  53. Thad Hughes and Daniel Ramage. Lexical semantic relatedness with random graph walks. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , pp. 581-589, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/D07-1061 . (cited on p. 36)
  54. Jay J. Jiang and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th Research on Computational Linguistics International Conference , pp. 19-33, Taipei, Taiwan, August 1997. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). URL https://aclanthology.org/O97-1002 . (cited on p. 36)
  55. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438, 2020. doi: 10.1162/tacl\_a\_00324. URL https://doi.org/10. 1162/tacl\_a\_00324 . (cited on pp. 11 and 12) · DOI
  56. Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS'15 , volume 1, pp. 190-198, Cambridge, MA, USA, 2015. MIT Press. doi: 10.5555/2969239.2969261. URL https://dl.acm.org/doi/10.5555/2969239.2969261 . (cited on p. 34) · DOI
  57. Lauri Karttunen. Simple and phrasal implicatives. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pp. 124-131, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1020 . (cited on p. 32)
  58. Faisal Khan, Bilge Mutlu, and Jerry Zhu. How do humans teach: On curriculum learning and teaching dimension. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper/2011/file/ f9028faec74be6ec9b852b0a542e2f39-Paper.pdf . (cited on p. 38)
  59. Milton King and Paul Cook. Evaluating approaches to personalizing language models. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 2461-2469, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.299 . (cited on p. 38)
  60. Christo Kirov and Ryan Cotterell. Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate. Transactions of the Association for Computational Linguistics , 6:651-665, 12 2018. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00247. URL https://doi.org/10.1162/tacl\_a\_00247 . (cited on p. 34) · DOI
  61. Joshua Knobe. Intentional action and side effects in ordinary language. Analysis , 63, 07 2003. doi: 10.1111/ 1467-8284.00419. URL https://www.researchgate.net/publication/28763794\_Intentional\_Action\_and\_Side\_Effects\_ in\_Ordinary\_Language . (cited on p. 29)
  62. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics , 6:317-328, 2018. doi: 10.1162/tacl\_a\_00023. URL https://aclanthology.org/Q18-1023 . (cited on p. 35) · DOI
  63. Alexander W. Kocurek, Ethan Jerzak, and Rachel Etta Rudolph. Against conventional wisdom. Philosophers' Imprint , 20(22): 1-27, 2020. URL http://hdl.handle.net/2027/spo.3521354.0020.022 . (cited on p. 29)
  64. Jarmo Korhonen. Sprichwörter und zweisprachige lexikographie: Deutsch-schwedische und deutsch-finnische wörtebücher im vergleich. In C. Földes (ed.), Phraseologie disziplinär und interdisziplinär , pp. 537-549. Gunter Narr Verlag, 2009. (cited on p. 40)
  65. Mahnaz Koupaee and William Yang Wang. WikiHow: A large scale text summarization dataset, 2018. URL https://arxiv. org/abs/1810.09305 . (cited on p. 5)
  66. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics , 10:50-72, 01 2022. doi: 10.1162/tacl\_a\_00447. URL https://doi.org/10.1162/tacl\_a\_00447 . (cited on p. 20) · DOI
  67. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 08 2019. doi: 10.1162/tacl\_a\_00276. URL https://doi.org/10.1162/tacl\_a\_00276 . (cited on pp. 5 and 37) · DOI
  68. Kevin Lacker. Giving GPT-3 a Turing test. Kevin Lacker's blog , July 2020. URL https://lacker.io/ai/2020/07/06/givinggpt-3-a-turing-test.html . (cited on pp. 33 and 41)
  69. George Lakoff and Mark Johnson. Metaphors We Live By . University of Chicago Press, Chicago, 2008. (cited on pp. 33 and 35)
  70. Guillaume Lample and François Charton. Deep learning for symbolic mathematics, 2019. URL https://arxiv.org/abs/1912. 01412 . (cited on p. 38)
  71. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 986-995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL https://aclanthology.org/I17-1099 . (cited on p. 40)
  72. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics , 4:521-535, 12 2016. doi: 10.1162/tacl\_a\_00115. URL https://doi.org/10.1162/tacl\_a\_00115 . (cited on p. 39) · DOI
  73. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019. URL https://arxiv.org/abs/1907. 11692 . (cited on pp. 39 and 41)
  74. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. In Vivek Nigam, Tajana Ban Kirigin, Carolyn Talcott, Joshua Guttman, Stepan Kuznetsov, Boon Thau Loo, and Mitsuhiro Okada (eds.), Logic, Language, and Security . Springer, Cham, 2020. URL https://www.springerprofessional. de/en/gender-bias-in-neural-natural-language-processing/18531692 . (cited on p. 33)
  75. Jihang Mao and Wanli Liu. A BERT-based approach for automatic humor detection and scoring. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019) , pp. 197-202, 2019. URL http://ceur-ws.org/Vol-2421/HAHA\_paper\_8.pdf . (cited on p. 39)
  76. Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence, 2020. URL https://arxiv.org/abs/ 2002.06177 . (cited on p. 30)
  77. Katja Markert, Yufang Hou, and Michael Strube. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 795-804, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1084 . (cited on p. 29)
  78. R. Thomas McCoy, Robert Frank, and Tal Linzen. Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks. Transactions of the Association for Computational Linguistics , 8:125-140, 01 2020. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00304. URL https://doi.org/10.1162/tacl\_a\_00304 . (cited on p. 34) · DOI
  79. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https: //arxiv.org/abs/1609.07843 . (cited on p. 31)
  80. John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 6905-6916. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/miller20a.html . (cited on p. 39)
  81. David Milne and Ian H. Witten. An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: An Evolving Synergy , pp. 25-30, Menlo Park, 2008. Association for the Advancement of Artificial Intelligence. URL https://www.aaai.org/Papers/Workshops/2008/WS08-15/WS08-15-005.pdf . (cited on p. 36)
  82. Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Natural reference to objects in a visual domain. In Proceedings of the 6th International Natural Language Generation Conference . Association for Computational Linguistics, July 2010. URL https://aclanthology.org/W10-4210 . (cited on p. 38)
  83. Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Generating expressions that refer to visible objects. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 1174-1184, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https:// aclanthology.org/N13-1137 . (cited on p. 38)
  84. Elham Mohammadi, Hessam Amini, and Leila Kosseim. CLaC at CLPsych 2019: Fusion of neural features and predicted class probabilities for suicide risk assessment based on online posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology , pp. 34-38, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-3004. URL https://aclanthology.org/W19-3004 . (cited on p. 39) · DOI
  85. Jane Morris and Graeme Hirst. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics , 17(1):21-48, 1991. URL https://aclanthology.org/J91-1002 . (cited on p. 36)
  86. Karl Mulligan, Robert Frank, and Tal Linzen. Structure here, bias there: Hierarchical generalization by jointly learning syntactic transformations. In Proceedings of the Society for Computation in Linguistics 2021 , pp. 125-135, Online, February 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.scil-1.12 . (cited on p. 34)
  87. Gregory L. Murphy. Comprehending complex concepts. Cognitive Science , 12(4):529-562, 1988. doi: https://doi.org/10.1207/ s15516709cog1204\_2. URL https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1204\_2 . (cited on p. 30) · DOI
  88. Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 8558cb408c1d76621371888657d2eb1d-Abstract.html . (cited on p. 11)
  89. Anthony M. Paul. Figurative language. Philosophy & Rhetoric , 3(4):225-248, 1970. URL http://www.jstor.org/stable/ 40237206 . (cited on p. 35)
  90. Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann, San Francisco, 1988. (cited on p. 32)
  91. Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, Cambridge, 2000. (cited on p. 30)
  92. Tony A. Plate. Distributed representations and nested compositional structure . PhD thesis, University of Toronto, Toronto, 1994. (cited on p. 29)
  93. Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving, 2020. URL https://arxiv. org/abs/2009.03393 . (cited on pp. 34 and 38)
  94. Willard V.O. Quine. Main trends in recent philosophy: Two dogmas of empiricism. The Philosophical Review , 60(1):20-43, 1951. URL http://www.jstor.org/stable/2181906 . (cited on p. 29)
  95. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/languagemodels.pdf . (cited on pp. 10, 30, 33, 35, 37, 38, and 39)
  96. Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pp. 777-789, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology. org/D12-1071 . (cited on p. 41)
  97. Ian Ravenscroft. Folk psychology as a theory. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Summer 2019 edition, 2019. (cited on p. 32)
  98. Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics , 7:249-266, 2019. doi: 10.1162/tacl\_a\_00266. URL https://aclanthology. org/Q19-1016 . (cited on pp. 30 and 40) · DOI
  99. Joseph Reisinger and Raymond J. Mooney. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 109-117, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology. org/N10-1013 . (cited on p. 36)
  100. Philip Resnik. Using information content to evaluate semantic similarity in a taxonomy. In IJCAI'95: Proceedings of the 14th International Joint Conference on Artificial Intelligence, Volume 1 , pp. 448-453, San Francisco, 1995. Morgan Kaufmann. doi: 10.5555/1625855.1625914. URL https://dl.acm.org/doi/10.5555/1625855.1625914 . (cited on p. 36) · DOI
  101. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. AAAI Spring Symposium , 2011. URL http://commonsensereasoning.org/2011/papers/Roemmele.pdf . (cited on p. 32)
  102. Joshua S. Rule. The child as hacker: Building more human-like models of learning . PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2020. URL https://hdl.handle.net/1721.1/129232 . (cited on p. 34)
  103. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 . (cited on pp. 9 and 25)
  104. Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 624-643, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.52. URL https://aclanthology.org/2021.naacl-main.52 . (cited on p. 41) · DOI
  105. Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021b. URL https://openreview.net/ forum?id=fe\_hCc4RBrg . (cited on p. 4)
  106. Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. Diagram understanding in geometry questions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, Menlo Park, CA, Jun. 2014. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9146 . (cited on p. 34)
  107. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL http: //arxiv.org/abs/1804.04235 . (cited on p. 43)
  108. Ekaterina Shutova. Automatic metaphor interpretation as a paraphrasing task. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 1029-1037, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology.org/N10-1147 . (cited on p. 35)
  109. Ekaterina Shutova and Simone Teufel. Metaphor corpus annotated for source-target domain mappings. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) , Valletta, Malta, May 2010. European Language Resources Association. URL http://www.lrec-conf.org/proceedings/lrec2010/pdf/612\_Paper.pdf . (cited on p. 35)
  110. Damien Sileo, Tim Van de Cruys, Camille Pradel, and Philippe Muller. DiscSense: Automated semantic analysis of discourse markers. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 991-999, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.125 . (cited on p. 31)
  111. Gurdeep Singh, Kaustubh Dhole, Priyadarshini P. Pai, and Sukanta Mondal. SPRINGS: Prediction of protein-protein interaction sites using artificial neural networks. Journal of Proteomics & Computational Biology , 1:7, 2014. URL https: //www.avensonline.org/fulltextarticles/JPCB-2572-8679-01-0001.html . (cited on p. 37)
  112. Douglas R. Smith. The synthesis of LISP programs from examples: A survey. In Alan W. Biermann, Gerhard Guiho, and Yves Kodratoff (eds.), Automatic Program Construction Techniques , pp. 307-324. Macmillan, New York, 1984. (cited on p. 34)
  113. Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences , 11(1):1-23, 1988. doi: 10.1017/ S0140525X00052432. (cited on p. 30)
  114. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://aclanthology.org/D13-1170 . (cited on p. 31)
  115. Shashank Srivastava, Snigdha Chaturvedi, and Tom Mitchell. Inferring interpersonal relations in narrative summaries. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16 , pp. 2807-2813, Menlo Park, CA, 2016. Association for the Advancement of Artificial Intelligence. doi: 10.5555/3016100.3016294. URL https://dl.acm.org/doi/10. 5555/3016100.3016294 . (cited on p. 34) · DOI
  116. Robert Stalnaker. Assertion. In P. Cole (ed.), Pragmatics , Syntax and Semantics 9, pp. 315-332. Brill, Leiden, 1978. (cited on p. 32)
  117. Bernd Steinbach and Roman Kohut. Neural networks - a model of boolean functions. 5th International Workshop on Boolean Problems, Freiburg, Sept. 2002. , 2002. URL https://www.researchgate.net/publication/246931125\_Neural\_Networks\_-\_A\_Model\_of\_Boolean\_Functions . (cited on p. 29)
  118. Andreas Stöckl. Watching a language model learning chess. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) , pp. 1369-1379, Held Online, September 2021. INCOMA Ltd. URL https://aclanthology.org/2021.ranlp-1.153 . (cited on p. 4)
  119. Kevin Stowe, Leonardo Ribeiro, and Iryna Gurevych. Metaphoric paraphrase generation, 2020. URL https://arxiv.org/abs/ 2002.12854 . (cited on p. 35)
  120. Michael Strube and Simone Paolo Ponzetto. Wikirelate! Computing semantic relatedness using Wikipedia. In AAAI'06: Proceedings of the 21st National Conference on Artificial Intelligence , volume 2, pp. 1419-1424. Association for the Advancement of Artificial Intelligence, 2006. doi: 10.5555/1597348.1597414. URL https://dl.acm.org/doi/10.5555/1597348. 1597414 . (cited on p. 36) · DOI
  121. Saku Sugawara, Hikaru Yokono, and Akiko Aizawa. Prerequisite skills for reading comprehension: Multi-perspective analysis of MCTest datasets and systems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, Menlo Park, CA, Feb. 2017. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/ view/10957 . (cited on p. 35)
  122. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks, 2014. URL https: //arxiv.org/abs/1409.3215 . (cited on p. 4)
  123. Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Tunuguntla Deepak, and Oskar van der Wal. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models , 2022. URL https://openreview.net/forum?id=rK-7NhfSIW5 . (cited on pp. 18 and 20)
  124. Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. Learning to recommend quotes for writing. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15 , pp. 2453-2459, Menlo Park, CA, 2015. Association for the Advancement of Artificial Intelligence. URL https://dl.acm.org/doi/10.5555/2886521.2886662 . (cited on p. 28) · DOI
  125. Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15 , pp. 1923-1929, 2015. URL https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/view/10957/10931 . (cited on p. 38)
  126. George Tsatsaronis, Iraklis Varlamis, Michalis Vazirgiannis, and Kjetil Nørvåg. Omiotis: A thesaurus-based measure of text relatedness. In Wray Buntine, Marko Grobelnik, Dunja Mladenić, and John Shawe-Taylor (eds.), Machine Learning and Knowledge Discovery in Databases , pp. 742-745, Berlin, 2009. Springer. (cited on p. 36)
  127. Shikhar Vashishth, Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Dating documents using graph convolution networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1605-1615, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ v1/P18-1149. URL https://aclanthology.org/P18-1149 . (cited on p. 31)
  128. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 30, pp. 5998--6008. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf . (cited on pp. 10 and 38)
  129. Jette Viethen and Robert Dale. The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference , pp. 59-67, Salt Fork, Ohio, USA, June 2008. Association for Computational Linguistics. URL https://aclanthology.org/W08-1109 . (cited on p. 38)
  130. Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers , pp. 176-187, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-1017 . (cited on p. 35)
  131. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019a. URL https://proceedings.neurips.cc/paper/2019/hash/ 4496bf24afe7fab6f046bf4923da8de6-Abstract.html . (cited on pp. 5, 7, 29, 32, and 41)
  132. David Wechsler. Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) . Pearson, San Antonio, 2008. (cited on p. 38)
  133. Ludwig Wittgenstein. Philosophical investigations . Basil Blackwell, Oxford, 1953. (cited on p. 29)
  134. Shijie Wu, Ryan Cotterell, and Mans Hulden. Applying the transformer to character-level transduction, 2020. URL https: //arxiv.org/abs/2005.10213 . (cited on p. 41)
  135. Scott Cheng-Hsin Yang and Patrick Shafto. Explainable artificial intelligence via Bayesian teaching. Workshop on Teaching Machines, Robots, and Humans, NIPS 2017 , 2017. URL http://shaftolab.com/assets/papers/yangShafto\_NIPS\_2017\_ machine\_teaching.pdf . (cited on p. 38)
  136. Eric Yeh, Daniel Ramage, Christopher D. Manning, Eneko Agirre, and Aitor Soroa. WikiWalk: Random walks on Wikipedia for semantic relatedness. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4) , pp. 41-49, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https: //aclanthology.org/W09-3206 . (cited on p. 36)
  137. Wenpeng Yin and Yulong Pei. Optimizing sentence modeling and selection for document summarization. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI-15 , pp. 1383-1389, 2015. doi: 10.5555/2832415.2832442. URL https://dl.acm.org/doi/10.5555/2832415.2832442 . (cited on p. 32) · DOI
  138. Eliezer Yudkowsky. Artificial intelligence as a positive and negative factor in global risk. In Nick Bostrom and Milan M. Ćirković (eds.), Global Catastrophic Risks , pp. 308-345. Oxford University Press, Oxford, 2008. URL https://web.archive.org/web/ 20210125025955/https://intelligence.org/files/AIPosNegFactor.pdf . (cited on p. 38)
  139. Omnia Zayed, John Philip McCrae, and Paul Buitelaar. Figure me out: A gold standard dataset for metaphor interpretation. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 5810-5819, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.712 . (cited on p. 35)
  140. Meishan Zhang, Yue Zhang, and Guohong Fu. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pp. 2449-2460, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https://aclanthology.org/C16-1231 . (cited on p. 39)
  141. Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, Menlo Park, CA, Mar. 2015. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9761 . (cited on p. 38)

Cited by

Sorted by citations

Related

Authors
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A., La, A., Lampinen, A., Zou, A., Jiang, A., Chen, A., Vuong, A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabassum, A., Menezes, A., Kirubarajan, A., Mullokandov, A., Sabharwal, A., Herrick, A., Efrat, A., Erdem, A., Karakaş, A., Roberts, B. R., Loe, B. S., Zoph, B., Bojanowski, B., Özyurt, B., Hedayatnia, B., Neyshabur, B., Inden, B., Stein, B., Ekmekci, B., Lin, B. Y., Howald, B., Orinion, B., Diao, C., Dour, C., Stinson, C., Argueta, C., Ramírez, C. F., Singh, C., Rathkopf, C., Meng, C., Baral, C., Wu, C., Callison-Burch, C., Waites, C., Voigt, C., Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C., Ashcraft, C., Garbacea, C., Sileo, D., Garrette, D., Hendrycks, D., Kilman, D., Roth, D., Freeman, D., Khashabi, D., Levy, D., González, D. M., Perszyk, D., Hernandez, D., Chen, D., Ippolito, D., Gilboa, D., Dohan, D., Drakard, D., Jurgens, D., Datta, D., Ganguli, D., Emelin, D., Kleyko, D., Yuret, D., Chen, D., Tam, D., Hupkes, D., Misra, D., Buzan, D., Mollo, D. C., Yang, D., Lee, D.-H., Schrader, D., Shutova, E., Cubuk, E. D., Segal, E., Hagerman, E., Barnes, E., Donoway, E., Pavlick, E., Rodola, E., Lam, E., Chu, E., Tang, E., Erdem, E., Chang, E., Chi, E. A., Dyer, E., Jerzak, E., Kim, E., Manyasi, E. E., Zheltonozhskii, E., Xia, F., Siar, F., Martínez-Plumed, F., Happé, F., Chollet, F., Rong, F., Mishra, G., Winata, G. I., de Melo, G., Kruszewski, G., Parascandolo, G., Mariani, G., Wang, G., Jaimovitch-López, G., Betz, G., Gur-Ari, G., Galijasevic, H., Kim, H., Rashkin, H., Hajishirzi, H., Mehta, H., Bogar, H., Shevlin, H., Schütze, H., Yakura, H., Zhang, H., Wong, H. M., Ng, I., Noble, I., Jumelet, J., Geissinger, J., Kernion, J., Hilton, J., Lee, J., Fisac, J. F., Simon, J. B., Koppel, J., Zheng, J., Zou, J., Kocoń, J., Thompson, J., Wingfield, J., Kaplan, J., Radom, J., Sohl-Dickstein, J., Phang, J., Wei, J., Yosinski, J., Novikova, J., Bosscher, J., Marsh, J., Kim, J., Taal, J., Engel, J., Alabi, J., Xu, J., Song, J., Tang, J., Waweru, J., Burden, J., Miller, J., Balis, J. U., Batchelder, J., Berant, J., Frohberg, J., Rozen, J., Hernandez-Orallo, J., Boudeman, J., Guerr, J., Jones, J., Tenenbaum, J. B., Rule, J. S., Chua, J., Kanclerz, K., Livescu, K., Krauth, K., Gopalakrishnan, K., Ignatyeva, K., Markert, K., Dhole, K. D., Gimpel, K., Omondi, K., Mathewson, K., Chiafullo, K., Shkaruta, K., Shridhar, K., McDonell, K., Richardson, K., Reynolds, L., Gao, L., Zhang, L., Dugan, L., Qin, L., Contreras-Ochando, L., Morency, L.-P., Moschella, L., Lam, L., Noble, L., Schmidt, L., He, L., Colón, L. O., Metz, L., Şenel, L. K., Bosma, M., Sap, M., ter Hoeve, M., Farooqi, M., Faruqui, M., Mazeika, M., Baturan, M., Marelli, M., Maru, M., Quintana, M. J. R., Tolkiehn, M., Giulianelli, M., Lewis, M., Potthast, M., Leavitt, M. L., Hagen, M., Schubert, M., Baitemirova, M. O., Arnaud, M., McElrath, M., Yee, M. A., Cohen, M., Gu, M., Ivanitskiy, M., Starritt, M., Strube, M., Swędrowski, M., Bevilacqua, M., Yasunaga, M., Kale, M., Cain, M., Xu, M., Suzgun, M., Walker, M., Tiwari, M., Bansal, M., Aminnaseri, M., Geva, M., Gheini, M., T, M. V., Peng, N., Chi, N. A., Lee, N., Krakover, N. G.-A., Cameron, N., Roberts, N., Doiron, N., Martinez, N., Nangia, N., Deckers, N., Muennighoff, N., Keskar, N. S., Iyer, N. S., Constant, N., Fiedel, N., Wen, N., Zhang, O., Agha, O., Elbaghdadi, O., Levy, O., Evans, O., Casares, P. A. M., Doshi, P., Fung, P., Liang, P. P., Vicol, P., Alipoormolabashi, P., Liao, P., Liang, P., Chang, P., Eckersley, P., Htut, P. M., Hwang, P., Miłkowski, P., Patil, P., Pezeshkpour, P., Oli, P., Mei, Q., Lyu, Q., Chen, Q., Banjade, R., Rudolph, R. E., Gabriel, R., Habacker, R., Risco, R., Millière, R., Garg, R., Barnes, R., Saurous, R. A., Arakawa, R., Raymaekers, R., Frank, R., Sikand, R., Novak, R., Sitelew, R., LeBras, R., Liu, R., Jacobs, R., Zhang, R., Salakhutdinov, R., Chi, R., Lee, R., Stovall, R., Teehan, R., Yang, R., Singh, S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff, S., Gehrmann, S., Schuster, S., Sadeghi, S., Hamdan, S., Zhou, S., Srivastava, S., Shi, S., Singh, S., Asaadi, S., Gu, S. S., Pachchigar, S., Toshniwal, S., Upadhyay, S., Shyamolima, Debnath, Shakeri, S., Thormeyer, S., Melzi, S., Reddy, S., Makini, S. P., Lee, S.-H., Torene, S., Hatwar, S., Dehaene, S., Divic, S., Ermon, S., Biderman, S., Lin, S., Prasad, S., Piantadosi, S. T., Shieber, S. M., Misherghi, S., Kiritchenko, S., Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L., Desbordes, T., Rothschild, T., Phan, T., Wang, T., Nkinyili, T., Schick, T., Kornev, T., Tunduny, T., Gerstenberg, T., Chang, T., Neeraj, T., Khot, T., Shultz, T., Shaham, U., Misra, V., Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V., Prabhu, V. U., Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, Y., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z., Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z., & Wu, Z.
Affiliations
Shammie
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.04615
arXiv
2206.04615
OpenAlex
W4281690148
Semantic Scholar
bd1331b233e84bab7eba503abc60b31ac08e7881
Cited by
569
References
0
Citations by year
2026: 29 · 2025: 114 · 2024: 169 · 2023: 232 · 2022: 24 · 2021: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
model scaling, sparse transformer, GPT, model calibration, social bias, commonsense reasoning, mathematical reasoning, prompting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review