The DOI matches its DataCite record; 527 of the 528 references it cites were found in a registry or an index.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Open accessAbstract
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even performing some traditionally symbolic tasks well with only a few training examples [61].”
From Symbols and grounding in large language models (2023) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Srivastava et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many benchmark sets do not allow for arbitrary generation of novel questions, or worse, draw data from publicly available sources-the same sources LLM trainers have access to [51, 39].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [39]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Surprisingly, we observe that using training FLOPs leads to reasonable predictions of LM performance with and without CoT on BBH tasks, possibly due to the denoising effect of aggregation over all tasks.”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In order to standardize our evaluation, we use the same prompt for all models evaluated, following the convention established by [83–85].”
From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [85]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One might thus posit, with some justification from the above-mentioned studies, that the emergence of skills arises from training on related tasks that were implicitly solved while solving next-word prediction in the training dataset.This is indeed our starting point.”
From A Theory for Emergence of Complex Skills in Language Models (2023) · cites this paper as Srivastava et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prominent among these are multi-task benchmarks such as MMLU [325] and BIG-bench [326], which evaluate knowledge and skills across diverse domains and scenarios.”
From A Survey of Large Language Models (2026) · cites this paper as [326]
The cited source says what the claim attributes to it.
In the citing paper
“To holistically assess the capabilities of LLMs, several comprehensive benchmarks have been developed [325-327].”
From A Survey of Large Language Models (2026) · cites this paper as [326]
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to prior benchmarks (e.g., SuperGLUE, 9 EleutherAI LM Harness, 10 and BIG-Bench 11 ), which enumerate a set of scenarios and metrics, we situate our scenarios in a larger taxonomy to make explicit what is currently missing.”
From Holistic Evaluation of Language Models (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“BIG-Bench Hard (BBH; Suzgun et al., 2022) is a set of 23 multiple-choice tasks from the BIG-Bench benchmark (Srivastava et al., 2022), on which current state-of-the-art models do substantially worse than the average human rater.”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“BIG-bench [182] introduces a diverse collection of 204 challenging tasks contributed by 450 authors from 132 institutions.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [182]
The cited source says what the claim attributes to it.
In the citing paper
“Popular tests, such as MMLU (137), BIG-Bench (138), BIG-Bench Hard (BBH) (139), or AGIEval (140) are typically based on human ability tasks (137) but test for knowledge recall or problem-solving (141), often by comparing performance against human experts doing the same tasks (138).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [138]
The cited source says what the claim attributes to it.
In the citing paper
“We show that when we use a format with visible lettered answer options, large models are very well calibrated on diverse multiple choice questions (e.g. from BIG Bench [Srivastava et al., 2022], MMLU[Hendrycks et al., 2021], and many other evaluations); see Figures 4, 5, and 6.”
From Language Models (Mostly) Know What They Know (2022) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Although biases already pose a challenge for conventional AI in health, they are of particular relevance for GMAI as a recent large-scale evaluation showed that social bias can increase with model scale 46 .”
From Foundation models for generalist medical artificial intelligence (2023) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Our experiments validate the effectiveness of this approach on a range of complex reasoning tasks, including several challenging Big-Bench tasks (Srivastava et al., 2022).”
From Large language models as tool makers (2023) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“We note that our evaluation is in no means comprehensive and a larger benchmarking akin to Srivastava et al. (2022) would be helpful.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Srivastava et al. (2022)
The cited source says what the claim attributes to it.
In the citing paper
“Large language models have demonstrated remarkable reasoning ability on a variety of reasoning tasks26–29.”
From Solving olympiad geometry without human demonstrations (2024) · cites this paper as [28]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Furthermore, using PC measures accurately captures the scaling trends in both setups, even when using training FLOPs leads to less tight captures in the”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 140
- Scott Alexander. A very unlikely chess game, 2020. URL https://slatestarcodex.com/2020/01/06/a-very-unlikely-chessgame/ . (cited on pp. 29 and 30)
- Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 245-256. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/alon20a.html . (cited on p. 37)
- Miriam Amin and Manuel Burghardt. A survey on approaches to computational humor generation. In Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature , pp. 29-41, Online, December 2020. International Committee on Computational Linguistics. URL https://aclanthology. org/2020.latechclfl-1.4 . (cited on p. 38)
- Brandon Amos and J. Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks, 2017. URL https: //arxiv.org/abs/1703.00443 . (cited on p. 38)
- Salvatore Attardo. Humor in language. In Oxford Research Encyclopedia of Linguistics . Oxford University Press, 2017. doi: 10. 1093/acrefore/9780199384655.013.342. URL https://oxfordre.com/linguistics/view/10.1093/acrefore/9780199384655. 001.0001/acrefore-9780199384655-e-342 . (cited on p. 38) · DOI
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: //arxiv.org/abs/2108.07732 . (cited on p. 4)
- Satanjeev Banerjee and Ted Pedersen. Extended gloss overlaps as a measure of semantic relatedness. In IJCAI'03: Proceedings of the 18th International Joint Conference on Artificial Intelligence , pp. 805-810, San Francisco, 2003. Morgan Kaufmann. doi: 10.5555/1630659.1630775. URL https://dl.acm.org/doi/10.5555/1630659.1630775 . (cited on p. 36) · DOI
- Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review , 104(3):671-732, 2016. URL http://www.jstor.org/stable/24758720 . (cited on p. 31)
- Gregor Betz, Christian Voigt, and Kyle Richardson. Critical thinking for language models, 2020. URL https://arxiv.org/abs/ 2009.07185 . (cited on p. 33)
- Stella Biderman and Edward Raff. Neural language models are effective plagiarists, 2022. URL https://arxiv.org/abs/2201. 07406 . (cited on p. 4)
- Julia Birke and Anoop Sarkar. A clustering approach for nearly unsupervised recognition of nonliteral language. In 11th Conference of the European Chapter of the Association for Computational Linguistics , pp. 329-336, Trento, Italy, April 2006. Association for Computational Linguistics. URL https://aclanthology.org/E06-1042 . (cited on p. 35)
- Nicholas Boillot. Vector forms as a foreign language, 24 June 2019. URL https://www.fluate.net/en/travaux/vectoglyph . (cited on pp. 31 and 33)
- Paul F. Boller, Jr. and John George. They Never Said It: A Book of Fake Quotes, Misquotes, and Misleading Attributions . Oxford University Press, Oxford, 1989. (cited on p. 35)
- Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings. neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html . (cited on pp. 17, 29, 31, and 33)
- Bozhidar Bozhanov and Ivan Derzhanski. Rosetta stone linguistic problems. In Proceedings of the Fourth Workshop on Teaching NLP and CL , pp. 1-8, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https: //aclanthology.org/W13-3401 . (cited on p. 34)
- Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable Forth interpreter. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 547-556, 2017. URL https: //arxiv.org/abs/1605.06640 . (cited on p. 34)
- Gwern Branwen. GPT-3 creative fiction. Gwern.net , June 2020. URL https://www.gwern.net/GPT-3 . (cited on pp. 19, 38, and 39)
- Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review , 78(1):1-3, 1950. (cited on pp. 8 and 11)
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html . (cited on pp. 4, 7, 8, 10, 11, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40, and 41)
- Corrado Böhm. On a family of Turing machines and the related programming language. ICC Bulletin , 3:187-194, 1964. (cited on p. 38)
- Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pp. 2633-2650. USENIX Association, August 2021. URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting . (cited on p. 19)
- Nathanael Chambers. Labeling documents with timestamps: Learning from their time expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 98-106, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1011 . (cited on p. 31)
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 1691-1703. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/chen20s.html . (cited on p. 29)
- Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. https://openreview.net/pdf?id= H1gfOiAqYm , 2019b. (cited on p. 34)
- Kate Crawford. The trouble with bias. https://www.youtube.com/watch?v=fMym\_BKWQzk , 2017. Keynote address, NIPS 2017, Long Beach CA. Dec. 5, 2017. (cited on p. 29)
- Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 933-941, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305478. URL https://dl.acm.org/doi/10.5555/3305381. 3305478 . (cited on p. 10) · DOI
- Wayne Davis. Implicature. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Fall 2019 edition, 2019. (cited on p. 33)
- Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. Finding contradictions in text. In Proceedings of ACL-08: HLT , pp. 1039-1047, Columbus, Ohio, June 2008. Association for Computational Linguistics. URL https: //aclanthology.org/P08-1118 . (cited on p. 32)
- Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. RobustFill: Neural program learning under noisy I/O. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 990-998, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305484. URL https://dl.acm.org/doi/10.5555/3305381.3305484 . (cited on p. 34) · DOI
- Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. Beyond English-centric multilingual machine translation. Journal of Machine Learning Research , 22(107): 1-48, 2021. URL http://jmlr.org/papers/v22/20-1307.html . (cited on p. 20)
- Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database . MIT Press, Cambridge, MA, 1998. (cited on p. 33)
- Jerry A. Fodor. The Language of Thought . Harvard University Press, Cambridge, MA, 1975. (cited on p. 30)
- Evgeniy Gabrilovich and Shaul Markovitch. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In IJCAI'07: Proceedings of the 20th International Joint Conference on Artifical Intelligence , pp. 1606-1611, San Francisco, 2007. Morgan Kaufmann. doi: 10.5555/1625275.1625535. URL https://dl.acm.org/doi/10.5555/1625275.1625535 . (cited on p. 36) · DOI
- Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics , 9: 346-361, 04 2021. doi: 10.1162/tacl\_a\_00370. URL https://doi.org/10.1162/tacl\_a\_00370 . (cited on pp. 29 and 39) · DOI
- Arthur S. Goldberger. Structural equation methods in the social sciences. Econometrica , 40(6):979-1001, 1972. URL http://www.jstor.org/stable/1913851 . (cited on p. 32)
- Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pp. 581-586, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https://aclanthology.org/P11-2102 . (cited on p. 39)
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pp. 1321-1330. PMLR, 06-11 Aug. 2017. URL https://proceedings.mlr.press/v70/guo17a. html . (cited on p. 11)
- Samuel Gyasi Obeng. The proverb as a mitigating and politeness strategy in Akan discourse. Anthropological Linguistics , 38(3): 521-549, 1996. URL http://www.jstor.org/stable/30028601 . (cited on p. 32)
- Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171, Dec. 2020. doi: 10.1162/tacl\_a\_00306. URL https://doi.org/10.1162/tacl\_a\_00306 . (cited on p. 31) · DOI
- Joseph Y. Halpern. Actual causality . MIT Press, Cambridge, MA, 2016. (cited on p. 29)
- Maria Hanzén. When in Rome, do as the Romans do: Proverbs as a part of EFL teaching. Master's thesis, Jönköping University, School of Education and Communication, Jönköping, 2007. URL http://www.diva-portal.org/smash/get/diva2: 3499/fulltext01.pdf . (cited on p. 40)
- Irene Heim. On the projection problem for presuppositions. In Paul Portner and Barbara H. Partee (eds.), Formal Semantics The Essential Readings , pp. 249-260. Blackwell, Oxford, 1983. (cited on p. 37)
- Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Hkg4TI9xl . (cited on pp. 11 and 12)
- Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 2712-2721. PMLR, 09-15 June 2019. URL https: //proceedings.mlr.press/v97/hendrycks19a.html . (cited on p. 11)
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS, 2021a. URL https: //arxiv.org/abs/2105.09938 . (cited on pp. 4, 37, and 41)
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021b. URL https: //openreview.net/forum?id=d7KBjmI3GmQ . (cited on pp. 25, 31, 35, 38, and 40)
- Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips. cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html . (cited on pp. 5 and 32)
- Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URL https: //arxiv.org/abs/2102.01293 . (cited on p. 41)
- Keith J. Holyoak. Analogy and relational reasoning. In Keith J. Holyoak and Robert G. Morrison (eds.), The Oxford Handbook of Thinking and Reasoning . Oxford University Press, Oxford, 2012. URL https://www.oxfordhandbooks.com/view/10.1093/ oxfordhb/9780199734689.001.0001/oxfordhb-9780199734689-e-13 . (cited on p. 28)
- Yufang Hou. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 1428-1438, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.132. URL https://aclanthology.org/2020.acl-main.132 . (cited on p. 29)
- Yufang Hou, Katja Markert, and Michael Strube. Global inference for bridging anaphora resolution. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 907-917, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology.org/N131111 . (cited on p. 29)
- Hrisztalina Hrisztova-Gotthardt and Melita Aleksa Varga. Introduction to Paremiology: A Comprehensive Guide to Proverb Studies . De Gruyter Open, Warsaw, 2015. URL https://www.degruyter.com/document/doi/10.2478/9783110410167/html . (cited on p. 32) · DOI
- Thad Hughes and Daniel Ramage. Lexical semantic relatedness with random graph walks. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , pp. 581-589, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/D07-1061 . (cited on p. 36)
- Jay J. Jiang and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th Research on Computational Linguistics International Conference , pp. 19-33, Taipei, Taiwan, August 1997. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). URL https://aclanthology.org/O97-1002 . (cited on p. 36)
- Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438, 2020. doi: 10.1162/tacl\_a\_00324. URL https://doi.org/10. 1162/tacl\_a\_00324 . (cited on pp. 11 and 12) · DOI
- Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS'15 , volume 1, pp. 190-198, Cambridge, MA, USA, 2015. MIT Press. doi: 10.5555/2969239.2969261. URL https://dl.acm.org/doi/10.5555/2969239.2969261 . (cited on p. 34) · DOI
- Lauri Karttunen. Simple and phrasal implicatives. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pp. 124-131, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1020 . (cited on p. 32)
- Faisal Khan, Bilge Mutlu, and Jerry Zhu. How do humans teach: On curriculum learning and teaching dimension. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper/2011/file/ f9028faec74be6ec9b852b0a542e2f39-Paper.pdf . (cited on p. 38)
- Milton King and Paul Cook. Evaluating approaches to personalizing language models. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 2461-2469, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.299 . (cited on p. 38)
- Christo Kirov and Ryan Cotterell. Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate. Transactions of the Association for Computational Linguistics , 6:651-665, 12 2018. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00247. URL https://doi.org/10.1162/tacl\_a\_00247 . (cited on p. 34) · DOI
- Joshua Knobe. Intentional action and side effects in ordinary language. Analysis , 63, 07 2003. doi: 10.1111/ 1467-8284.00419. URL https://www.researchgate.net/publication/28763794\_Intentional\_Action\_and\_Side\_Effects\_ in\_Ordinary\_Language . (cited on p. 29)
- Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics , 6:317-328, 2018. doi: 10.1162/tacl\_a\_00023. URL https://aclanthology.org/Q18-1023 . (cited on p. 35) · DOI
- Alexander W. Kocurek, Ethan Jerzak, and Rachel Etta Rudolph. Against conventional wisdom. Philosophers' Imprint , 20(22): 1-27, 2020. URL http://hdl.handle.net/2027/spo.3521354.0020.022 . (cited on p. 29)
- Jarmo Korhonen. Sprichwörter und zweisprachige lexikographie: Deutsch-schwedische und deutsch-finnische wörtebücher im vergleich. In C. Földes (ed.), Phraseologie disziplinär und interdisziplinär , pp. 537-549. Gunter Narr Verlag, 2009. (cited on p. 40)
- Mahnaz Koupaee and William Yang Wang. WikiHow: A large scale text summarization dataset, 2018. URL https://arxiv. org/abs/1810.09305 . (cited on p. 5)
- Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics , 10:50-72, 01 2022. doi: 10.1162/tacl\_a\_00447. URL https://doi.org/10.1162/tacl\_a\_00447 . (cited on p. 20) · DOI
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 08 2019. doi: 10.1162/tacl\_a\_00276. URL https://doi.org/10.1162/tacl\_a\_00276 . (cited on pp. 5 and 37) · DOI
- Kevin Lacker. Giving GPT-3 a Turing test. Kevin Lacker's blog , July 2020. URL https://lacker.io/ai/2020/07/06/givinggpt-3-a-turing-test.html . (cited on pp. 33 and 41)
- George Lakoff and Mark Johnson. Metaphors We Live By . University of Chicago Press, Chicago, 2008. (cited on pp. 33 and 35)
- Guillaume Lample and François Charton. Deep learning for symbolic mathematics, 2019. URL https://arxiv.org/abs/1912. 01412 . (cited on p. 38)
- Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 986-995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL https://aclanthology.org/I17-1099 . (cited on p. 40)
- Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics , 4:521-535, 12 2016. doi: 10.1162/tacl\_a\_00115. URL https://doi.org/10.1162/tacl\_a\_00115 . (cited on p. 39) · DOI
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019. URL https://arxiv.org/abs/1907. 11692 . (cited on pp. 39 and 41)
- Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. In Vivek Nigam, Tajana Ban Kirigin, Carolyn Talcott, Joshua Guttman, Stepan Kuznetsov, Boon Thau Loo, and Mitsuhiro Okada (eds.), Logic, Language, and Security . Springer, Cham, 2020. URL https://www.springerprofessional. de/en/gender-bias-in-neural-natural-language-processing/18531692 . (cited on p. 33)
- Jihang Mao and Wanli Liu. A BERT-based approach for automatic humor detection and scoring. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019) , pp. 197-202, 2019. URL http://ceur-ws.org/Vol-2421/HAHA\_paper\_8.pdf . (cited on p. 39)
- Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence, 2020. URL https://arxiv.org/abs/ 2002.06177 . (cited on p. 30)
- Katja Markert, Yufang Hou, and Michael Strube. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 795-804, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1084 . (cited on p. 29)
- R. Thomas McCoy, Robert Frank, and Tal Linzen. Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks. Transactions of the Association for Computational Linguistics , 8:125-140, 01 2020. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00304. URL https://doi.org/10.1162/tacl\_a\_00304 . (cited on p. 34) · DOI
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https: //arxiv.org/abs/1609.07843 . (cited on p. 31)
- John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 6905-6916. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/miller20a.html . (cited on p. 39)
- David Milne and Ian H. Witten. An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: An Evolving Synergy , pp. 25-30, Menlo Park, 2008. Association for the Advancement of Artificial Intelligence. URL https://www.aaai.org/Papers/Workshops/2008/WS08-15/WS08-15-005.pdf . (cited on p. 36)
- Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Natural reference to objects in a visual domain. In Proceedings of the 6th International Natural Language Generation Conference . Association for Computational Linguistics, July 2010. URL https://aclanthology.org/W10-4210 . (cited on p. 38)
- Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Generating expressions that refer to visible objects. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 1174-1184, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https:// aclanthology.org/N13-1137 . (cited on p. 38)
- Elham Mohammadi, Hessam Amini, and Leila Kosseim. CLaC at CLPsych 2019: Fusion of neural features and predicted class probabilities for suicide risk assessment based on online posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology , pp. 34-38, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-3004. URL https://aclanthology.org/W19-3004 . (cited on p. 39) · DOI
- Jane Morris and Graeme Hirst. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics , 17(1):21-48, 1991. URL https://aclanthology.org/J91-1002 . (cited on p. 36)
- Karl Mulligan, Robert Frank, and Tal Linzen. Structure here, bias there: Hierarchical generalization by jointly learning syntactic transformations. In Proceedings of the Society for Computation in Linguistics 2021 , pp. 125-135, Online, February 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.scil-1.12 . (cited on p. 34)
- Gregory L. Murphy. Comprehending complex concepts. Cognitive Science , 12(4):529-562, 1988. doi: https://doi.org/10.1207/ s15516709cog1204\_2. URL https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1204\_2 . (cited on p. 30) · DOI
- Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 8558cb408c1d76621371888657d2eb1d-Abstract.html . (cited on p. 11)
- Anthony M. Paul. Figurative language. Philosophy & Rhetoric , 3(4):225-248, 1970. URL http://www.jstor.org/stable/ 40237206 . (cited on p. 35)
- Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann, San Francisco, 1988. (cited on p. 32)
- Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, Cambridge, 2000. (cited on p. 30)
- Tony A. Plate. Distributed representations and nested compositional structure . PhD thesis, University of Toronto, Toronto, 1994. (cited on p. 29)
- Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving, 2020. URL https://arxiv. org/abs/2009.03393 . (cited on pp. 34 and 38)
- Willard V.O. Quine. Main trends in recent philosophy: Two dogmas of empiricism. The Philosophical Review , 60(1):20-43, 1951. URL http://www.jstor.org/stable/2181906 . (cited on p. 29)
- Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/languagemodels.pdf . (cited on pp. 10, 30, 33, 35, 37, 38, and 39)
- Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pp. 777-789, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology. org/D12-1071 . (cited on p. 41)
- Ian Ravenscroft. Folk psychology as a theory. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Summer 2019 edition, 2019. (cited on p. 32)
- Joseph Reisinger and Raymond J. Mooney. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 109-117, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology. org/N10-1013 . (cited on p. 36)
- Philip Resnik. Using information content to evaluate semantic similarity in a taxonomy. In IJCAI'95: Proceedings of the 14th International Joint Conference on Artificial Intelligence, Volume 1 , pp. 448-453, San Francisco, 1995. Morgan Kaufmann. doi: 10.5555/1625855.1625914. URL https://dl.acm.org/doi/10.5555/1625855.1625914 . (cited on p. 36) · DOI
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. AAAI Spring Symposium , 2011. URL http://commonsensereasoning.org/2011/papers/Roemmele.pdf . (cited on p. 32)
- Joshua S. Rule. The child as hacker: Building more human-like models of learning . PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2020. URL https://hdl.handle.net/1721.1/129232 . (cited on p. 34)
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 . (cited on pp. 9 and 25)
- Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 624-643, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.52. URL https://aclanthology.org/2021.naacl-main.52 . (cited on p. 41) · DOI
- Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021b. URL https://openreview.net/ forum?id=fe\_hCc4RBrg . (cited on p. 4)
- Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. Diagram understanding in geometry questions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, Menlo Park, CA, Jun. 2014. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9146 . (cited on p. 34)
- Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL http: //arxiv.org/abs/1804.04235 . (cited on p. 43)
- Ekaterina Shutova. Automatic metaphor interpretation as a paraphrasing task. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 1029-1037, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology.org/N10-1147 . (cited on p. 35)
- Ekaterina Shutova and Simone Teufel. Metaphor corpus annotated for source-target domain mappings. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) , Valletta, Malta, May 2010. European Language Resources Association. URL http://www.lrec-conf.org/proceedings/lrec2010/pdf/612\_Paper.pdf . (cited on p. 35)
- Damien Sileo, Tim Van de Cruys, Camille Pradel, and Philippe Muller. DiscSense: Automated semantic analysis of discourse markers. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 991-999, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.125 . (cited on p. 31)
- Gurdeep Singh, Kaustubh Dhole, Priyadarshini P. Pai, and Sukanta Mondal. SPRINGS: Prediction of protein-protein interaction sites using artificial neural networks. Journal of Proteomics & Computational Biology , 1:7, 2014. URL https: //www.avensonline.org/fulltextarticles/JPCB-2572-8679-01-0001.html . (cited on p. 37)
- Douglas R. Smith. The synthesis of LISP programs from examples: A survey. In Alan W. Biermann, Gerhard Guiho, and Yves Kodratoff (eds.), Automatic Program Construction Techniques , pp. 307-324. Macmillan, New York, 1984. (cited on p. 34)
- Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences , 11(1):1-23, 1988. doi: 10.1017/ S0140525X00052432. (cited on p. 30)
- Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://aclanthology.org/D13-1170 . (cited on p. 31)
- Shashank Srivastava, Snigdha Chaturvedi, and Tom Mitchell. Inferring interpersonal relations in narrative summaries. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16 , pp. 2807-2813, Menlo Park, CA, 2016. Association for the Advancement of Artificial Intelligence. doi: 10.5555/3016100.3016294. URL https://dl.acm.org/doi/10. 5555/3016100.3016294 . (cited on p. 34) · DOI
- Robert Stalnaker. Assertion. In P. Cole (ed.), Pragmatics , Syntax and Semantics 9, pp. 315-332. Brill, Leiden, 1978. (cited on p. 32)
- Bernd Steinbach and Roman Kohut. Neural networks - a model of boolean functions. 5th International Workshop on Boolean Problems, Freiburg, Sept. 2002. , 2002. URL https://www.researchgate.net/publication/246931125\_Neural\_Networks\_-\_A\_Model\_of\_Boolean\_Functions . (cited on p. 29)
- Andreas Stöckl. Watching a language model learning chess. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) , pp. 1369-1379, Held Online, September 2021. INCOMA Ltd. URL https://aclanthology.org/2021.ranlp-1.153 . (cited on p. 4)
- Kevin Stowe, Leonardo Ribeiro, and Iryna Gurevych. Metaphoric paraphrase generation, 2020. URL https://arxiv.org/abs/ 2002.12854 . (cited on p. 35)
- Michael Strube and Simone Paolo Ponzetto. Wikirelate! Computing semantic relatedness using Wikipedia. In AAAI'06: Proceedings of the 21st National Conference on Artificial Intelligence , volume 2, pp. 1419-1424. Association for the Advancement of Artificial Intelligence, 2006. doi: 10.5555/1597348.1597414. URL https://dl.acm.org/doi/10.5555/1597348. 1597414 . (cited on p. 36) · DOI
- Saku Sugawara, Hikaru Yokono, and Akiko Aizawa. Prerequisite skills for reading comprehension: Multi-perspective analysis of MCTest datasets and systems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, Menlo Park, CA, Feb. 2017. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/ view/10957 . (cited on p. 35)
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks, 2014. URL https: //arxiv.org/abs/1409.3215 . (cited on p. 4)
- Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Tunuguntla Deepak, and Oskar van der Wal. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models , 2022. URL https://openreview.net/forum?id=rK-7NhfSIW5 . (cited on pp. 18 and 20)
- Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. Learning to recommend quotes for writing. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15 , pp. 2453-2459, Menlo Park, CA, 2015. Association for the Advancement of Artificial Intelligence. URL https://dl.acm.org/doi/10.5555/2886521.2886662 . (cited on p. 28) · DOI
- Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15 , pp. 1923-1929, 2015. URL https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/view/10957/10931 . (cited on p. 38)
- George Tsatsaronis, Iraklis Varlamis, Michalis Vazirgiannis, and Kjetil Nørvåg. Omiotis: A thesaurus-based measure of text relatedness. In Wray Buntine, Marko Grobelnik, Dunja Mladenić, and John Shawe-Taylor (eds.), Machine Learning and Knowledge Discovery in Databases , pp. 742-745, Berlin, 2009. Springer. (cited on p. 36)
- Shikhar Vashishth, Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Dating documents using graph convolution networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1605-1615, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ v1/P18-1149. URL https://aclanthology.org/P18-1149 . (cited on p. 31)
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 30, pp. 5998--6008. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf . (cited on pp. 10 and 38)
- Jette Viethen and Robert Dale. The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference , pp. 59-67, Salt Fork, Ohio, USA, June 2008. Association for Computational Linguistics. URL https://aclanthology.org/W08-1109 . (cited on p. 38)
- Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers , pp. 176-187, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-1017 . (cited on p. 35)
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019a. URL https://proceedings.neurips.cc/paper/2019/hash/ 4496bf24afe7fab6f046bf4923da8de6-Abstract.html . (cited on pp. 5, 7, 29, 32, and 41)
- David Wechsler. Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) . Pearson, San Antonio, 2008. (cited on p. 38)
- Ludwig Wittgenstein. Philosophical investigations . Basil Blackwell, Oxford, 1953. (cited on p. 29)
- Shijie Wu, Ryan Cotterell, and Mans Hulden. Applying the transformer to character-level transduction, 2020. URL https: //arxiv.org/abs/2005.10213 . (cited on p. 41)
- Scott Cheng-Hsin Yang and Patrick Shafto. Explainable artificial intelligence via Bayesian teaching. Workshop on Teaching Machines, Robots, and Humans, NIPS 2017 , 2017. URL http://shaftolab.com/assets/papers/yangShafto\_NIPS\_2017\_ machine\_teaching.pdf . (cited on p. 38)
- Eric Yeh, Daniel Ramage, Christopher D. Manning, Eneko Agirre, and Aitor Soroa. WikiWalk: Random walks on Wikipedia for semantic relatedness. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4) , pp. 41-49, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https: //aclanthology.org/W09-3206 . (cited on p. 36)
- Wenpeng Yin and Yulong Pei. Optimizing sentence modeling and selection for document summarization. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI-15 , pp. 1383-1389, 2015. doi: 10.5555/2832415.2832442. URL https://dl.acm.org/doi/10.5555/2832415.2832442 . (cited on p. 32) · DOI
- Eliezer Yudkowsky. Artificial intelligence as a positive and negative factor in global risk. In Nick Bostrom and Milan M. Ćirković (eds.), Global Catastrophic Risks , pp. 308-345. Oxford University Press, Oxford, 2008. URL https://web.archive.org/web/ 20210125025955/https://intelligence.org/files/AIPosNegFactor.pdf . (cited on p. 38)
- Omnia Zayed, John Philip McCrae, and Paul Buitelaar. Figure me out: A gold standard dataset for metaphor interpretation. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 5810-5819, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.712 . (cited on p. 35)
- Meishan Zhang, Yue Zhang, and Guohong Fu. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pp. 2449-2460, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https://aclanthology.org/C16-1231 . (cited on p. 39)
- Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, Menlo Park, CA, Mar. 2015. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9761 . (cited on p. 38)
Cited by
Sorted by citations
Items 101–200 of 681
Understanding Emergent Abilities of Language Models from the Loss Perspective
Du, Z., Zeng, A., Dong, Y., & Tang, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1683 · Cited by 76
Self-citationGenerating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2023) · arXiv · doi:10.48550/arxiv.2309.17012 · Cited by 72
“BigBench (Srivastava et al., 2023) is a collection of benchmarks that look to probe the abilities of language models over a diverse range of tasks.”
BackgroundMethodInfluentialLarge language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71
Can we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72
A taxonomy and review of generalization research in NLP
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00729-y · Cited by 145
Self-citationMuennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candes, E., & Hashimoto, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1025 · Cited by 76
Self-citationInstruction Induction: From Few Examples to Natural Language Task Descriptions
Honovich, O., Shaham, U., Bowman, S. R., & Levy, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.108 · Cited by 160
Self-citationNicheformer: A Foundation Model for Single-Cell and Spatial OmicsPreprint
Schaar, A. C., Tejada-Lapuerta, A., Palla, G., Gutgesell, R. M., Halle, L., Minaeva, M., Vornholz, L., Dony, L., Drummer, F., Bahrami, M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.15.589472 · Cited by 66
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data
Shum, K., Diao, S., & Zhang, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.811 · Cited by 79
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2022) · arXiv · doi:10.48550/arxiv.2210.13382 · Cited by 61
Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Wang, Y., Li, P., Sun, M., & Liu, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.691 · Cited by 141
Implicit bias in large language models: Experimental proof and implications for education
Warr, M., Oster, N. J., & Isaac, R. (2024) · Journal of Research on Technology in Education · doi:10.1080/15391523.2024.2395295 · Cited by 73
Toward large reasoning models: A survey of reinforced reasoning with large language models
Xu, F., Hao, Q., Shao, C., Zong, Z., Li, Y., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101370 · Cited by 533
Self-citationOn Data Scaling in Masked Image Modeling
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Wei, Y., Dai, Q., & Hu, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00999 · Cited by 70
Natural Language Reasoning, A Survey
Yu, F., Zhang, H., Tiwari, P., & Wang, B. (2024) · ACM Computing Surveys · doi:10.1145/3664194 · Cited by 131
Self-citationFairness in Large Language Models: A Taxonomic Survey
Chu, Z., Wang, Z., & Zhang, W. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3682112.3682117 · Cited by 106
Self-citationVisual cognition in multimodal large language models
Schulze Buschoff, L. M., Akata, E., Bethge, M., & Schulz, E. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00963-y · Cited by 83
Olausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57
“These findings suggest that such models may be relying on approximate heuristics based on surface-level statistical patterns in reasoning tasks, rather than consistent, gener-alizable representations and strategies (Srivastava et al., 2023; Creswell et al., 2023).”
BackgroundSelf-citationDo Large Language Models Know What They Don’t Know?
Yin, Z., Sun, Q., Guo, Q., Wu, J., Qiu, X., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.551 · Cited by 75
The Quantization Model of Neural Scaling
Michaud, E. J., Liu, Z., Girit, U., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2303.13506 · Cited by 57
People cannot distinguish GPT-4 from a human in a Turing test
Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732108 · Cited by 76
Integrating chemistry knowledge in large language models via prompt engineering
Liu, H., Yin, H., Luo, Z., & Wang, X. (2024) · Synthetic and Systems Biotechnology · doi:10.1016/j.synbio.2024.07.004 · Cited by 63
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
“There is a need for challenging benchmarks that more accurately reflect real-world applications of LMs to help shape their future development and usage (Srivastava et al., 2023).”
BackgroundInfluentialCan ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 127
Foundation models in ophthalmology
Chia, M. A., Antaki, F., Zhou, Y., Turner, A. W., Lee, A., & Keane, P. A. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2024-325459 · Cited by 61
Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and BardPreprint
Borji, A., & Mohammadian, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4476855 · Cited by 58
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 71
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53
“Work on “scaling laws” has found trends in model behaviors across smaller models continue for larger ones (Kaplan et al., 2020; Henighan et al., 2020), but recent work has found exceptions to scaling trends (Srivastava et al., 2022; Wei et al., 2022a).”
ResultSelf-citationCan LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2023) · arXiv · doi:10.48550/arxiv.2306.13063 · Cited by 618
Testing AI on language comprehension tasks reveals insensitivity to underlying meaning
Dentella, V., Günther, F., Murphy, E., Marcus, G., & Leivada, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-79531-8 · Cited by 54
Inductive reasoning in humans and large language models
Han, S. J., Ransom, K., Perfors, A., & Kemp, C. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101155 · Cited by 82
Self-citationEdgeLLM: Fast On-Device LLM Inference With Speculative Decoding
Xu, D., Yin, W., Zhang, H., Jin, X., Zhang, Y., Wei, S., Xu, M., & Liu, X. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3513457 · Cited by 65
Self-citationFrom Deep Learning to Rational Machines
Buckner, C. J. (2023) · doi:10.1093/oso/9780197653302.001.0001 · Cited by 57
Experiential Narratives in Marketing: A Comparison of Generative AI and Human Content
Wen, Y., & Laporte, S. (2024) · Journal of Public Policy & Marketing · doi:10.1177/07439156241297973 · Cited by 62
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47
Self-citationLarge language model-based evolutionary optimizer: Reasoning with elitism
Brahmachary, S., Joshi, S. M., Panda, A., Koneripalli, K., Sagotra, A. K., Patel, H., Sharma, A., Jagtap, A. D., & Kalyanaraman, K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129272 · Cited by 66
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
Zhang, Y., Sun, R., Chen, Y., Pfister, T., Zhang, R., & Arik, S. Ö. (2024) · arXiv · doi:10.48550/arxiv.2406.02818 · Cited by 47
“Despite their impressive performance across a wide range of scenarios, LLMs struggle with tasks that involve long contexts [8, 63, 57].”
Self-citationFaithful Logical Reasoning via Symbolic Chain-of-Thought
Xu, J., Fei, H., Pan, L., Liu, Q., Lee, M.-L., & Hsu, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.720 · Cited by 53
Self-citationUniquely human intelligence arose from expanded information capacity
Cantlon, J. F., & Piantadosi, S. T. (2024) · Nature Reviews Psychology · doi:10.1038/s44159-024-00283-3 · Cited by 61
Self-citationThe Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
Self-citationThe Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges
Winata, G. I., Aji, A. F., Yong, Z., & Solorio, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.185 · Cited by 75
Self-citationFLUTE: Figurative Language Understanding through Textual Explanations
Chakrabarty, T., Saakyan, A., Ghosh, D., & Muresan, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.481 · Cited by 50
Large scale foundation models for intelligent manufacturing applications: a survey
Zhang, H., Semujju, S. D., Wang, Z., Lv, X., Xu, K., Wu, L., Jia, Y., Wu, J., Liang, W., Zhuang, R., et al. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-024-02536-7 · Cited by 61
Self-citationSymbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step
Li, L. H., Hessel, J., Yu, Y., Ren, X., Chang, K.-W., & Choi, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.150 · Cited by 148
Self-citationChakraborty, N., Ornik, M., & Driggs-Campbell, K. (2025) · ACM Computing Surveys · doi:10.1145/3716846 · Cited by 60
Hutchins, D., Schlag, I., Wu, Y., Dyer, E., & Neyshabur, B. (2022) · arXiv · doi:10.48550/arxiv.2203.07852 · Cited by 43
Self-citationReCode: Robustness Evaluation of Code Generation Models
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.773 · Cited by 42
Self-citationYampolskiy, R. V. (2024) · AI and Ethics · doi:10.1007/s43681-024-00420-x · Cited by 64
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S., Lee, S., & Kim, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.155 · Cited by 91
Self-citationShi, H. H., Fang, L., Chen, X., Gu, C., Ma, K., Zhang, X., Zhang, Z., Gu, J., & Lim, E. G. (2024) · IET Smart Grid · doi:10.1049/stg2.12191 · Cited by 52
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51
Peng, Y., Han, J., Zhang, Z., Fan, L., Liu, T., Qi, S., Feng, X., Ma, Y., Wang, Y., & Zhu, S. C. (2023) · Engineering · doi:10.1016/j.eng.2023.07.006 · Cited by 46
Probing the limitations of multimodal language models for chemistry and materials research
Alampara, N., Schilling-Wilhelmi, M., Ríos-García, M., Mandal, I., Khetarpal, P., Grover, H. S., Krishnan, N. M. A., & Jablonka, K. M. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00836-3 · Cited by 90
A taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41
Self-citationWhen Language Model Meets Private Library
Zan, D., Chen, B., Lin, Z., Guan, B., Yongji, W., & Lou, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.21 · Cited by 72
Autonomous medical evaluation for guideline adherence of large language models
Fast, D., Adams, L. C., Busch, F., Fallon, C., Huppertz, M., Siepmann, R. M., Prucker, P., Bayerl, N., Truhn, D., Makowski, M. R., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01356-6 · Cited by 51
MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
Chen, J., Prasad, A., Saha, S., Stengel-Eskin, E., & Bansal, M. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1660 · Cited by 38
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research
Sun, L., Yang, H., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., & Yu, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29872 · Cited by 100
Self-citationStudying and improving reasoning in humans and machines
Yax, N., Anlló, H., & Palminteri, S. (2024) · Communications Psychology · doi:10.1038/s44271-024-00091-8 · Cited by 40
Lion: Adversarial Distillation of Proprietary Large Language Models
Jiang, Y., Chan, C., Chen, M., & Wang, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.189 · Cited by 93
Higher education assessment practice in the era of generative AI tools
(2024) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2024.7.1.28 · Cited by 37
Do Language Models Know When They’re Hallucinating References?
Agrawal, A., Suzgun, M., Mackey, L., & Kalai, A. (2024) · Findings of the Association for Computational Linguistics: EACL 2024 · doi:10.18653/v1/2024.findings-eacl.62 · Cited by 37
Self-citationYang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 81
Self-citationJacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.308 · Cited by 36
Wickramasekara, A., Breitinger, F., & Scanlon, M. (2025) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2024.301859 · Cited by 58
Pan, J., Chou, G., & Berenson, D. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161125 · Cited by 54
Automatic Generation of Socratic Subquestions for Teaching Math Word Problems
Shridhar, K., Mačina, J., El‐Assady, M., Sinha, T., Kapur, M., & Sachan, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.277 · Cited by 62
Self-citationJacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36
INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
Chia, Y. K., Hong, P., Bing, L., & Poria, S. (2023) · arXiv · doi:10.48550/arxiv.2306.04757 · Cited by 36
“Complex Instructions BIG-Bench Hard (BBH) is a subset of 23 challenging tasks from the BIG-Bench benchmark (Srivastava et al., 2022), which focuses on tasks believed to be beyond the capabilities of current language models (Suzgun et al., 2022).”
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
Shapira, N., Levy, M., Alavi, S. H., Zhou, X., Choi, Y., Goldberg, Y., Sap, M., & Shwartz, V. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.138 · Cited by 44
Self-citationPei, Z. (2024) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2024.102334 · Cited by 37
Enhancing Accuracy in Large Language Models Through Dynamic Real-Time Information InjectionPreprint
Ouyang, Q., Wang, S., & Wang, B. (2023) · Preprints.org · doi:10.20944/preprints202312.1987.v1 · Cited by 36
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06559 · Cited by 35
POSIX: A Prompt Sensitivity Index For Large Language Models
Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., & Chakraborty, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.852 · Cited by 34
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
Towards the ultimate brain: Exploring scientific discovery with ChatGPT AI
Adesso, G. (2023) · AI Magazine · doi:10.1002/aaai.12113 · Cited by 48
Loya, M., Sinha, D., & Futrell, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.241 · Cited by 33
Lampinen, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00525 · Cited by 32
Self-citationScaling Data-Constrained Language Models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2305.16264 · Cited by 49
Self-citationCalibration and Correctness of Language Models for Code
Spieß, C. K., Gros, D., Pai, K. S., Pradel, M., Rabin, M. R. I., Alipour, M. A., Jha, S., Dévanbu, P., & Ahmed, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00040 · Cited by 102
Warr, M., & Heath, M. K. (2025) · Journal of Teacher Education · doi:10.1177/00224871251325073 · Cited by 35
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. J. L., Reinecke, K., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.505 · Cited by 74
Self-citationZhou, D., Wang, K., Gu, J., Peng, X., Lian, D., Zhang, Y., You, Y., & Feng, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01578 · Cited by 35
Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors
Ichien, N., Stamenković, D., & Holyoak, K. J. (2024) · Metaphor and Symbol · doi:10.1080/10926488.2024.2380348 · Cited by 61
Prompting Techniques for Secure Code Generation: A Systematic Investigation
Tony, C., Ferreyra, N. E. D., Mutas, M., Dhiff, S., & Scandariato, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3722108 · Cited by 85
Aydın, Ö., Karaarslan, E., Erenay, F. S., & Bačanin, N. (2026) · Turkish Journal of Engineering · doi:10.31127/tuje.1838765 · Cited by 69
Arakawa, R., Yakura, H., & Goto, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581133 · Cited by 29
Self-citationKoLA: Carefully Benchmarking World Knowledge of Large Language Models
Yu, J., Wang, X., Tu, S., Cao, S., Zhang-Li, D., Lv, X., Peng, H., Yao, Z., Zhang, X., Li, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.09296 · Cited by 29
“To this end, various benchmarks are proposed, focusing on extending the evaluation scope to cover broader abilities (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023) or more challenging tasks (Srivastava et al., 2022; Suzgun et al., 2022).”
BackgroundSelf-citationLarge language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28
Symbol tuning improves in-context learning in language models
Wei, J., Hou, L., Lampinen, A. K., Chen, X., Huang, D. Y., Tay, Y., Chen, X., Lu, Y., Zhou, D., Ma, T., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.61 · Cited by 80
Self-citationXiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y., Jiang, S., Xiong, Z., Li, Z., Wu, W., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29767 · Cited by 28
Self-citationThe Platonic Representation Hypothesis
Huh, M., Cheung, B., Wang, T., & Isola, P. (2024) · arXiv · doi:10.48550/arxiv.2405.07987 · Cited by 35
“For example, whereas in the past we had special-purpose solutions for different language processing tasks ( e.g. , sentiment analysis, parsing, dialogue), modern large language models (LLMs) are competent at all these tasks using a single set of weights (Srivastava et al., 2022).”
BackgroundSelf-citationTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483
Self-citationSeedLLM·Rice: A large language model integrated with rice biological knowledge graph
Yang, F., Kong, H., Ying, J., Chen, Z., Luo, T., Jiang, W., Yuan, Z., Wang, Z., Ma, Z., Wang, S., et al. (2025) · Molecular Plant · doi:10.1016/j.molp.2025.05.013 · Cited by 29
Self-citationTowards Evaluation and Understanding of Large Language Models for Cyber Operation Automation
Sultana, M., Taylor, A., Li, L., & Majumdar, S. (2023) · IEEE Conference on Communications and Network Security · doi:10.1109/cns59707.2023.10288677 · Cited by 29
Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities
Hu, H., Luan, Y., Chen, Y., Khandelwal, U., Joshi, M., Lee, K., Toutanova, K., & Chang, M.-W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01108 · Cited by 46
Self-citationSelf-Evaluation Guided Beam Search for Reasoning
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J. X., Kan, M.-Y., He, J., & Xie, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1802 · Cited by 27
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H. W., Tran, V., So, D. R., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 95
Self-citation
Metadata
- Authors
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A., La, A., Lampinen, A., Zou, A., Jiang, A., Chen, A., Vuong, A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabassum, A., Menezes, A., Kirubarajan, A., Mullokandov, A., Sabharwal, A., Herrick, A., Efrat, A., Erdem, A., Karakaş, A., Roberts, B. R., Loe, B. S., Zoph, B., Bojanowski, B., Özyurt, B., Hedayatnia, B., Neyshabur, B., Inden, B., Stein, B., Ekmekci, B., Lin, B. Y., Howald, B., Orinion, B., Diao, C., Dour, C., Stinson, C., Argueta, C., Ramírez, C. F., Singh, C., Rathkopf, C., Meng, C., Baral, C., Wu, C., Callison-Burch, C., Waites, C., Voigt, C., Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C., Ashcraft, C., Garbacea, C., Sileo, D., Garrette, D., Hendrycks, D., Kilman, D., Roth, D., Freeman, D., Khashabi, D., Levy, D., González, D. M., Perszyk, D., Hernandez, D., Chen, D., Ippolito, D., Gilboa, D., Dohan, D., Drakard, D., Jurgens, D., Datta, D., Ganguli, D., Emelin, D., Kleyko, D., Yuret, D., Chen, D., Tam, D., Hupkes, D., Misra, D., Buzan, D., Mollo, D. C., Yang, D., Lee, D.-H., Schrader, D., Shutova, E., Cubuk, E. D., Segal, E., Hagerman, E., Barnes, E., Donoway, E., Pavlick, E., Rodola, E., Lam, E., Chu, E., Tang, E., Erdem, E., Chang, E., Chi, E. A., Dyer, E., Jerzak, E., Kim, E., Manyasi, E. E., Zheltonozhskii, E., Xia, F., Siar, F., Martínez-Plumed, F., Happé, F., Chollet, F., Rong, F., Mishra, G., Winata, G. I., de Melo, G., Kruszewski, G., Parascandolo, G., Mariani, G., Wang, G., Jaimovitch-López, G., Betz, G., Gur-Ari, G., Galijasevic, H., Kim, H., Rashkin, H., Hajishirzi, H., Mehta, H., Bogar, H., Shevlin, H., Schütze, H., Yakura, H., Zhang, H., Wong, H. M., Ng, I., Noble, I., Jumelet, J., Geissinger, J., Kernion, J., Hilton, J., Lee, J., Fisac, J. F., Simon, J. B., Koppel, J., Zheng, J., Zou, J., Kocoń, J., Thompson, J., Wingfield, J., Kaplan, J., Radom, J., Sohl-Dickstein, J., Phang, J., Wei, J., Yosinski, J., Novikova, J., Bosscher, J., Marsh, J., Kim, J., Taal, J., Engel, J., Alabi, J., Xu, J., Song, J., Tang, J., Waweru, J., Burden, J., Miller, J., Balis, J. U., Batchelder, J., Berant, J., Frohberg, J., Rozen, J., Hernandez-Orallo, J., Boudeman, J., Guerr, J., Jones, J., Tenenbaum, J. B., Rule, J. S., Chua, J., Kanclerz, K., Livescu, K., Krauth, K., Gopalakrishnan, K., Ignatyeva, K., Markert, K., Dhole, K. D., Gimpel, K., Omondi, K., Mathewson, K., Chiafullo, K., Shkaruta, K., Shridhar, K., McDonell, K., Richardson, K., Reynolds, L., Gao, L., Zhang, L., Dugan, L., Qin, L., Contreras-Ochando, L., Morency, L.-P., Moschella, L., Lam, L., Noble, L., Schmidt, L., He, L., Colón, L. O., Metz, L., Şenel, L. K., Bosma, M., Sap, M., ter Hoeve, M., Farooqi, M., Faruqui, M., Mazeika, M., Baturan, M., Marelli, M., Maru, M., Quintana, M. J. R., Tolkiehn, M., Giulianelli, M., Lewis, M., Potthast, M., Leavitt, M. L., Hagen, M., Schubert, M., Baitemirova, M. O., Arnaud, M., McElrath, M., Yee, M. A., Cohen, M., Gu, M., Ivanitskiy, M., Starritt, M., Strube, M., Swędrowski, M., Bevilacqua, M., Yasunaga, M., Kale, M., Cain, M., Xu, M., Suzgun, M., Walker, M., Tiwari, M., Bansal, M., Aminnaseri, M., Geva, M., Gheini, M., T, M. V., Peng, N., Chi, N. A., Lee, N., Krakover, N. G.-A., Cameron, N., Roberts, N., Doiron, N., Martinez, N., Nangia, N., Deckers, N., Muennighoff, N., Keskar, N. S., Iyer, N. S., Constant, N., Fiedel, N., Wen, N., Zhang, O., Agha, O., Elbaghdadi, O., Levy, O., Evans, O., Casares, P. A. M., Doshi, P., Fung, P., Liang, P. P., Vicol, P., Alipoormolabashi, P., Liao, P., Liang, P., Chang, P., Eckersley, P., Htut, P. M., Hwang, P., Miłkowski, P., Patil, P., Pezeshkpour, P., Oli, P., Mei, Q., Lyu, Q., Chen, Q., Banjade, R., Rudolph, R. E., Gabriel, R., Habacker, R., Risco, R., Millière, R., Garg, R., Barnes, R., Saurous, R. A., Arakawa, R., Raymaekers, R., Frank, R., Sikand, R., Novak, R., Sitelew, R., LeBras, R., Liu, R., Jacobs, R., Zhang, R., Salakhutdinov, R., Chi, R., Lee, R., Stovall, R., Teehan, R., Yang, R., Singh, S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff, S., Gehrmann, S., Schuster, S., Sadeghi, S., Hamdan, S., Zhou, S., Srivastava, S., Shi, S., Singh, S., Asaadi, S., Gu, S. S., Pachchigar, S., Toshniwal, S., Upadhyay, S., Shyamolima, Debnath, Shakeri, S., Thormeyer, S., Melzi, S., Reddy, S., Makini, S. P., Lee, S.-H., Torene, S., Hatwar, S., Dehaene, S., Divic, S., Ermon, S., Biderman, S., Lin, S., Prasad, S., Piantadosi, S. T., Shieber, S. M., Misherghi, S., Kiritchenko, S., Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L., Desbordes, T., Rothschild, T., Phan, T., Wang, T., Nkinyili, T., Schick, T., Kornev, T., Tunduny, T., Gerstenberg, T., Chang, T., Neeraj, T., Khot, T., Shultz, T., Shaham, U., Misra, V., Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V., Prabhu, V. U., Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, Y., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z., Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z., & Wu, Z.
- Affiliations
- Shammie
- ORCID
- Abid, Abubakar
- Brown, Adam R.
- Gupta, Aditya
- Power, Alethea
- Warstadt, Alex
- Kocurek, Alexander W.
- Safaya, Ali
- Tazarv, Ali
- Xiang, Alice
- Parrish, Alicia
- Nie, Allen
- Dsouza, Amanda
- Rahane, Ameet
- Andreassen, Anders
- Madotto, Andrea
- Santilli, Andrea
- Dai, Andrew
- Lampinen, Andrew
- Gupta, Animesh
- Norelli, Antonio
- Venkatesh, Anu
- Mullokandov, Asher
- Erdem, Aykut
- Loe, Bao Sheng
- Bojanowski, Bartłomiej
- Inden, Benjamin
- Stein, Benno
- Howald, Blake
- Stinson, Catherine
- Singh, Chandan
- Rathkopf, Charles
- Meng, Chenlin
- Baral, Chitta
- Callison-Burch, Chris
- Voigt, Christian
- Manning, Christopher D.
- Potts, Christopher
- Siro, Clemencia
- Raffel, Colin
- Sileo, Damien
- Roth, Dan
- Khashabi, Daniel
- Chen, Danqi
- Ippolito, Daphne
- Jurgens, David
- Datta, Debajyoti
- Ganguli, Deep
- Kleyko, Denis
- Chen, Derek
- Misra, Diganta
- Mollo, Dimitri Coelho
- Yang, Diyi
- Lee, Dong-Ho
- Shutova, Ekaterina
- Cubuk, Ekin Dogus
- Segal, Elad
- Barnes, Elizabeth
- Donoway, Elizabeth
- Pavlick, Ellie
- Erdem, Erkut
- Dyer, Ethan
- Jerzak, Ethan
- Kim, Ethan
- Zheltonozhskii, Evgenii
- Mishra, Gaurav
- Betz, Gregor
- Kim, Hannah
- Hajishirzi, Hannaneh
- Mehta, Harsh
- Shevlin, Henry
- Schütze, Hinrich
- Yakura, Hiromu
- Zhang, Hongming
- Geissinger, Jack
- Lee, Jaehoon
- Fisac, Jaime Fernández
- Simon, James B.
- Koppel, James
- Zou, James
- Kocoń, Jan
- Phang, Jason
- Novikova, Jekaterina
- Kim, Jeremy
- Xu, Jiacheng
- Song, Jiaming
- Burden, John
- Rozen, Jos
- Tenenbaum, Joshua B.
- Rule, Joshua S.
- Kanclerz, Kamil
- Livescu, Karen
- Krauth, Karl
- Dhole, Kaustubh D.
- Mathewson, Kory
- Richardson, Kyle
- Zhang, Li
- Dugan, Liam
- Contreras-Ochando, Lidia
- Morency, Louis-Philippe
- Moschella, Luca
- Lam, Lucas
- Şenel, Lütfi Kerem
- Sap, Maarten
- Farooqi, Maheen
- Marelli, Marco
- Tolkiehn, Marie
- Giulianelli, Mario
- Lewis, Martha
- Potthast, Martin
- Leavitt, Matthew L.
- Hagen, Matthias
- Yee, Michael A.
- Ivanitskiy, Michael
- Strube, Michael
- Swędrowski, Michał
- Yasunaga, Michihiro
- Bansal, Mohit
- Geva, Mor
- T, Mukund Varma
- Peng, Nanyun
- Chi, Nathan A.
- Lee, Nayeon
- Deckers, Niklas
- Muennighoff, Niklas
- Keskar, Nitish Shirish
- Iyer, Niveditha S.
- Wen, Nuan
- Zhang, Oliver
- Levy, Omer
- Casares, Pablo Antonio Moreno
- Fung, Pascale
- Liang, Paul Pu
- Liao, Peiyuan
- Liang, Percy
- Miłkowski, Piotr
- Pezeshkpour, Pouya
- Oli, Priti
- Mei, Qiaozhu
- Banjade, Rabin
- Habacker, Rahel
- Millière, Raphaël
- Saurous, Rif A.
- Arakawa, Riku
- Frank, Robert
- Zhang, Rui
- Salakhutdinov, Ruslan
- Teehan, Ryan
- Singh, Sahib
- Mohammad, Saif M.
- Anand, Sajant
- Dillavou, Sam
- Wiseman, Sam
- Gruetter, Samuel
- Schoenholz, Samuel S.
- Rous, Sarah A.
- Ghosh, Sayan
- Casey, Sean
- Bischoff, Sebastian
- Gehrmann, Sebastian
- Schuster, Sebastian
- Hamdan, Shadi
- Zhou, Sharon
- Srivastava, Shashank
- Shi, Sherry
- Singh, Shikhar
- Asaadi, Shima
- Gu, Shixiang Shane
- Melzi, Simone
- Reddy, Siva
- Makini, Sneha Priscilla
- Lee, Soo-Hwan
- Dehaene, Stanislas
- Divic, Stefan
- Ermon, Stefano
- Biderman, Stella
- Lin, Stephanie
- Piantadosi, Steven T.
- Shieber, Stuart M.
- Kiritchenko, Svetlana
- Mishra, Swaroop
- Linzen, Tal
- Schuster, Tal
- Li, Tao
- Wu, Te-Lin
- Desbordes, Théo
- Wang, Tianle
- Nkinyili, Tiberius
- Tunduny, Titus
- Gerstenberg, Tobias
- Chang, Trenton
- Shaham, Uri
- Demberg, Vera
- Ramasesh, Vinay
- Prabhu, Vinay Uday
- Padmakumar, Vishakh
- Srikumar, Vivek
- Zhang, William
- Vossen, Wout
- Ren, Xiang
- Zhao, Xinran
- Wu, Xinyi
- Shen, Xudong
- Yaghoobzadeh, Yadollah
- Lakretz, Yair
- Song, Yangqiu
- Choi, Yejin
- Yang, Yichi
- Hao, Yiding
- Chen, Yifu
- Belinkov, Yonatan
- Hou, Yu
- Hou, Yufang
- Bai, Yuntao
- Zhao, Zhuoye
- Wang, Zijian
- Wang, Zijie J.
- Wu, Ziyi
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2206.04615
- OpenAlex
- W4281690148
- Semantic Scholar
- bd1331b233e84bab7eba503abc60b31ac08e7881
- Cited by
- 569
- References
- 0
- Citations by year
- 2026: 29 · 2025: 114 · 2024: 169 · 2023: 232 · 2022: 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- model scaling, sparse transformer, GPT, model calibration, social bias, commonsense reasoning, mathematical reasoning, prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
