The DOI matches its DataCite record; 527 of the 528 references it cites were found in a registry or an index.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Open accessAbstract
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even performing some traditionally symbolic tasks well with only a few training examples [61].”
From Symbols and grounding in large language models (2023) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Srivastava et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many benchmark sets do not allow for arbitrary generation of novel questions, or worse, draw data from publicly available sources-the same sources LLM trainers have access to [51, 39].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [39]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Surprisingly, we observe that using training FLOPs leads to reasonable predictions of LM performance with and without CoT on BBH tasks, possibly due to the denoising effect of aggregation over all tasks.”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In order to standardize our evaluation, we use the same prompt for all models evaluated, following the convention established by [83–85].”
From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [85]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One might thus posit, with some justification from the above-mentioned studies, that the emergence of skills arises from training on related tasks that were implicitly solved while solving next-word prediction in the training dataset.This is indeed our starting point.”
From A Theory for Emergence of Complex Skills in Language Models (2023) · cites this paper as Srivastava et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prominent among these are multi-task benchmarks such as MMLU [325] and BIG-bench [326], which evaluate knowledge and skills across diverse domains and scenarios.”
From A Survey of Large Language Models (2026) · cites this paper as [326]
The cited source says what the claim attributes to it.
In the citing paper
“To holistically assess the capabilities of LLMs, several comprehensive benchmarks have been developed [325-327].”
From A Survey of Large Language Models (2026) · cites this paper as [326]
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to prior benchmarks (e.g., SuperGLUE, 9 EleutherAI LM Harness, 10 and BIG-Bench 11 ), which enumerate a set of scenarios and metrics, we situate our scenarios in a larger taxonomy to make explicit what is currently missing.”
From Holistic Evaluation of Language Models (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“BIG-Bench Hard (BBH; Suzgun et al., 2022) is a set of 23 multiple-choice tasks from the BIG-Bench benchmark (Srivastava et al., 2022), on which current state-of-the-art models do substantially worse than the average human rater.”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“BIG-bench [182] introduces a diverse collection of 204 challenging tasks contributed by 450 authors from 132 institutions.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [182]
The cited source says what the claim attributes to it.
In the citing paper
“Popular tests, such as MMLU (137), BIG-Bench (138), BIG-Bench Hard (BBH) (139), or AGIEval (140) are typically based on human ability tasks (137) but test for knowledge recall or problem-solving (141), often by comparing performance against human experts doing the same tasks (138).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [138]
The cited source says what the claim attributes to it.
In the citing paper
“We show that when we use a format with visible lettered answer options, large models are very well calibrated on diverse multiple choice questions (e.g. from BIG Bench [Srivastava et al., 2022], MMLU[Hendrycks et al., 2021], and many other evaluations); see Figures 4, 5, and 6.”
From Language Models (Mostly) Know What They Know (2022) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Although biases already pose a challenge for conventional AI in health, they are of particular relevance for GMAI as a recent large-scale evaluation showed that social bias can increase with model scale 46 .”
From Foundation models for generalist medical artificial intelligence (2023) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Our experiments validate the effectiveness of this approach on a range of complex reasoning tasks, including several challenging Big-Bench tasks (Srivastava et al., 2022).”
From Large language models as tool makers (2023) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“We note that our evaluation is in no means comprehensive and a larger benchmarking akin to Srivastava et al. (2022) would be helpful.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Srivastava et al. (2022)
The cited source says what the claim attributes to it.
In the citing paper
“Large language models have demonstrated remarkable reasoning ability on a variety of reasoning tasks26–29.”
From Solving olympiad geometry without human demonstrations (2024) · cites this paper as [28]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Furthermore, using PC measures accurately captures the scaling trends in both setups, even when using training FLOPs leads to less tight captures in the”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 140
- Scott Alexander. A very unlikely chess game, 2020. URL https://slatestarcodex.com/2020/01/06/a-very-unlikely-chessgame/ . (cited on pp. 29 and 30)
- Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 245-256. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/alon20a.html . (cited on p. 37)
- Miriam Amin and Manuel Burghardt. A survey on approaches to computational humor generation. In Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature , pp. 29-41, Online, December 2020. International Committee on Computational Linguistics. URL https://aclanthology. org/2020.latechclfl-1.4 . (cited on p. 38)
- Brandon Amos and J. Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks, 2017. URL https: //arxiv.org/abs/1703.00443 . (cited on p. 38)
- Salvatore Attardo. Humor in language. In Oxford Research Encyclopedia of Linguistics . Oxford University Press, 2017. doi: 10. 1093/acrefore/9780199384655.013.342. URL https://oxfordre.com/linguistics/view/10.1093/acrefore/9780199384655. 001.0001/acrefore-9780199384655-e-342 . (cited on p. 38) · DOI
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: //arxiv.org/abs/2108.07732 . (cited on p. 4)
- Satanjeev Banerjee and Ted Pedersen. Extended gloss overlaps as a measure of semantic relatedness. In IJCAI'03: Proceedings of the 18th International Joint Conference on Artificial Intelligence , pp. 805-810, San Francisco, 2003. Morgan Kaufmann. doi: 10.5555/1630659.1630775. URL https://dl.acm.org/doi/10.5555/1630659.1630775 . (cited on p. 36) · DOI
- Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review , 104(3):671-732, 2016. URL http://www.jstor.org/stable/24758720 . (cited on p. 31)
- Gregor Betz, Christian Voigt, and Kyle Richardson. Critical thinking for language models, 2020. URL https://arxiv.org/abs/ 2009.07185 . (cited on p. 33)
- Stella Biderman and Edward Raff. Neural language models are effective plagiarists, 2022. URL https://arxiv.org/abs/2201. 07406 . (cited on p. 4)
- Julia Birke and Anoop Sarkar. A clustering approach for nearly unsupervised recognition of nonliteral language. In 11th Conference of the European Chapter of the Association for Computational Linguistics , pp. 329-336, Trento, Italy, April 2006. Association for Computational Linguistics. URL https://aclanthology.org/E06-1042 . (cited on p. 35)
- Nicholas Boillot. Vector forms as a foreign language, 24 June 2019. URL https://www.fluate.net/en/travaux/vectoglyph . (cited on pp. 31 and 33)
- Paul F. Boller, Jr. and John George. They Never Said It: A Book of Fake Quotes, Misquotes, and Misleading Attributions . Oxford University Press, Oxford, 1989. (cited on p. 35)
- Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings. neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html . (cited on pp. 17, 29, 31, and 33)
- Bozhidar Bozhanov and Ivan Derzhanski. Rosetta stone linguistic problems. In Proceedings of the Fourth Workshop on Teaching NLP and CL , pp. 1-8, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https: //aclanthology.org/W13-3401 . (cited on p. 34)
- Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable Forth interpreter. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 547-556, 2017. URL https: //arxiv.org/abs/1605.06640 . (cited on p. 34)
- Gwern Branwen. GPT-3 creative fiction. Gwern.net , June 2020. URL https://www.gwern.net/GPT-3 . (cited on pp. 19, 38, and 39)
- Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review , 78(1):1-3, 1950. (cited on pp. 8 and 11)
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html . (cited on pp. 4, 7, 8, 10, 11, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40, and 41)
- Corrado Böhm. On a family of Turing machines and the related programming language. ICC Bulletin , 3:187-194, 1964. (cited on p. 38)
- Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pp. 2633-2650. USENIX Association, August 2021. URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting . (cited on p. 19)
- Nathanael Chambers. Labeling documents with timestamps: Learning from their time expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 98-106, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1011 . (cited on p. 31)
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 1691-1703. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/chen20s.html . (cited on p. 29)
- Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. https://openreview.net/pdf?id= H1gfOiAqYm , 2019b. (cited on p. 34)
- Kate Crawford. The trouble with bias. https://www.youtube.com/watch?v=fMym\_BKWQzk , 2017. Keynote address, NIPS 2017, Long Beach CA. Dec. 5, 2017. (cited on p. 29)
- Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 933-941, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305478. URL https://dl.acm.org/doi/10.5555/3305381. 3305478 . (cited on p. 10) · DOI
- Wayne Davis. Implicature. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Fall 2019 edition, 2019. (cited on p. 33)
- Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. Finding contradictions in text. In Proceedings of ACL-08: HLT , pp. 1039-1047, Columbus, Ohio, June 2008. Association for Computational Linguistics. URL https: //aclanthology.org/P08-1118 . (cited on p. 32)
- Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. RobustFill: Neural program learning under noisy I/O. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 990-998, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305484. URL https://dl.acm.org/doi/10.5555/3305381.3305484 . (cited on p. 34) · DOI
- Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. Beyond English-centric multilingual machine translation. Journal of Machine Learning Research , 22(107): 1-48, 2021. URL http://jmlr.org/papers/v22/20-1307.html . (cited on p. 20)
- Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database . MIT Press, Cambridge, MA, 1998. (cited on p. 33)
- Jerry A. Fodor. The Language of Thought . Harvard University Press, Cambridge, MA, 1975. (cited on p. 30)
- Evgeniy Gabrilovich and Shaul Markovitch. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In IJCAI'07: Proceedings of the 20th International Joint Conference on Artifical Intelligence , pp. 1606-1611, San Francisco, 2007. Morgan Kaufmann. doi: 10.5555/1625275.1625535. URL https://dl.acm.org/doi/10.5555/1625275.1625535 . (cited on p. 36) · DOI
- Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics , 9: 346-361, 04 2021. doi: 10.1162/tacl\_a\_00370. URL https://doi.org/10.1162/tacl\_a\_00370 . (cited on pp. 29 and 39) · DOI
- Arthur S. Goldberger. Structural equation methods in the social sciences. Econometrica , 40(6):979-1001, 1972. URL http://www.jstor.org/stable/1913851 . (cited on p. 32)
- Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pp. 581-586, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https://aclanthology.org/P11-2102 . (cited on p. 39)
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pp. 1321-1330. PMLR, 06-11 Aug. 2017. URL https://proceedings.mlr.press/v70/guo17a. html . (cited on p. 11)
- Samuel Gyasi Obeng. The proverb as a mitigating and politeness strategy in Akan discourse. Anthropological Linguistics , 38(3): 521-549, 1996. URL http://www.jstor.org/stable/30028601 . (cited on p. 32)
- Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171, Dec. 2020. doi: 10.1162/tacl\_a\_00306. URL https://doi.org/10.1162/tacl\_a\_00306 . (cited on p. 31) · DOI
- Joseph Y. Halpern. Actual causality . MIT Press, Cambridge, MA, 2016. (cited on p. 29)
- Maria Hanzén. When in Rome, do as the Romans do: Proverbs as a part of EFL teaching. Master's thesis, Jönköping University, School of Education and Communication, Jönköping, 2007. URL http://www.diva-portal.org/smash/get/diva2: 3499/fulltext01.pdf . (cited on p. 40)
- Irene Heim. On the projection problem for presuppositions. In Paul Portner and Barbara H. Partee (eds.), Formal Semantics The Essential Readings , pp. 249-260. Blackwell, Oxford, 1983. (cited on p. 37)
- Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Hkg4TI9xl . (cited on pp. 11 and 12)
- Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 2712-2721. PMLR, 09-15 June 2019. URL https: //proceedings.mlr.press/v97/hendrycks19a.html . (cited on p. 11)
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS, 2021a. URL https: //arxiv.org/abs/2105.09938 . (cited on pp. 4, 37, and 41)
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021b. URL https: //openreview.net/forum?id=d7KBjmI3GmQ . (cited on pp. 25, 31, 35, 38, and 40)
- Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips. cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html . (cited on pp. 5 and 32)
- Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URL https: //arxiv.org/abs/2102.01293 . (cited on p. 41)
- Keith J. Holyoak. Analogy and relational reasoning. In Keith J. Holyoak and Robert G. Morrison (eds.), The Oxford Handbook of Thinking and Reasoning . Oxford University Press, Oxford, 2012. URL https://www.oxfordhandbooks.com/view/10.1093/ oxfordhb/9780199734689.001.0001/oxfordhb-9780199734689-e-13 . (cited on p. 28)
- Yufang Hou. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 1428-1438, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.132. URL https://aclanthology.org/2020.acl-main.132 . (cited on p. 29)
- Yufang Hou, Katja Markert, and Michael Strube. Global inference for bridging anaphora resolution. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 907-917, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology.org/N131111 . (cited on p. 29)
- Hrisztalina Hrisztova-Gotthardt and Melita Aleksa Varga. Introduction to Paremiology: A Comprehensive Guide to Proverb Studies . De Gruyter Open, Warsaw, 2015. URL https://www.degruyter.com/document/doi/10.2478/9783110410167/html . (cited on p. 32) · DOI
- Thad Hughes and Daniel Ramage. Lexical semantic relatedness with random graph walks. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , pp. 581-589, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/D07-1061 . (cited on p. 36)
- Jay J. Jiang and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th Research on Computational Linguistics International Conference , pp. 19-33, Taipei, Taiwan, August 1997. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). URL https://aclanthology.org/O97-1002 . (cited on p. 36)
- Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438, 2020. doi: 10.1162/tacl\_a\_00324. URL https://doi.org/10. 1162/tacl\_a\_00324 . (cited on pp. 11 and 12) · DOI
- Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS'15 , volume 1, pp. 190-198, Cambridge, MA, USA, 2015. MIT Press. doi: 10.5555/2969239.2969261. URL https://dl.acm.org/doi/10.5555/2969239.2969261 . (cited on p. 34) · DOI
- Lauri Karttunen. Simple and phrasal implicatives. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pp. 124-131, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1020 . (cited on p. 32)
- Faisal Khan, Bilge Mutlu, and Jerry Zhu. How do humans teach: On curriculum learning and teaching dimension. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper/2011/file/ f9028faec74be6ec9b852b0a542e2f39-Paper.pdf . (cited on p. 38)
- Milton King and Paul Cook. Evaluating approaches to personalizing language models. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 2461-2469, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.299 . (cited on p. 38)
- Christo Kirov and Ryan Cotterell. Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate. Transactions of the Association for Computational Linguistics , 6:651-665, 12 2018. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00247. URL https://doi.org/10.1162/tacl\_a\_00247 . (cited on p. 34) · DOI
- Joshua Knobe. Intentional action and side effects in ordinary language. Analysis , 63, 07 2003. doi: 10.1111/ 1467-8284.00419. URL https://www.researchgate.net/publication/28763794\_Intentional\_Action\_and\_Side\_Effects\_ in\_Ordinary\_Language . (cited on p. 29)
- Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics , 6:317-328, 2018. doi: 10.1162/tacl\_a\_00023. URL https://aclanthology.org/Q18-1023 . (cited on p. 35) · DOI
- Alexander W. Kocurek, Ethan Jerzak, and Rachel Etta Rudolph. Against conventional wisdom. Philosophers' Imprint , 20(22): 1-27, 2020. URL http://hdl.handle.net/2027/spo.3521354.0020.022 . (cited on p. 29)
- Jarmo Korhonen. Sprichwörter und zweisprachige lexikographie: Deutsch-schwedische und deutsch-finnische wörtebücher im vergleich. In C. Földes (ed.), Phraseologie disziplinär und interdisziplinär , pp. 537-549. Gunter Narr Verlag, 2009. (cited on p. 40)
- Mahnaz Koupaee and William Yang Wang. WikiHow: A large scale text summarization dataset, 2018. URL https://arxiv. org/abs/1810.09305 . (cited on p. 5)
- Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics , 10:50-72, 01 2022. doi: 10.1162/tacl\_a\_00447. URL https://doi.org/10.1162/tacl\_a\_00447 . (cited on p. 20) · DOI
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 08 2019. doi: 10.1162/tacl\_a\_00276. URL https://doi.org/10.1162/tacl\_a\_00276 . (cited on pp. 5 and 37) · DOI
- Kevin Lacker. Giving GPT-3 a Turing test. Kevin Lacker's blog , July 2020. URL https://lacker.io/ai/2020/07/06/givinggpt-3-a-turing-test.html . (cited on pp. 33 and 41)
- George Lakoff and Mark Johnson. Metaphors We Live By . University of Chicago Press, Chicago, 2008. (cited on pp. 33 and 35)
- Guillaume Lample and François Charton. Deep learning for symbolic mathematics, 2019. URL https://arxiv.org/abs/1912. 01412 . (cited on p. 38)
- Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 986-995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL https://aclanthology.org/I17-1099 . (cited on p. 40)
- Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics , 4:521-535, 12 2016. doi: 10.1162/tacl\_a\_00115. URL https://doi.org/10.1162/tacl\_a\_00115 . (cited on p. 39) · DOI
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019. URL https://arxiv.org/abs/1907. 11692 . (cited on pp. 39 and 41)
- Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. In Vivek Nigam, Tajana Ban Kirigin, Carolyn Talcott, Joshua Guttman, Stepan Kuznetsov, Boon Thau Loo, and Mitsuhiro Okada (eds.), Logic, Language, and Security . Springer, Cham, 2020. URL https://www.springerprofessional. de/en/gender-bias-in-neural-natural-language-processing/18531692 . (cited on p. 33)
- Jihang Mao and Wanli Liu. A BERT-based approach for automatic humor detection and scoring. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019) , pp. 197-202, 2019. URL http://ceur-ws.org/Vol-2421/HAHA\_paper\_8.pdf . (cited on p. 39)
- Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence, 2020. URL https://arxiv.org/abs/ 2002.06177 . (cited on p. 30)
- Katja Markert, Yufang Hou, and Michael Strube. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 795-804, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1084 . (cited on p. 29)
- R. Thomas McCoy, Robert Frank, and Tal Linzen. Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks. Transactions of the Association for Computational Linguistics , 8:125-140, 01 2020. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00304. URL https://doi.org/10.1162/tacl\_a\_00304 . (cited on p. 34) · DOI
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https: //arxiv.org/abs/1609.07843 . (cited on p. 31)
- John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 6905-6916. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/miller20a.html . (cited on p. 39)
- David Milne and Ian H. Witten. An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: An Evolving Synergy , pp. 25-30, Menlo Park, 2008. Association for the Advancement of Artificial Intelligence. URL https://www.aaai.org/Papers/Workshops/2008/WS08-15/WS08-15-005.pdf . (cited on p. 36)
- Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Natural reference to objects in a visual domain. In Proceedings of the 6th International Natural Language Generation Conference . Association for Computational Linguistics, July 2010. URL https://aclanthology.org/W10-4210 . (cited on p. 38)
- Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Generating expressions that refer to visible objects. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 1174-1184, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https:// aclanthology.org/N13-1137 . (cited on p. 38)
- Elham Mohammadi, Hessam Amini, and Leila Kosseim. CLaC at CLPsych 2019: Fusion of neural features and predicted class probabilities for suicide risk assessment based on online posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology , pp. 34-38, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-3004. URL https://aclanthology.org/W19-3004 . (cited on p. 39) · DOI
- Jane Morris and Graeme Hirst. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics , 17(1):21-48, 1991. URL https://aclanthology.org/J91-1002 . (cited on p. 36)
- Karl Mulligan, Robert Frank, and Tal Linzen. Structure here, bias there: Hierarchical generalization by jointly learning syntactic transformations. In Proceedings of the Society for Computation in Linguistics 2021 , pp. 125-135, Online, February 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.scil-1.12 . (cited on p. 34)
- Gregory L. Murphy. Comprehending complex concepts. Cognitive Science , 12(4):529-562, 1988. doi: https://doi.org/10.1207/ s15516709cog1204\_2. URL https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1204\_2 . (cited on p. 30) · DOI
- Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 8558cb408c1d76621371888657d2eb1d-Abstract.html . (cited on p. 11)
- Anthony M. Paul. Figurative language. Philosophy & Rhetoric , 3(4):225-248, 1970. URL http://www.jstor.org/stable/ 40237206 . (cited on p. 35)
- Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann, San Francisco, 1988. (cited on p. 32)
- Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, Cambridge, 2000. (cited on p. 30)
- Tony A. Plate. Distributed representations and nested compositional structure . PhD thesis, University of Toronto, Toronto, 1994. (cited on p. 29)
- Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving, 2020. URL https://arxiv. org/abs/2009.03393 . (cited on pp. 34 and 38)
- Willard V.O. Quine. Main trends in recent philosophy: Two dogmas of empiricism. The Philosophical Review , 60(1):20-43, 1951. URL http://www.jstor.org/stable/2181906 . (cited on p. 29)
- Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/languagemodels.pdf . (cited on pp. 10, 30, 33, 35, 37, 38, and 39)
- Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pp. 777-789, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology. org/D12-1071 . (cited on p. 41)
- Ian Ravenscroft. Folk psychology as a theory. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Summer 2019 edition, 2019. (cited on p. 32)
- Joseph Reisinger and Raymond J. Mooney. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 109-117, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology. org/N10-1013 . (cited on p. 36)
- Philip Resnik. Using information content to evaluate semantic similarity in a taxonomy. In IJCAI'95: Proceedings of the 14th International Joint Conference on Artificial Intelligence, Volume 1 , pp. 448-453, San Francisco, 1995. Morgan Kaufmann. doi: 10.5555/1625855.1625914. URL https://dl.acm.org/doi/10.5555/1625855.1625914 . (cited on p. 36) · DOI
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. AAAI Spring Symposium , 2011. URL http://commonsensereasoning.org/2011/papers/Roemmele.pdf . (cited on p. 32)
- Joshua S. Rule. The child as hacker: Building more human-like models of learning . PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2020. URL https://hdl.handle.net/1721.1/129232 . (cited on p. 34)
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 . (cited on pp. 9 and 25)
- Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 624-643, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.52. URL https://aclanthology.org/2021.naacl-main.52 . (cited on p. 41) · DOI
- Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021b. URL https://openreview.net/ forum?id=fe\_hCc4RBrg . (cited on p. 4)
- Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. Diagram understanding in geometry questions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, Menlo Park, CA, Jun. 2014. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9146 . (cited on p. 34)
- Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL http: //arxiv.org/abs/1804.04235 . (cited on p. 43)
- Ekaterina Shutova. Automatic metaphor interpretation as a paraphrasing task. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 1029-1037, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology.org/N10-1147 . (cited on p. 35)
- Ekaterina Shutova and Simone Teufel. Metaphor corpus annotated for source-target domain mappings. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) , Valletta, Malta, May 2010. European Language Resources Association. URL http://www.lrec-conf.org/proceedings/lrec2010/pdf/612\_Paper.pdf . (cited on p. 35)
- Damien Sileo, Tim Van de Cruys, Camille Pradel, and Philippe Muller. DiscSense: Automated semantic analysis of discourse markers. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 991-999, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.125 . (cited on p. 31)
- Gurdeep Singh, Kaustubh Dhole, Priyadarshini P. Pai, and Sukanta Mondal. SPRINGS: Prediction of protein-protein interaction sites using artificial neural networks. Journal of Proteomics & Computational Biology , 1:7, 2014. URL https: //www.avensonline.org/fulltextarticles/JPCB-2572-8679-01-0001.html . (cited on p. 37)
- Douglas R. Smith. The synthesis of LISP programs from examples: A survey. In Alan W. Biermann, Gerhard Guiho, and Yves Kodratoff (eds.), Automatic Program Construction Techniques , pp. 307-324. Macmillan, New York, 1984. (cited on p. 34)
- Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences , 11(1):1-23, 1988. doi: 10.1017/ S0140525X00052432. (cited on p. 30)
- Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://aclanthology.org/D13-1170 . (cited on p. 31)
- Shashank Srivastava, Snigdha Chaturvedi, and Tom Mitchell. Inferring interpersonal relations in narrative summaries. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16 , pp. 2807-2813, Menlo Park, CA, 2016. Association for the Advancement of Artificial Intelligence. doi: 10.5555/3016100.3016294. URL https://dl.acm.org/doi/10. 5555/3016100.3016294 . (cited on p. 34) · DOI
- Robert Stalnaker. Assertion. In P. Cole (ed.), Pragmatics , Syntax and Semantics 9, pp. 315-332. Brill, Leiden, 1978. (cited on p. 32)
- Bernd Steinbach and Roman Kohut. Neural networks - a model of boolean functions. 5th International Workshop on Boolean Problems, Freiburg, Sept. 2002. , 2002. URL https://www.researchgate.net/publication/246931125\_Neural\_Networks\_-\_A\_Model\_of\_Boolean\_Functions . (cited on p. 29)
- Andreas Stöckl. Watching a language model learning chess. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) , pp. 1369-1379, Held Online, September 2021. INCOMA Ltd. URL https://aclanthology.org/2021.ranlp-1.153 . (cited on p. 4)
- Kevin Stowe, Leonardo Ribeiro, and Iryna Gurevych. Metaphoric paraphrase generation, 2020. URL https://arxiv.org/abs/ 2002.12854 . (cited on p. 35)
- Michael Strube and Simone Paolo Ponzetto. Wikirelate! Computing semantic relatedness using Wikipedia. In AAAI'06: Proceedings of the 21st National Conference on Artificial Intelligence , volume 2, pp. 1419-1424. Association for the Advancement of Artificial Intelligence, 2006. doi: 10.5555/1597348.1597414. URL https://dl.acm.org/doi/10.5555/1597348. 1597414 . (cited on p. 36) · DOI
- Saku Sugawara, Hikaru Yokono, and Akiko Aizawa. Prerequisite skills for reading comprehension: Multi-perspective analysis of MCTest datasets and systems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, Menlo Park, CA, Feb. 2017. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/ view/10957 . (cited on p. 35)
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks, 2014. URL https: //arxiv.org/abs/1409.3215 . (cited on p. 4)
- Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Tunuguntla Deepak, and Oskar van der Wal. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models , 2022. URL https://openreview.net/forum?id=rK-7NhfSIW5 . (cited on pp. 18 and 20)
- Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. Learning to recommend quotes for writing. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15 , pp. 2453-2459, Menlo Park, CA, 2015. Association for the Advancement of Artificial Intelligence. URL https://dl.acm.org/doi/10.5555/2886521.2886662 . (cited on p. 28) · DOI
- Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15 , pp. 1923-1929, 2015. URL https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/view/10957/10931 . (cited on p. 38)
- George Tsatsaronis, Iraklis Varlamis, Michalis Vazirgiannis, and Kjetil Nørvåg. Omiotis: A thesaurus-based measure of text relatedness. In Wray Buntine, Marko Grobelnik, Dunja Mladenić, and John Shawe-Taylor (eds.), Machine Learning and Knowledge Discovery in Databases , pp. 742-745, Berlin, 2009. Springer. (cited on p. 36)
- Shikhar Vashishth, Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Dating documents using graph convolution networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1605-1615, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ v1/P18-1149. URL https://aclanthology.org/P18-1149 . (cited on p. 31)
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 30, pp. 5998--6008. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf . (cited on pp. 10 and 38)
- Jette Viethen and Robert Dale. The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference , pp. 59-67, Salt Fork, Ohio, USA, June 2008. Association for Computational Linguistics. URL https://aclanthology.org/W08-1109 . (cited on p. 38)
- Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers , pp. 176-187, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-1017 . (cited on p. 35)
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019a. URL https://proceedings.neurips.cc/paper/2019/hash/ 4496bf24afe7fab6f046bf4923da8de6-Abstract.html . (cited on pp. 5, 7, 29, 32, and 41)
- David Wechsler. Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) . Pearson, San Antonio, 2008. (cited on p. 38)
- Ludwig Wittgenstein. Philosophical investigations . Basil Blackwell, Oxford, 1953. (cited on p. 29)
- Shijie Wu, Ryan Cotterell, and Mans Hulden. Applying the transformer to character-level transduction, 2020. URL https: //arxiv.org/abs/2005.10213 . (cited on p. 41)
- Scott Cheng-Hsin Yang and Patrick Shafto. Explainable artificial intelligence via Bayesian teaching. Workshop on Teaching Machines, Robots, and Humans, NIPS 2017 , 2017. URL http://shaftolab.com/assets/papers/yangShafto\_NIPS\_2017\_ machine\_teaching.pdf . (cited on p. 38)
- Eric Yeh, Daniel Ramage, Christopher D. Manning, Eneko Agirre, and Aitor Soroa. WikiWalk: Random walks on Wikipedia for semantic relatedness. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4) , pp. 41-49, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https: //aclanthology.org/W09-3206 . (cited on p. 36)
- Wenpeng Yin and Yulong Pei. Optimizing sentence modeling and selection for document summarization. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI-15 , pp. 1383-1389, 2015. doi: 10.5555/2832415.2832442. URL https://dl.acm.org/doi/10.5555/2832415.2832442 . (cited on p. 32) · DOI
- Eliezer Yudkowsky. Artificial intelligence as a positive and negative factor in global risk. In Nick Bostrom and Milan M. Ćirković (eds.), Global Catastrophic Risks , pp. 308-345. Oxford University Press, Oxford, 2008. URL https://web.archive.org/web/ 20210125025955/https://intelligence.org/files/AIPosNegFactor.pdf . (cited on p. 38)
- Omnia Zayed, John Philip McCrae, and Paul Buitelaar. Figure me out: A gold standard dataset for metaphor interpretation. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 5810-5819, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.712 . (cited on p. 35)
- Meishan Zhang, Yue Zhang, and Guohong Fu. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pp. 2449-2460, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https://aclanthology.org/C16-1231 . (cited on p. 39)
- Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, Menlo Park, CA, Mar. 2015. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9761 . (cited on p. 38)
Cited by
Sorted by citations
Items 201–300 of 681
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26
TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications
Nikbakht, R., Benzaghta, M., & Geraci, G. (2024) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshp64532.2024.11101012 · Cited by 58
Introducing MBIB - The First Media Bias Identification Benchmark Task and Dataset Collection
Wessel, M. P., Horych, T., Ruas, T., Aizawa, A., Gipp, B., & Spinde, T. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591882 · Cited by 31
Don't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25
Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions
Thakur, H., Jain, A., Vaddamanu, P., Liang, P. P., & Morency, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.30 · Cited by 39
Self-citationCredible without Credit: Domain Experts Assess Generative Language Models
Peskoff, D., & Stewart, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.37 · Cited by 31
Can Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Liang, J., & Xiao, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29777 · Cited by 45
AI and the Future of Skills, Volume 2
OECD (2023) · Educational research and innovation · doi:10.1787/a9fe53cb-en · Cited by 43
Systematic Literature Review of Prompt Engineering Patterns in Software Engineering
Sasaki, Y., Washizaki, H., Li, J., Sander, D., Yoshioka, N., & Fukazawa, Y. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00096 · Cited by 27
MoT: Memory-of-Thought Enables ChatGPT to Self-Improve
Li, X., & Qiu, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.392 · Cited by 53
Post Hoc Explanations of Language Models Can Improve Language ModelsPreprint
Krishna, S., Ma, J., Slack, D., Ghandeharioun, A., Singh, S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3006112/v1 · Cited by 24
Self-citationA Close Look into the Calibration of Pre-trained Language Models
Chen, Y., Yuan, L., Cui, G., Liu, Z., & Ji, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.75 · Cited by 22
Self-citationInverse Scaling Can Become U-Shaped
Wei, J., Kim, N., Tay, Y., & Le, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.963 · Cited by 25
Grammatical cues to subjecthood are redundant in a majority of simple clauses across languages
Mahowald, K., Diachek, E., Gibson, E. A. F., Fedorenko, E. G., & Futrell, R. (2023) · Cognition · doi:10.1016/j.cognition.2023.105543 · Cited by 31
Emerging trends: When can users trust GPT, and when should they intervene?
Church, K. (2024) · Natural Language Engineering · doi:10.1017/s1351324923000578 · Cited by 23
Probing Pretrained Models of Source Codes
Troshin, S., & Chirkova, N. (2022) · Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2022.blackboxnlp-1.31 · Cited by 23
ZeroSCROLLS: A zero-shot benchmark for long text understanding
Shaham, U., Ivgi, M., Efrat, A., Berant, J., & Levy, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.536 · Cited by 23
Self-citationRed teaming large language models: A comprehensive review and critical analysis
Jabbar, M. S., Al-Azani, S., Alotaibi, A., & Ahmed, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104239 · Cited by 31
Human agents, generative AI, and innovation: A formal model of hybrid creative process
Pedota, M., Cicala, F., & Basti, A. (2025) · Technovation · doi:10.1016/j.technovation.2025.103323 · Cited by 31
Language acquisition: do children and language models follow similar learning stages?
Evanson, L., Lakretz, Y., & King, J. R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.773 · Cited by 41
Self-citationChain of Thought Prompting Elicits Knowledge Augmentation
Wu, D., Zhang, J., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.408 · Cited by 33
Evaluating human resources management literacy: A performance analysis of ChatGPT and bard
Raghu Raman, R., Venugopalan, M., & Kamal, A. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e27026 · Cited by 21
Natural Language Processing with Commonsense Knowledge: A SurveyPreprint
Xie, Y., Liu, Z., Ma, Z., Meng, F., Xiao, Y., Miao, F., & Pu, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04674 · Cited by 25
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L., Zhang, R., et al. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3188 · Cited by 21
Self-citationCONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation
Ravichander, A., Gardner, M., & Marasović, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.598 · Cited by 34
KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts
Coscia, A. J., & Endert, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346713 · Cited by 22
Woźniak, S., & Kocoń, J. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00108 · Cited by 23
Self-citationCalibrating Large Language Models with Sample Consistency
Lyu, Q., Shridhar, K., Malaviya, C., Zhang, L., Elazar, Y., Tandon, N., Apidianaki, M., Sachan, M., & Callison-Burch, C. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i18.34120 · Cited by 21
Self-citationSo you want your private LLM at home? A survey and benchmark of methods for efficient GPTs
Tuggener, L., Sager, P., Taoudi-Benchekroun, Y., Grewe, B. F., & Stadelmann, T. (2024) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds60720.2024.00036 · Cited by 23
How Do In-Context Examples Affect Compositional Generalization?
An, S., Lin, Z., Fu, Q., Chen, B., Zheng, N., Lou, J., & Zhang, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.618 · Cited by 46
Equipping Llama with Google Query API for Improved Accuracy and Reduced HallucinationPreprint
Bae, Y. S., Kim, H. R., & Kim, J. H. (2024) · Research Square · doi:10.21203/rs.3.rs-4014474/v1 · Cited by 19
Self-citationA Comparative Analysis to Evaluate Bias and Fairness Across Large Language Models with BenchmarksPreprint
陳文意, & 黃兆明 (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/mc762 · Cited by 23
盖振华, Tong, L., & Ge, Q. (2024) · doi:10.31219/osf.io/ctw8v · Cited by 19
LLM Evaluators Recognize and Favor Their Own Generations
Panickssery, A., Bowman, S. R., & Feng, S. (2024) · arXiv · doi:10.48550/arxiv.2404.13076 · Cited by 559
“This effect persists when fine-tuning on one dataset and measuring results on the other dataset.”
Self-citationPrettyman, A. (2026) · Inquiry · doi:10.1080/0020174x.2024.2439989 · Cited by 18
Artificial Intelligence in Science and Society: The Vision of USERN
Dorigo, T., Brown, G. D., Casonato, C., Cerdà, A., Ciarrochi, J., Lio, M. D., D’Souza, N., Gauger, N. R., Hayes, S. C., Hofmann, S. G., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3529357 · Cited by 21
Transforming agency: On the mode of existence of large language models
Barandiaran, X. E., & Almendros, L. S. (2025) · Phenomenology and the Cognitive Sciences · doi:10.1007/s11097-025-10094-3 · Cited by 35
Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling
Šakota, M., Peyrard, M., & West, R. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635825 · Cited by 49
Bellini-Leite, S. C. (2023) · Adaptive Behavior · doi:10.1177/10597123231206604 · Cited by 25
Integration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 20
KoBBQ: Korean Bias Benchmark for Question Answering
Jin, J., Kim, J., Lee, N., Yoo, H., Oh, A. H., & Lee, H. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00661 · Cited by 27
Self-citationArtificial intelligence as a catalyst for combustion science and engineering
Ihme, M., & Chung, W. T. (2024) · Proceedings of the Combustion Institute · doi:10.1016/j.proci.2024.105730 · Cited by 22
SmartLLMSentry: A Comprehensive LLM Based Smart Contract Vulnerability Detection Framework
Zaazaa, O., & Bakkali, H. E. (2024) · Journal of Metaverse · doi:10.57019/jmv.1489060 · Cited by 23
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Self-citationThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“Large-scale benchmarking has proven useful in many areas of natural language processing (Wang et al., 2019; Kiela et al., 2021; Srivastava et al., 2022; Muennighoff et al., 2022a).”
BackgroundSelf-citationLiu, F., Zhou, H., Wang, K., Yu, Y., Gao, Y., Sun, Z., Liu, S., Sun, S., Zou, Z., Li, Z., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102056 · Cited by 22
Self-citationOn the Calibration of Large Language Models and Alignment
Zhu, C., Xu, B., Wang, Q., Zhang, Y., & Mao, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.654 · Cited by 50
Massively Multilingual Shallow Fusion with Large Language Models
Hu, K., Sainath, T. N., Li, B., Du, N., Huang, Y., Dai, A. M., Zhang, Y., Cabrera, R., Chen, Z., & Strohman, T. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094796 · Cited by 19
Self-citationPotential benefits of employing large language models in research in moral education and development
Han, H. (2023) · Journal of Moral Education · doi:10.1080/03057240.2023.2250570 · Cited by 22
Prompting Large Language Models for Malicious Webpage Detection
Li, L., & Gong, B. (2023) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml59573.2023.10348229 · Cited by 19
Bringing Worker Voice into Generative AI
Kochan, T. A., Armstrong, B., Shah, J., Castilla, E. J., Likis, B., & Mangelsdorf, M. E. (2024) · doi:10.21428/e4baedd9.0d255ab6 · Cited by 17
Quantum many-body physics calculations with large language models
Pan, H., Mudur, N., Taranto, W., Tikhanovskaya, M., Venugopalan, S., Bahri, Y., Brenner, M. P., & Kim, E.-A. (2025) · Communications Physics · doi:10.1038/s42005-025-01956-y · Cited by 15
Self-citationSociotechnical Implications of Generative Artificial Intelligence for Information Access
Mitra, B., Cramer, H. S. M., & Gurevich, O. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_7 · Cited by 28
Large language models for causal hypothesis generation in science
Cohrs, K.-H., Díaz, E., Sitokonstantinou, V., Varando, G., & Camps‐Valls, G. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ada47f · Cited by 23
Skill-Based Few-Shot Selection for In-Context Learning
An, S., Zhou, B., Lin, Z., Fu, Q., Chen, B., Zheng, N., Chen, W., & Lou, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.831 · Cited by 42
Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement
Xi, Z., Jin, S., Zhou, Y., Zheng, R., Gao, S., Liu, J., Gui, T., Zhang, Q., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.762 · Cited by 40
Bioinfo-Bench: A Simple Benchmark Framework for LLM Bioinformatics Skills EvaluationPreprint
Chen, Q., & Deng, C. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.18.563023 · Cited by 16
Self-citationThe bewitching AI: The Illusion of Communication with Large Language Models
Bottazzi Grifoni, E., & Ferrario, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00893-6 · Cited by 15
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1416-x · Cited by 18
Pitfalls (and advantages) of sophisticated large language models
Strasser, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-18851-0.00007-x · Cited by 20
Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
Nagar, A., Jaiswal, S., & Tan, C. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650020 · Cited by 21
A Vision Check-up for Language Models
Sharma, P., Shaham, T. R., Baradad, M., Rodriíuez-Muñoz, A., Duggal, S., Isola, P., Torralba, A., & Fu, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01366 · Cited by 15
Landscape and Taxonomy of Prompt Engineering Patterns in Software Engineering
Sasaki, Y., Washizaki, H., Li, J., Yoshioka, N., Ubayashi, N., & Fukazawa, Y. (2025) · IT Professional · doi:10.1109/mitp.2024.3525458 · Cited by 19
Modify Mistral Large Performance with Low-Rank Adaptation (LoRA) on the BIG-Bench DatasetPreprint
Fujiwara, H., Kimura, R., & Nakano, T. (2024) · Research Square · doi:10.21203/rs.3.rs-4440421/v1 · Cited by 15
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14499 · Cited by 14
Self-citationeP-ALM: Efficient Perceptual Augmentation of Language Models
Shukor, M., Dancette, C., & Cord, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02016 · Cited by 16
Michaelov, J. A., & Bergen, B. K. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.891 · Cited by 16
MCC-KD: Multi-CoT Consistent Knowledge Distillation
Chen, H., Wu, S., Quan, X., Wang, R., Yan, M., & Zhang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.454 · Cited by 36
CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answering
Rony, M. R. A. H., Suess, C., Bhat, S. R., Sudhi, V., Schneider, J., Vogel, M., Teucher, R., Friedl, K. E., & Sahoo, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.56 · Cited by 15
CRAB: Assessing the Strength of Causal Relationships Between Real-world Events
Romanou, A., Montariol, S., Paul, D., Laugier, L., Aberer, K., & Bosselut, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.940 · Cited by 36
GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation
Dhole, K., & Agichtein, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56063-7_24 · Cited by 23
Self-citationOn the Unexpected Abilities of Large Language Models
Nolfi, S. (2024) · Adaptive Behavior · doi:10.1177/10597123241256754 · Cited by 19
Cook2LTL: Translating Cooking Recipes to LTL Formulae using Large Language Models
Mavrogiannis, A., Mavrogiannis, C., & Aloimonos, Y. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611086 · Cited by 19
The Two Word Test as a semantic benchmark for large language models
Riccardi, N., Yang, X., & Desai, R. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72528-3 · Cited by 23
(2022) · doi:10.2139/ssrn.4496380 · Cited by 20
The Future of Fundamental Science Led by Generative Closed-Loop Artificial Intelligence
Zenil, H., Tegner, J. N., Abrahão, F. S., Lavin, A., Kumar, V., Frey, J. G., Weller, A., Soldatova, L., Bundy, A., Jennings, N. R., et al. (2023) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.3389/frai.2026.1678539
Training Trajectories of Language Models Across Scales
Xia, M., Artetxe, M., Zhou, C., Lin, X. V., Pasunuru, R., Chen, D., Zettlemoyer, L., & Stoyanov, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.767 · Cited by 49
Self-citationA step-by-step method for cultural annotation by LLMs
Dubourg, E., Thouzeau, V., & Baumard, N. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1365508 · Cited by 18
Optimizing Machine Translation: A Benchmarking Suite for Efficiency and Quality Enhancement
Kamuni, N., Dodda, S., Chintala, S., & Kunchakuri, N. (2024) · International Conference on Computing, Sciences and Communications (ICCSC) · doi:10.1109/iccsc62048.2024.10830335 · Cited by 14
Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., Khelladi, D. E., & Combemale, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112562 · Cited by 67
InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems
Chung, W., Cahyawijaya, S., Wilie, B., Lovenia, H., & Fung, P. (2023) · doi:10.18653/v1/2023.nlint-1.1 · Cited by 36
Self-citationThe pragmatic profile of ChatGPT: assessing the communicative skills of a conversational agentPreprint
Pietro, C. B. D. S., Frau, F., Mangiaterra, V., & Bambini, V. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ckghw · Cited by 14
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12
Self-citationTextGrad: Automatic "Differentiation" via Text
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07496 · Cited by 12
Self-citationLarge Language Models for Combinatorial Optimization: A Systematic Review
Ros, F. D., Soprano, M., Gaspero, L. D., & Roitero, K. (2026) · ACM Computing Surveys · doi:10.1145/3801961 · Cited by 40
Multimodal Neurons in Pretrained Text-Only Transformers
Schwettmann, S., Chowdhury, N., Klein, S., Bau, D., & Torralba, A. B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00308 · Cited by 18
Current opinions on large cellular models
Hao, M., Wei, L., Yang, F., Yao, J., Theodoris, C. V., Wang, B., Li, X., Yang, G., & Zhang, X. (2024) · Quantitative Biology · doi:10.1002/qub2.65 · Cited by 12
MindLLM: Lightweight large language model pre-training, evaluation and domain application
Yang, Y., Sun, H., Li, J., Liu, R., Li, Y., Liu, Y., Gao, Y., & Huang, H. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.001 · Cited by 13
Self-citationIntegrating Pre-Trained Language Model With Physical Layer Communications
Lee, J.-H., Lee, D., Lee, J., & Pujara, J. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3452481 · Cited by 14
Self-citationFodor, J., Deyne, S. D., & Suzuki, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00536 · Cited by 16
Balancing Large Language Model Alignment and Algorithmic Fidelity in Social Science Research
Lyman, A., Hepner, B., Argyle, L. P., Busby, E. C., Gubler, J. R., & Wingate, D. (2025) · Sociological Methods & Research · doi:10.1177/00491241251342008 · Cited by 22
Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding
Süzgün, M., Melas-Kyriazi, L., & Jurafsky, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.262 · Cited by 27
Are Language Models Worse than Humans at Following Prompts? It’s Complicated
Webson, A., Loo, A. M., Yu, Q., & Pavlick, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.514 · Cited by 19
Self-citationGPT as a Financial AdvisorPreprint
Niszczota, P., & Abbas, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4384861 · Cited by 12
An LLM Compiler for Parallel Function Calling
Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., & Gholami, A. (2023) · arXiv · doi:10.48550/arxiv.2312.04511 · Cited by 12
“• Movie Recommendation: A dataset with 500 examples that asks to identify the most similar movie out of four options to another set of four movies, exhibiting an 8-way embarrassingly parallel pattern [4].”
BackgroundMethodInfluentialSelf-citationAre Emergent Abilities in Large Language Models just In-Context Learning?
Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T., & Gurevych, I. (2023) · arXiv · doi:10.48550/arxiv.2309.01809 · Cited by 12
FinGPT: Large Generative Models for a Small Language
Luukkonen, R., Komulainen, V., Luoma, J., Eskelinen, A., Kanerva, J., Kupari, H.-M., Ginter, F., Laippala, V., Muennighoff, N., Piktus, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.164 · Cited by 53
Self-citationWhat is it like to be a bot? The world according to GPT-4
Lloyd, D. E. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1292675 · Cited by 15
Dove, G. O. (2024) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2023.0149 · Cited by 15
Metadata
- Authors
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A., La, A., Lampinen, A., Zou, A., Jiang, A., Chen, A., Vuong, A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabassum, A., Menezes, A., Kirubarajan, A., Mullokandov, A., Sabharwal, A., Herrick, A., Efrat, A., Erdem, A., Karakaş, A., Roberts, B. R., Loe, B. S., Zoph, B., Bojanowski, B., Özyurt, B., Hedayatnia, B., Neyshabur, B., Inden, B., Stein, B., Ekmekci, B., Lin, B. Y., Howald, B., Orinion, B., Diao, C., Dour, C., Stinson, C., Argueta, C., Ramírez, C. F., Singh, C., Rathkopf, C., Meng, C., Baral, C., Wu, C., Callison-Burch, C., Waites, C., Voigt, C., Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C., Ashcraft, C., Garbacea, C., Sileo, D., Garrette, D., Hendrycks, D., Kilman, D., Roth, D., Freeman, D., Khashabi, D., Levy, D., González, D. M., Perszyk, D., Hernandez, D., Chen, D., Ippolito, D., Gilboa, D., Dohan, D., Drakard, D., Jurgens, D., Datta, D., Ganguli, D., Emelin, D., Kleyko, D., Yuret, D., Chen, D., Tam, D., Hupkes, D., Misra, D., Buzan, D., Mollo, D. C., Yang, D., Lee, D.-H., Schrader, D., Shutova, E., Cubuk, E. D., Segal, E., Hagerman, E., Barnes, E., Donoway, E., Pavlick, E., Rodola, E., Lam, E., Chu, E., Tang, E., Erdem, E., Chang, E., Chi, E. A., Dyer, E., Jerzak, E., Kim, E., Manyasi, E. E., Zheltonozhskii, E., Xia, F., Siar, F., Martínez-Plumed, F., Happé, F., Chollet, F., Rong, F., Mishra, G., Winata, G. I., de Melo, G., Kruszewski, G., Parascandolo, G., Mariani, G., Wang, G., Jaimovitch-López, G., Betz, G., Gur-Ari, G., Galijasevic, H., Kim, H., Rashkin, H., Hajishirzi, H., Mehta, H., Bogar, H., Shevlin, H., Schütze, H., Yakura, H., Zhang, H., Wong, H. M., Ng, I., Noble, I., Jumelet, J., Geissinger, J., Kernion, J., Hilton, J., Lee, J., Fisac, J. F., Simon, J. B., Koppel, J., Zheng, J., Zou, J., Kocoń, J., Thompson, J., Wingfield, J., Kaplan, J., Radom, J., Sohl-Dickstein, J., Phang, J., Wei, J., Yosinski, J., Novikova, J., Bosscher, J., Marsh, J., Kim, J., Taal, J., Engel, J., Alabi, J., Xu, J., Song, J., Tang, J., Waweru, J., Burden, J., Miller, J., Balis, J. U., Batchelder, J., Berant, J., Frohberg, J., Rozen, J., Hernandez-Orallo, J., Boudeman, J., Guerr, J., Jones, J., Tenenbaum, J. B., Rule, J. S., Chua, J., Kanclerz, K., Livescu, K., Krauth, K., Gopalakrishnan, K., Ignatyeva, K., Markert, K., Dhole, K. D., Gimpel, K., Omondi, K., Mathewson, K., Chiafullo, K., Shkaruta, K., Shridhar, K., McDonell, K., Richardson, K., Reynolds, L., Gao, L., Zhang, L., Dugan, L., Qin, L., Contreras-Ochando, L., Morency, L.-P., Moschella, L., Lam, L., Noble, L., Schmidt, L., He, L., Colón, L. O., Metz, L., Şenel, L. K., Bosma, M., Sap, M., ter Hoeve, M., Farooqi, M., Faruqui, M., Mazeika, M., Baturan, M., Marelli, M., Maru, M., Quintana, M. J. R., Tolkiehn, M., Giulianelli, M., Lewis, M., Potthast, M., Leavitt, M. L., Hagen, M., Schubert, M., Baitemirova, M. O., Arnaud, M., McElrath, M., Yee, M. A., Cohen, M., Gu, M., Ivanitskiy, M., Starritt, M., Strube, M., Swędrowski, M., Bevilacqua, M., Yasunaga, M., Kale, M., Cain, M., Xu, M., Suzgun, M., Walker, M., Tiwari, M., Bansal, M., Aminnaseri, M., Geva, M., Gheini, M., T, M. V., Peng, N., Chi, N. A., Lee, N., Krakover, N. G.-A., Cameron, N., Roberts, N., Doiron, N., Martinez, N., Nangia, N., Deckers, N., Muennighoff, N., Keskar, N. S., Iyer, N. S., Constant, N., Fiedel, N., Wen, N., Zhang, O., Agha, O., Elbaghdadi, O., Levy, O., Evans, O., Casares, P. A. M., Doshi, P., Fung, P., Liang, P. P., Vicol, P., Alipoormolabashi, P., Liao, P., Liang, P., Chang, P., Eckersley, P., Htut, P. M., Hwang, P., Miłkowski, P., Patil, P., Pezeshkpour, P., Oli, P., Mei, Q., Lyu, Q., Chen, Q., Banjade, R., Rudolph, R. E., Gabriel, R., Habacker, R., Risco, R., Millière, R., Garg, R., Barnes, R., Saurous, R. A., Arakawa, R., Raymaekers, R., Frank, R., Sikand, R., Novak, R., Sitelew, R., LeBras, R., Liu, R., Jacobs, R., Zhang, R., Salakhutdinov, R., Chi, R., Lee, R., Stovall, R., Teehan, R., Yang, R., Singh, S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff, S., Gehrmann, S., Schuster, S., Sadeghi, S., Hamdan, S., Zhou, S., Srivastava, S., Shi, S., Singh, S., Asaadi, S., Gu, S. S., Pachchigar, S., Toshniwal, S., Upadhyay, S., Shyamolima, Debnath, Shakeri, S., Thormeyer, S., Melzi, S., Reddy, S., Makini, S. P., Lee, S.-H., Torene, S., Hatwar, S., Dehaene, S., Divic, S., Ermon, S., Biderman, S., Lin, S., Prasad, S., Piantadosi, S. T., Shieber, S. M., Misherghi, S., Kiritchenko, S., Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L., Desbordes, T., Rothschild, T., Phan, T., Wang, T., Nkinyili, T., Schick, T., Kornev, T., Tunduny, T., Gerstenberg, T., Chang, T., Neeraj, T., Khot, T., Shultz, T., Shaham, U., Misra, V., Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V., Prabhu, V. U., Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, Y., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z., Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z., & Wu, Z.
- Affiliations
- Shammie
- ORCID
- Abid, Abubakar
- Brown, Adam R.
- Gupta, Aditya
- Power, Alethea
- Warstadt, Alex
- Kocurek, Alexander W.
- Safaya, Ali
- Tazarv, Ali
- Xiang, Alice
- Parrish, Alicia
- Nie, Allen
- Dsouza, Amanda
- Rahane, Ameet
- Andreassen, Anders
- Madotto, Andrea
- Santilli, Andrea
- Dai, Andrew
- Lampinen, Andrew
- Gupta, Animesh
- Norelli, Antonio
- Venkatesh, Anu
- Mullokandov, Asher
- Erdem, Aykut
- Loe, Bao Sheng
- Bojanowski, Bartłomiej
- Inden, Benjamin
- Stein, Benno
- Howald, Blake
- Stinson, Catherine
- Singh, Chandan
- Rathkopf, Charles
- Meng, Chenlin
- Baral, Chitta
- Callison-Burch, Chris
- Voigt, Christian
- Manning, Christopher D.
- Potts, Christopher
- Siro, Clemencia
- Raffel, Colin
- Sileo, Damien
- Roth, Dan
- Khashabi, Daniel
- Chen, Danqi
- Ippolito, Daphne
- Jurgens, David
- Datta, Debajyoti
- Ganguli, Deep
- Kleyko, Denis
- Chen, Derek
- Misra, Diganta
- Mollo, Dimitri Coelho
- Yang, Diyi
- Lee, Dong-Ho
- Shutova, Ekaterina
- Cubuk, Ekin Dogus
- Segal, Elad
- Barnes, Elizabeth
- Donoway, Elizabeth
- Pavlick, Ellie
- Erdem, Erkut
- Dyer, Ethan
- Jerzak, Ethan
- Kim, Ethan
- Zheltonozhskii, Evgenii
- Mishra, Gaurav
- Betz, Gregor
- Kim, Hannah
- Hajishirzi, Hannaneh
- Mehta, Harsh
- Shevlin, Henry
- Schütze, Hinrich
- Yakura, Hiromu
- Zhang, Hongming
- Geissinger, Jack
- Lee, Jaehoon
- Fisac, Jaime Fernández
- Simon, James B.
- Koppel, James
- Zou, James
- Kocoń, Jan
- Phang, Jason
- Novikova, Jekaterina
- Kim, Jeremy
- Xu, Jiacheng
- Song, Jiaming
- Burden, John
- Rozen, Jos
- Tenenbaum, Joshua B.
- Rule, Joshua S.
- Kanclerz, Kamil
- Livescu, Karen
- Krauth, Karl
- Dhole, Kaustubh D.
- Mathewson, Kory
- Richardson, Kyle
- Zhang, Li
- Dugan, Liam
- Contreras-Ochando, Lidia
- Morency, Louis-Philippe
- Moschella, Luca
- Lam, Lucas
- Şenel, Lütfi Kerem
- Sap, Maarten
- Farooqi, Maheen
- Marelli, Marco
- Tolkiehn, Marie
- Giulianelli, Mario
- Lewis, Martha
- Potthast, Martin
- Leavitt, Matthew L.
- Hagen, Matthias
- Yee, Michael A.
- Ivanitskiy, Michael
- Strube, Michael
- Swędrowski, Michał
- Yasunaga, Michihiro
- Bansal, Mohit
- Geva, Mor
- T, Mukund Varma
- Peng, Nanyun
- Chi, Nathan A.
- Lee, Nayeon
- Deckers, Niklas
- Muennighoff, Niklas
- Keskar, Nitish Shirish
- Iyer, Niveditha S.
- Wen, Nuan
- Zhang, Oliver
- Levy, Omer
- Casares, Pablo Antonio Moreno
- Fung, Pascale
- Liang, Paul Pu
- Liao, Peiyuan
- Liang, Percy
- Miłkowski, Piotr
- Pezeshkpour, Pouya
- Oli, Priti
- Mei, Qiaozhu
- Banjade, Rabin
- Habacker, Rahel
- Millière, Raphaël
- Saurous, Rif A.
- Arakawa, Riku
- Frank, Robert
- Zhang, Rui
- Salakhutdinov, Ruslan
- Teehan, Ryan
- Singh, Sahib
- Mohammad, Saif M.
- Anand, Sajant
- Dillavou, Sam
- Wiseman, Sam
- Gruetter, Samuel
- Schoenholz, Samuel S.
- Rous, Sarah A.
- Ghosh, Sayan
- Casey, Sean
- Bischoff, Sebastian
- Gehrmann, Sebastian
- Schuster, Sebastian
- Hamdan, Shadi
- Zhou, Sharon
- Srivastava, Shashank
- Shi, Sherry
- Singh, Shikhar
- Asaadi, Shima
- Gu, Shixiang Shane
- Melzi, Simone
- Reddy, Siva
- Makini, Sneha Priscilla
- Lee, Soo-Hwan
- Dehaene, Stanislas
- Divic, Stefan
- Ermon, Stefano
- Biderman, Stella
- Lin, Stephanie
- Piantadosi, Steven T.
- Shieber, Stuart M.
- Kiritchenko, Svetlana
- Mishra, Swaroop
- Linzen, Tal
- Schuster, Tal
- Li, Tao
- Wu, Te-Lin
- Desbordes, Théo
- Wang, Tianle
- Nkinyili, Tiberius
- Tunduny, Titus
- Gerstenberg, Tobias
- Chang, Trenton
- Shaham, Uri
- Demberg, Vera
- Ramasesh, Vinay
- Prabhu, Vinay Uday
- Padmakumar, Vishakh
- Srikumar, Vivek
- Zhang, William
- Vossen, Wout
- Ren, Xiang
- Zhao, Xinran
- Wu, Xinyi
- Shen, Xudong
- Yaghoobzadeh, Yadollah
- Lakretz, Yair
- Song, Yangqiu
- Choi, Yejin
- Yang, Yichi
- Hao, Yiding
- Chen, Yifu
- Belinkov, Yonatan
- Hou, Yu
- Hou, Yufang
- Bai, Yuntao
- Zhao, Zhuoye
- Wang, Zijian
- Wang, Zijie J.
- Wu, Ziyi
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2206.04615
- OpenAlex
- W4281690148
- Semantic Scholar
- bd1331b233e84bab7eba503abc60b31ac08e7881
- Cited by
- 569
- References
- 0
- Citations by year
- 2026: 29 · 2025: 114 · 2024: 169 · 2023: 232 · 2022: 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- model scaling, sparse transformer, GPT, model calibration, social bias, commonsense reasoning, mathematical reasoning, prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
