The DOI matches its DataCite record; 527 of the 528 references it cites were found in a registry or an index.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Open accessAbstract
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even performing some traditionally symbolic tasks well with only a few training examples [61].”
From Symbols and grounding in large language models (2023) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Srivastava et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many benchmark sets do not allow for arbitrary generation of novel questions, or worse, draw data from publicly available sources-the same sources LLM trainers have access to [51, 39].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [39]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Surprisingly, we observe that using training FLOPs leads to reasonable predictions of LM performance with and without CoT on BBH tasks, possibly due to the denoising effect of aggregation over all tasks.”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In order to standardize our evaluation, we use the same prompt for all models evaluated, following the convention established by [83–85].”
From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [85]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One might thus posit, with some justification from the above-mentioned studies, that the emergence of skills arises from training on related tasks that were implicitly solved while solving next-word prediction in the training dataset.This is indeed our starting point.”
From A Theory for Emergence of Complex Skills in Language Models (2023) · cites this paper as Srivastava et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prominent among these are multi-task benchmarks such as MMLU [325] and BIG-bench [326], which evaluate knowledge and skills across diverse domains and scenarios.”
From A Survey of Large Language Models (2026) · cites this paper as [326]
The cited source says what the claim attributes to it.
In the citing paper
“To holistically assess the capabilities of LLMs, several comprehensive benchmarks have been developed [325-327].”
From A Survey of Large Language Models (2026) · cites this paper as [326]
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to prior benchmarks (e.g., SuperGLUE, 9 EleutherAI LM Harness, 10 and BIG-Bench 11 ), which enumerate a set of scenarios and metrics, we situate our scenarios in a larger taxonomy to make explicit what is currently missing.”
From Holistic Evaluation of Language Models (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“BIG-Bench Hard (BBH; Suzgun et al., 2022) is a set of 23 multiple-choice tasks from the BIG-Bench benchmark (Srivastava et al., 2022), on which current state-of-the-art models do substantially worse than the average human rater.”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“BIG-bench [182] introduces a diverse collection of 204 challenging tasks contributed by 450 authors from 132 institutions.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [182]
The cited source says what the claim attributes to it.
In the citing paper
“Popular tests, such as MMLU (137), BIG-Bench (138), BIG-Bench Hard (BBH) (139), or AGIEval (140) are typically based on human ability tasks (137) but test for knowledge recall or problem-solving (141), often by comparing performance against human experts doing the same tasks (138).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [138]
The cited source says what the claim attributes to it.
In the citing paper
“We show that when we use a format with visible lettered answer options, large models are very well calibrated on diverse multiple choice questions (e.g. from BIG Bench [Srivastava et al., 2022], MMLU[Hendrycks et al., 2021], and many other evaluations); see Figures 4, 5, and 6.”
From Language Models (Mostly) Know What They Know (2022) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Although biases already pose a challenge for conventional AI in health, they are of particular relevance for GMAI as a recent large-scale evaluation showed that social bias can increase with model scale 46 .”
From Foundation models for generalist medical artificial intelligence (2023) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Our experiments validate the effectiveness of this approach on a range of complex reasoning tasks, including several challenging Big-Bench tasks (Srivastava et al., 2022).”
From Large language models as tool makers (2023) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“We note that our evaluation is in no means comprehensive and a larger benchmarking akin to Srivastava et al. (2022) would be helpful.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Srivastava et al. (2022)
The cited source says what the claim attributes to it.
In the citing paper
“Large language models have demonstrated remarkable reasoning ability on a variety of reasoning tasks26–29.”
From Solving olympiad geometry without human demonstrations (2024) · cites this paper as [28]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Furthermore, using PC measures accurately captures the scaling trends in both setups, even when using training FLOPs leads to less tight captures in the”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 140
- Scott Alexander. A very unlikely chess game, 2020. URL https://slatestarcodex.com/2020/01/06/a-very-unlikely-chessgame/ . (cited on pp. 29 and 30)
- Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 245-256. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/alon20a.html . (cited on p. 37)
- Miriam Amin and Manuel Burghardt. A survey on approaches to computational humor generation. In Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature , pp. 29-41, Online, December 2020. International Committee on Computational Linguistics. URL https://aclanthology. org/2020.latechclfl-1.4 . (cited on p. 38)
- Brandon Amos and J. Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks, 2017. URL https: //arxiv.org/abs/1703.00443 . (cited on p. 38)
- Salvatore Attardo. Humor in language. In Oxford Research Encyclopedia of Linguistics . Oxford University Press, 2017. doi: 10. 1093/acrefore/9780199384655.013.342. URL https://oxfordre.com/linguistics/view/10.1093/acrefore/9780199384655. 001.0001/acrefore-9780199384655-e-342 . (cited on p. 38) · DOI
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: //arxiv.org/abs/2108.07732 . (cited on p. 4)
- Satanjeev Banerjee and Ted Pedersen. Extended gloss overlaps as a measure of semantic relatedness. In IJCAI'03: Proceedings of the 18th International Joint Conference on Artificial Intelligence , pp. 805-810, San Francisco, 2003. Morgan Kaufmann. doi: 10.5555/1630659.1630775. URL https://dl.acm.org/doi/10.5555/1630659.1630775 . (cited on p. 36) · DOI
- Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review , 104(3):671-732, 2016. URL http://www.jstor.org/stable/24758720 . (cited on p. 31)
- Gregor Betz, Christian Voigt, and Kyle Richardson. Critical thinking for language models, 2020. URL https://arxiv.org/abs/ 2009.07185 . (cited on p. 33)
- Stella Biderman and Edward Raff. Neural language models are effective plagiarists, 2022. URL https://arxiv.org/abs/2201. 07406 . (cited on p. 4)
- Julia Birke and Anoop Sarkar. A clustering approach for nearly unsupervised recognition of nonliteral language. In 11th Conference of the European Chapter of the Association for Computational Linguistics , pp. 329-336, Trento, Italy, April 2006. Association for Computational Linguistics. URL https://aclanthology.org/E06-1042 . (cited on p. 35)
- Nicholas Boillot. Vector forms as a foreign language, 24 June 2019. URL https://www.fluate.net/en/travaux/vectoglyph . (cited on pp. 31 and 33)
- Paul F. Boller, Jr. and John George. They Never Said It: A Book of Fake Quotes, Misquotes, and Misleading Attributions . Oxford University Press, Oxford, 1989. (cited on p. 35)
- Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings. neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html . (cited on pp. 17, 29, 31, and 33)
- Bozhidar Bozhanov and Ivan Derzhanski. Rosetta stone linguistic problems. In Proceedings of the Fourth Workshop on Teaching NLP and CL , pp. 1-8, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https: //aclanthology.org/W13-3401 . (cited on p. 34)
- Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable Forth interpreter. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 547-556, 2017. URL https: //arxiv.org/abs/1605.06640 . (cited on p. 34)
- Gwern Branwen. GPT-3 creative fiction. Gwern.net , June 2020. URL https://www.gwern.net/GPT-3 . (cited on pp. 19, 38, and 39)
- Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review , 78(1):1-3, 1950. (cited on pp. 8 and 11)
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html . (cited on pp. 4, 7, 8, 10, 11, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40, and 41)
- Corrado Böhm. On a family of Turing machines and the related programming language. ICC Bulletin , 3:187-194, 1964. (cited on p. 38)
- Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pp. 2633-2650. USENIX Association, August 2021. URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting . (cited on p. 19)
- Nathanael Chambers. Labeling documents with timestamps: Learning from their time expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 98-106, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1011 . (cited on p. 31)
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 1691-1703. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/chen20s.html . (cited on p. 29)
- Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. https://openreview.net/pdf?id= H1gfOiAqYm , 2019b. (cited on p. 34)
- Kate Crawford. The trouble with bias. https://www.youtube.com/watch?v=fMym\_BKWQzk , 2017. Keynote address, NIPS 2017, Long Beach CA. Dec. 5, 2017. (cited on p. 29)
- Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 933-941, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305478. URL https://dl.acm.org/doi/10.5555/3305381. 3305478 . (cited on p. 10) · DOI
- Wayne Davis. Implicature. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Fall 2019 edition, 2019. (cited on p. 33)
- Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. Finding contradictions in text. In Proceedings of ACL-08: HLT , pp. 1039-1047, Columbus, Ohio, June 2008. Association for Computational Linguistics. URL https: //aclanthology.org/P08-1118 . (cited on p. 32)
- Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. RobustFill: Neural program learning under noisy I/O. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 990-998, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305484. URL https://dl.acm.org/doi/10.5555/3305381.3305484 . (cited on p. 34) · DOI
- Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. Beyond English-centric multilingual machine translation. Journal of Machine Learning Research , 22(107): 1-48, 2021. URL http://jmlr.org/papers/v22/20-1307.html . (cited on p. 20)
- Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database . MIT Press, Cambridge, MA, 1998. (cited on p. 33)
- Jerry A. Fodor. The Language of Thought . Harvard University Press, Cambridge, MA, 1975. (cited on p. 30)
- Evgeniy Gabrilovich and Shaul Markovitch. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In IJCAI'07: Proceedings of the 20th International Joint Conference on Artifical Intelligence , pp. 1606-1611, San Francisco, 2007. Morgan Kaufmann. doi: 10.5555/1625275.1625535. URL https://dl.acm.org/doi/10.5555/1625275.1625535 . (cited on p. 36) · DOI
- Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics , 9: 346-361, 04 2021. doi: 10.1162/tacl\_a\_00370. URL https://doi.org/10.1162/tacl\_a\_00370 . (cited on pp. 29 and 39) · DOI
- Arthur S. Goldberger. Structural equation methods in the social sciences. Econometrica , 40(6):979-1001, 1972. URL http://www.jstor.org/stable/1913851 . (cited on p. 32)
- Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pp. 581-586, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https://aclanthology.org/P11-2102 . (cited on p. 39)
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pp. 1321-1330. PMLR, 06-11 Aug. 2017. URL https://proceedings.mlr.press/v70/guo17a. html . (cited on p. 11)
- Samuel Gyasi Obeng. The proverb as a mitigating and politeness strategy in Akan discourse. Anthropological Linguistics , 38(3): 521-549, 1996. URL http://www.jstor.org/stable/30028601 . (cited on p. 32)
- Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171, Dec. 2020. doi: 10.1162/tacl\_a\_00306. URL https://doi.org/10.1162/tacl\_a\_00306 . (cited on p. 31) · DOI
- Joseph Y. Halpern. Actual causality . MIT Press, Cambridge, MA, 2016. (cited on p. 29)
- Maria Hanzén. When in Rome, do as the Romans do: Proverbs as a part of EFL teaching. Master's thesis, Jönköping University, School of Education and Communication, Jönköping, 2007. URL http://www.diva-portal.org/smash/get/diva2: 3499/fulltext01.pdf . (cited on p. 40)
- Irene Heim. On the projection problem for presuppositions. In Paul Portner and Barbara H. Partee (eds.), Formal Semantics The Essential Readings , pp. 249-260. Blackwell, Oxford, 1983. (cited on p. 37)
- Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Hkg4TI9xl . (cited on pp. 11 and 12)
- Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 2712-2721. PMLR, 09-15 June 2019. URL https: //proceedings.mlr.press/v97/hendrycks19a.html . (cited on p. 11)
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS, 2021a. URL https: //arxiv.org/abs/2105.09938 . (cited on pp. 4, 37, and 41)
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021b. URL https: //openreview.net/forum?id=d7KBjmI3GmQ . (cited on pp. 25, 31, 35, 38, and 40)
- Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips. cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html . (cited on pp. 5 and 32)
- Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URL https: //arxiv.org/abs/2102.01293 . (cited on p. 41)
- Keith J. Holyoak. Analogy and relational reasoning. In Keith J. Holyoak and Robert G. Morrison (eds.), The Oxford Handbook of Thinking and Reasoning . Oxford University Press, Oxford, 2012. URL https://www.oxfordhandbooks.com/view/10.1093/ oxfordhb/9780199734689.001.0001/oxfordhb-9780199734689-e-13 . (cited on p. 28)
- Yufang Hou. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 1428-1438, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.132. URL https://aclanthology.org/2020.acl-main.132 . (cited on p. 29)
- Yufang Hou, Katja Markert, and Michael Strube. Global inference for bridging anaphora resolution. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 907-917, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology.org/N131111 . (cited on p. 29)
- Hrisztalina Hrisztova-Gotthardt and Melita Aleksa Varga. Introduction to Paremiology: A Comprehensive Guide to Proverb Studies . De Gruyter Open, Warsaw, 2015. URL https://www.degruyter.com/document/doi/10.2478/9783110410167/html . (cited on p. 32) · DOI
- Thad Hughes and Daniel Ramage. Lexical semantic relatedness with random graph walks. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , pp. 581-589, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/D07-1061 . (cited on p. 36)
- Jay J. Jiang and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th Research on Computational Linguistics International Conference , pp. 19-33, Taipei, Taiwan, August 1997. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). URL https://aclanthology.org/O97-1002 . (cited on p. 36)
- Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438, 2020. doi: 10.1162/tacl\_a\_00324. URL https://doi.org/10. 1162/tacl\_a\_00324 . (cited on pp. 11 and 12) · DOI
- Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS'15 , volume 1, pp. 190-198, Cambridge, MA, USA, 2015. MIT Press. doi: 10.5555/2969239.2969261. URL https://dl.acm.org/doi/10.5555/2969239.2969261 . (cited on p. 34) · DOI
- Lauri Karttunen. Simple and phrasal implicatives. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pp. 124-131, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1020 . (cited on p. 32)
- Faisal Khan, Bilge Mutlu, and Jerry Zhu. How do humans teach: On curriculum learning and teaching dimension. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper/2011/file/ f9028faec74be6ec9b852b0a542e2f39-Paper.pdf . (cited on p. 38)
- Milton King and Paul Cook. Evaluating approaches to personalizing language models. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 2461-2469, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.299 . (cited on p. 38)
- Christo Kirov and Ryan Cotterell. Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate. Transactions of the Association for Computational Linguistics , 6:651-665, 12 2018. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00247. URL https://doi.org/10.1162/tacl\_a\_00247 . (cited on p. 34) · DOI
- Joshua Knobe. Intentional action and side effects in ordinary language. Analysis , 63, 07 2003. doi: 10.1111/ 1467-8284.00419. URL https://www.researchgate.net/publication/28763794\_Intentional\_Action\_and\_Side\_Effects\_ in\_Ordinary\_Language . (cited on p. 29)
- Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics , 6:317-328, 2018. doi: 10.1162/tacl\_a\_00023. URL https://aclanthology.org/Q18-1023 . (cited on p. 35) · DOI
- Alexander W. Kocurek, Ethan Jerzak, and Rachel Etta Rudolph. Against conventional wisdom. Philosophers' Imprint , 20(22): 1-27, 2020. URL http://hdl.handle.net/2027/spo.3521354.0020.022 . (cited on p. 29)
- Jarmo Korhonen. Sprichwörter und zweisprachige lexikographie: Deutsch-schwedische und deutsch-finnische wörtebücher im vergleich. In C. Földes (ed.), Phraseologie disziplinär und interdisziplinär , pp. 537-549. Gunter Narr Verlag, 2009. (cited on p. 40)
- Mahnaz Koupaee and William Yang Wang. WikiHow: A large scale text summarization dataset, 2018. URL https://arxiv. org/abs/1810.09305 . (cited on p. 5)
- Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics , 10:50-72, 01 2022. doi: 10.1162/tacl\_a\_00447. URL https://doi.org/10.1162/tacl\_a\_00447 . (cited on p. 20) · DOI
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 08 2019. doi: 10.1162/tacl\_a\_00276. URL https://doi.org/10.1162/tacl\_a\_00276 . (cited on pp. 5 and 37) · DOI
- Kevin Lacker. Giving GPT-3 a Turing test. Kevin Lacker's blog , July 2020. URL https://lacker.io/ai/2020/07/06/givinggpt-3-a-turing-test.html . (cited on pp. 33 and 41)
- George Lakoff and Mark Johnson. Metaphors We Live By . University of Chicago Press, Chicago, 2008. (cited on pp. 33 and 35)
- Guillaume Lample and François Charton. Deep learning for symbolic mathematics, 2019. URL https://arxiv.org/abs/1912. 01412 . (cited on p. 38)
- Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 986-995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL https://aclanthology.org/I17-1099 . (cited on p. 40)
- Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics , 4:521-535, 12 2016. doi: 10.1162/tacl\_a\_00115. URL https://doi.org/10.1162/tacl\_a\_00115 . (cited on p. 39) · DOI
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019. URL https://arxiv.org/abs/1907. 11692 . (cited on pp. 39 and 41)
- Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. In Vivek Nigam, Tajana Ban Kirigin, Carolyn Talcott, Joshua Guttman, Stepan Kuznetsov, Boon Thau Loo, and Mitsuhiro Okada (eds.), Logic, Language, and Security . Springer, Cham, 2020. URL https://www.springerprofessional. de/en/gender-bias-in-neural-natural-language-processing/18531692 . (cited on p. 33)
- Jihang Mao and Wanli Liu. A BERT-based approach for automatic humor detection and scoring. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019) , pp. 197-202, 2019. URL http://ceur-ws.org/Vol-2421/HAHA\_paper\_8.pdf . (cited on p. 39)
- Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence, 2020. URL https://arxiv.org/abs/ 2002.06177 . (cited on p. 30)
- Katja Markert, Yufang Hou, and Michael Strube. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 795-804, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1084 . (cited on p. 29)
- R. Thomas McCoy, Robert Frank, and Tal Linzen. Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks. Transactions of the Association for Computational Linguistics , 8:125-140, 01 2020. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00304. URL https://doi.org/10.1162/tacl\_a\_00304 . (cited on p. 34) · DOI
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https: //arxiv.org/abs/1609.07843 . (cited on p. 31)
- John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 6905-6916. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/miller20a.html . (cited on p. 39)
- David Milne and Ian H. Witten. An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: An Evolving Synergy , pp. 25-30, Menlo Park, 2008. Association for the Advancement of Artificial Intelligence. URL https://www.aaai.org/Papers/Workshops/2008/WS08-15/WS08-15-005.pdf . (cited on p. 36)
- Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Natural reference to objects in a visual domain. In Proceedings of the 6th International Natural Language Generation Conference . Association for Computational Linguistics, July 2010. URL https://aclanthology.org/W10-4210 . (cited on p. 38)
- Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Generating expressions that refer to visible objects. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 1174-1184, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https:// aclanthology.org/N13-1137 . (cited on p. 38)
- Elham Mohammadi, Hessam Amini, and Leila Kosseim. CLaC at CLPsych 2019: Fusion of neural features and predicted class probabilities for suicide risk assessment based on online posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology , pp. 34-38, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-3004. URL https://aclanthology.org/W19-3004 . (cited on p. 39) · DOI
- Jane Morris and Graeme Hirst. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics , 17(1):21-48, 1991. URL https://aclanthology.org/J91-1002 . (cited on p. 36)
- Karl Mulligan, Robert Frank, and Tal Linzen. Structure here, bias there: Hierarchical generalization by jointly learning syntactic transformations. In Proceedings of the Society for Computation in Linguistics 2021 , pp. 125-135, Online, February 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.scil-1.12 . (cited on p. 34)
- Gregory L. Murphy. Comprehending complex concepts. Cognitive Science , 12(4):529-562, 1988. doi: https://doi.org/10.1207/ s15516709cog1204\_2. URL https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1204\_2 . (cited on p. 30) · DOI
- Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 8558cb408c1d76621371888657d2eb1d-Abstract.html . (cited on p. 11)
- Anthony M. Paul. Figurative language. Philosophy & Rhetoric , 3(4):225-248, 1970. URL http://www.jstor.org/stable/ 40237206 . (cited on p. 35)
- Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann, San Francisco, 1988. (cited on p. 32)
- Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, Cambridge, 2000. (cited on p. 30)
- Tony A. Plate. Distributed representations and nested compositional structure . PhD thesis, University of Toronto, Toronto, 1994. (cited on p. 29)
- Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving, 2020. URL https://arxiv. org/abs/2009.03393 . (cited on pp. 34 and 38)
- Willard V.O. Quine. Main trends in recent philosophy: Two dogmas of empiricism. The Philosophical Review , 60(1):20-43, 1951. URL http://www.jstor.org/stable/2181906 . (cited on p. 29)
- Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/languagemodels.pdf . (cited on pp. 10, 30, 33, 35, 37, 38, and 39)
- Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pp. 777-789, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology. org/D12-1071 . (cited on p. 41)
- Ian Ravenscroft. Folk psychology as a theory. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Summer 2019 edition, 2019. (cited on p. 32)
- Joseph Reisinger and Raymond J. Mooney. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 109-117, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology. org/N10-1013 . (cited on p. 36)
- Philip Resnik. Using information content to evaluate semantic similarity in a taxonomy. In IJCAI'95: Proceedings of the 14th International Joint Conference on Artificial Intelligence, Volume 1 , pp. 448-453, San Francisco, 1995. Morgan Kaufmann. doi: 10.5555/1625855.1625914. URL https://dl.acm.org/doi/10.5555/1625855.1625914 . (cited on p. 36) · DOI
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. AAAI Spring Symposium , 2011. URL http://commonsensereasoning.org/2011/papers/Roemmele.pdf . (cited on p. 32)
- Joshua S. Rule. The child as hacker: Building more human-like models of learning . PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2020. URL https://hdl.handle.net/1721.1/129232 . (cited on p. 34)
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 . (cited on pp. 9 and 25)
- Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 624-643, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.52. URL https://aclanthology.org/2021.naacl-main.52 . (cited on p. 41) · DOI
- Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021b. URL https://openreview.net/ forum?id=fe\_hCc4RBrg . (cited on p. 4)
- Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. Diagram understanding in geometry questions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, Menlo Park, CA, Jun. 2014. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9146 . (cited on p. 34)
- Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL http: //arxiv.org/abs/1804.04235 . (cited on p. 43)
- Ekaterina Shutova. Automatic metaphor interpretation as a paraphrasing task. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 1029-1037, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology.org/N10-1147 . (cited on p. 35)
- Ekaterina Shutova and Simone Teufel. Metaphor corpus annotated for source-target domain mappings. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) , Valletta, Malta, May 2010. European Language Resources Association. URL http://www.lrec-conf.org/proceedings/lrec2010/pdf/612\_Paper.pdf . (cited on p. 35)
- Damien Sileo, Tim Van de Cruys, Camille Pradel, and Philippe Muller. DiscSense: Automated semantic analysis of discourse markers. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 991-999, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.125 . (cited on p. 31)
- Gurdeep Singh, Kaustubh Dhole, Priyadarshini P. Pai, and Sukanta Mondal. SPRINGS: Prediction of protein-protein interaction sites using artificial neural networks. Journal of Proteomics & Computational Biology , 1:7, 2014. URL https: //www.avensonline.org/fulltextarticles/JPCB-2572-8679-01-0001.html . (cited on p. 37)
- Douglas R. Smith. The synthesis of LISP programs from examples: A survey. In Alan W. Biermann, Gerhard Guiho, and Yves Kodratoff (eds.), Automatic Program Construction Techniques , pp. 307-324. Macmillan, New York, 1984. (cited on p. 34)
- Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences , 11(1):1-23, 1988. doi: 10.1017/ S0140525X00052432. (cited on p. 30)
- Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://aclanthology.org/D13-1170 . (cited on p. 31)
- Shashank Srivastava, Snigdha Chaturvedi, and Tom Mitchell. Inferring interpersonal relations in narrative summaries. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16 , pp. 2807-2813, Menlo Park, CA, 2016. Association for the Advancement of Artificial Intelligence. doi: 10.5555/3016100.3016294. URL https://dl.acm.org/doi/10. 5555/3016100.3016294 . (cited on p. 34) · DOI
- Robert Stalnaker. Assertion. In P. Cole (ed.), Pragmatics , Syntax and Semantics 9, pp. 315-332. Brill, Leiden, 1978. (cited on p. 32)
- Bernd Steinbach and Roman Kohut. Neural networks - a model of boolean functions. 5th International Workshop on Boolean Problems, Freiburg, Sept. 2002. , 2002. URL https://www.researchgate.net/publication/246931125\_Neural\_Networks\_-\_A\_Model\_of\_Boolean\_Functions . (cited on p. 29)
- Andreas Stöckl. Watching a language model learning chess. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) , pp. 1369-1379, Held Online, September 2021. INCOMA Ltd. URL https://aclanthology.org/2021.ranlp-1.153 . (cited on p. 4)
- Kevin Stowe, Leonardo Ribeiro, and Iryna Gurevych. Metaphoric paraphrase generation, 2020. URL https://arxiv.org/abs/ 2002.12854 . (cited on p. 35)
- Michael Strube and Simone Paolo Ponzetto. Wikirelate! Computing semantic relatedness using Wikipedia. In AAAI'06: Proceedings of the 21st National Conference on Artificial Intelligence , volume 2, pp. 1419-1424. Association for the Advancement of Artificial Intelligence, 2006. doi: 10.5555/1597348.1597414. URL https://dl.acm.org/doi/10.5555/1597348. 1597414 . (cited on p. 36) · DOI
- Saku Sugawara, Hikaru Yokono, and Akiko Aizawa. Prerequisite skills for reading comprehension: Multi-perspective analysis of MCTest datasets and systems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, Menlo Park, CA, Feb. 2017. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/ view/10957 . (cited on p. 35)
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks, 2014. URL https: //arxiv.org/abs/1409.3215 . (cited on p. 4)
- Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Tunuguntla Deepak, and Oskar van der Wal. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models , 2022. URL https://openreview.net/forum?id=rK-7NhfSIW5 . (cited on pp. 18 and 20)
- Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. Learning to recommend quotes for writing. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15 , pp. 2453-2459, Menlo Park, CA, 2015. Association for the Advancement of Artificial Intelligence. URL https://dl.acm.org/doi/10.5555/2886521.2886662 . (cited on p. 28) · DOI
- Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15 , pp. 1923-1929, 2015. URL https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/view/10957/10931 . (cited on p. 38)
- George Tsatsaronis, Iraklis Varlamis, Michalis Vazirgiannis, and Kjetil Nørvåg. Omiotis: A thesaurus-based measure of text relatedness. In Wray Buntine, Marko Grobelnik, Dunja Mladenić, and John Shawe-Taylor (eds.), Machine Learning and Knowledge Discovery in Databases , pp. 742-745, Berlin, 2009. Springer. (cited on p. 36)
- Shikhar Vashishth, Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Dating documents using graph convolution networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1605-1615, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ v1/P18-1149. URL https://aclanthology.org/P18-1149 . (cited on p. 31)
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 30, pp. 5998--6008. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf . (cited on pp. 10 and 38)
- Jette Viethen and Robert Dale. The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference , pp. 59-67, Salt Fork, Ohio, USA, June 2008. Association for Computational Linguistics. URL https://aclanthology.org/W08-1109 . (cited on p. 38)
- Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers , pp. 176-187, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-1017 . (cited on p. 35)
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019a. URL https://proceedings.neurips.cc/paper/2019/hash/ 4496bf24afe7fab6f046bf4923da8de6-Abstract.html . (cited on pp. 5, 7, 29, 32, and 41)
- David Wechsler. Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) . Pearson, San Antonio, 2008. (cited on p. 38)
- Ludwig Wittgenstein. Philosophical investigations . Basil Blackwell, Oxford, 1953. (cited on p. 29)
- Shijie Wu, Ryan Cotterell, and Mans Hulden. Applying the transformer to character-level transduction, 2020. URL https: //arxiv.org/abs/2005.10213 . (cited on p. 41)
- Scott Cheng-Hsin Yang and Patrick Shafto. Explainable artificial intelligence via Bayesian teaching. Workshop on Teaching Machines, Robots, and Humans, NIPS 2017 , 2017. URL http://shaftolab.com/assets/papers/yangShafto\_NIPS\_2017\_ machine\_teaching.pdf . (cited on p. 38)
- Eric Yeh, Daniel Ramage, Christopher D. Manning, Eneko Agirre, and Aitor Soroa. WikiWalk: Random walks on Wikipedia for semantic relatedness. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4) , pp. 41-49, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https: //aclanthology.org/W09-3206 . (cited on p. 36)
- Wenpeng Yin and Yulong Pei. Optimizing sentence modeling and selection for document summarization. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI-15 , pp. 1383-1389, 2015. doi: 10.5555/2832415.2832442. URL https://dl.acm.org/doi/10.5555/2832415.2832442 . (cited on p. 32) · DOI
- Eliezer Yudkowsky. Artificial intelligence as a positive and negative factor in global risk. In Nick Bostrom and Milan M. Ćirković (eds.), Global Catastrophic Risks , pp. 308-345. Oxford University Press, Oxford, 2008. URL https://web.archive.org/web/ 20210125025955/https://intelligence.org/files/AIPosNegFactor.pdf . (cited on p. 38)
- Omnia Zayed, John Philip McCrae, and Paul Buitelaar. Figure me out: A gold standard dataset for metaphor interpretation. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 5810-5819, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.712 . (cited on p. 35)
- Meishan Zhang, Yue Zhang, and Guohong Fu. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pp. 2449-2460, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https://aclanthology.org/C16-1231 . (cited on p. 39)
- Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, Menlo Park, CA, Mar. 2015. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9761 . (cited on p. 38)
Cited by
Sorted by citations
Items 301–400 of 681
On Sarcasm Detection with OpenAI GPT-Based Models
Gole, M., Nwadiugwu, W., & Miranskyy, A. V. (2024) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon62161.2024.10837875 · Cited by 21
Large Language Model in Materials Science: Roles, Challenges, and Strategic Outlook
Zhang, J., Chen, X., Ye, X., Yang, Y., & Ai, B. (2025) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500085 · Cited by 17
Self-citationHow Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench
Ye, Q., Fu, H. Y., Ren, X., & Jia, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.503 · Cited by 28
Self-citationGenerative AI from Theory to Practice: A Case Study of Financial Advice
Lo, A. W., & Ross, J. (2024) · doi:10.21428/e4baedd9.a1f6a281 · Cited by 25
Task Compass: Scaling Multi-task Pre-training with Task Prefix
Zhang, Z., Wang, S., Yi‐chong, X., Fang, Y., Yu, W., Liu, Y., Zhao, H. Y., Zhu, C., & Zeng, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.416 · Cited by 17
LongHealth: A Question Answering Benchmark with Long Clinical Documents
Adams, L., Busch, F., Han, T., Excoffier, J.-B., Ortala, M., Löser, A., Aerts, H. J., Kather, J. N., Truhn, D., & Bressem, K. (2024) · arXiv · doi:10.48550/arxiv.2401.14490 · Cited by 10
MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
Sileo, D., & Lernould, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.303 · Cited by 22
Self-citationEvaluating Synthetic Data Generation from User Generated Text
Chim, J., Ive, J., & Liakata, M. (2024) · Computational Linguistics · doi:10.1162/coli_a_00540 · Cited by 12
ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Language
Piau, M., Lotufo, R., & Nogueira, R. F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-79032-4_23 · Cited by 14
Human vs. Machine: Comparing AI-Generated and Human-Written Psychological Reports
Lockwood, A. B., Farmer, R. L., Shergill, G., Benson, N. F., & Gilbert, K. (2025) · Journal of Psychoeducational Assessment · doi:10.1177/07342829251346623 · Cited by 13
Modeling Uncertainty in Personalized Emotion Prediction with Normalizing Flows
Miłkowski, P., Karanowski, K., Wielopolski, P., Kocoń, J., Kazienko, P., & Zięba, M. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00103 · Cited by 10
Self-citationExploring Universal Intrinsic Task Subspace for Few-Shot Learning via Prompt Tuning
Qin, Y., Wang, X., Su, Y., Lin, Y., Ding, N., Yi, J., Chen, W., Liu, Z., Li, J., Hou, L., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3430545 · Cited by 72
Taking Advice from ChatGPTPreprint
Zhang, P. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/b53vn · Cited by 24
Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation
Yang, Z., Li, P., & Liu, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.109 · Cited by 24
The Antecedents of Transformer Models
Dennis, S. J., Shabahang, K. D., & Yim, H. (2024) · Current Directions in Psychological Science · doi:10.1177/09637214241279504 · Cited by 11
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 10
Why did the human brain size evolve? A way forward
González-Forero, M., & Gómez‐Robles, A. (2025) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2024.0114 · Cited by 9
From Prompt to Drug: Toward Pharmaceutical Superintelligence
Zhavoronkov, A. A., Gennert, D. G., & Shi, J. (2026) · ACS Central Science · doi:10.1021/acscentsci.5c01473 · Cited by 15
GlobalBench: A Benchmark for Global Progress in Natural Language Processing
Song, Y., Khanuja, S., Liu, P., Faisal, F., Ostapenko, A., Winata, G. I., Aji, A. F., Cahyawijaya, S., Tsvetkov, Y., Anastasopoulos, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.875 · Cited by 9
Self-citationScaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments
Ivgi, M., Carmon, Y., & Berant, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.544 · Cited by 17
Self-citationMultilingual Conceptual Coverage in Text-to-Image Models
Saxon, M., & Wang, W. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.266 · Cited by 14
GPT Struct Me: Probing GPT Models on Narrative Entity Extraction
Sousa, H., Guimar�ães, N., Jorge, A., & Campos, R. (2023) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat59888.2023.00063 · Cited by 11
SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research
Antypas, D., Ushio, A., Barbieri, F., Neves, L., Rezaee, K., Espinosa-Anke, L., Pei, J., & Camacho-Collados, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.838 · Cited by 11
PALS: Personalized Active Learning for Subjective Tasks in NLP
Kanclerz, K., Karanowski, K., Bielaniewicz, J., Gruza, M., Miłkowski, P., Kocoń, J., & Kazienko, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.823 · Cited by 9
Self-citationSelf-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations
Chen, W., Wu, C.-K., Chen, Y.-N., & Chen, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.968 · Cited by 33
Self-citationDeep Emotions Across Languages: A Novel Approach for Sentiment Propagation in Multilingual WordNets
Kocoń, J. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00101 · Cited by 10
Self-citationGenerating Meaning: Active Inference and the Scope and Limits of Passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2023) · doi:10.31234/osf.io/8xgzv · Cited by 8
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8
“Ye et al. (2023) considered reducing the number of tasks in Big-bench (Srivastava et al., 2022).”
BackgroundLLMs beyond the lab: the ethics and epistemics of real-world AI research
Mollen, J. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09819-w · Cited by 11
Dang, P., Zhu, J., Dang, C., & Zhang, H. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106399 · Cited by 9
Self-citationJiao, Y., Han, J., & Huang, C. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00982-0 · Cited by 9
Benchmarking Large Language Model Capabilities for Conditional Generation
Maynez, J., Agrawal, P., & Gehrmann, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.511 · Cited by 21
Self-citationPOE: Process of Elimination for Multiple Choice Reasoning
Ma, C., & Du, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.273 · Cited by 11
Granato, G., & Baldassarre, G. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106292 · Cited by 10
SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models
Nagireddy, M., Chiazor, L., Singh, M., & Baldini, I. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30142 · Cited by 8
Do large language models know chemistry?Preprint
White, A. D., Hocky, G. M., Gandhi, H. A., Ansari, M., Cox, S., Wellawatte, G. P., Sasmal, S., Yang, Z., Liu, K., Singh, Y., et al. (2022) · ChemRxiv · doi:10.26434/chemrxiv-2022-3md3n · Cited by 8
GPT 4 : The Ultimate BrainPreprint
Adesso, G. (2022) · doi:10.22541/au.167052124.41804127/v1 · Cited by 8
Ho-tin, W., & Yip, G.-L. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/wkb32 · Cited by 8
CLadder: Assessing Causal Reasoning in Language Models
Jin, Z., Chen, Y., Leeb, F., Gresele, L., Kamal, O., Lyu, Z., Blin, K., Adauto, F. G., Kleiman-Weiner, M., Sachan, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.04350 · Cited by 8
Self-citationCoelho Mollo, D., & Millière, R. (2026) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2026.12307 · Cited by 48
Self-citationInterpreting Metaphorical Language: A Challenge to Artificial Intelligence
Skrynnikova, I. (2024) · Vestnik Volgogradskogo gosudarstvennogo universiteta Serija 2 Jazykoznanije · doi:10.15688/jvolsu2.2024.5.8 · Cited by 7
Cao, H., Ma, R., Zhai, Y., & Shen, J. (2024) · Applied Computing and Intelligence · doi:10.3934/aci.2024019 · Cited by 8
Soliman, G., Zaki, H., & Kilany, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100131 · Cited by 13
Zhang, F., Luo, Y., Gao, Z., & Han, A. (2025) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2025.102070 · Cited by 8
Schleifer, A. G., Klebanov, B. B., & Alexandron, G. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00485-7 · Cited by 7
GRACE: Discriminator-Guided Chain-of-Thought Reasoning
Khalifa, M., Logeswaran, L., Lee, M., Lee, H., & Wang, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1022 · Cited by 31
Li, R., Wang, Y., Wen, Z., Cui, M., & Miao, Q. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112789 · Cited by 10
Continued Pretraining for Better Zero- and Few-Shot Promptability
Wu, Z., Logan, R. L., Walsh, P., Bhagia, A., Groeneveld, D., Singh, S., & Beltagy, I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.300 · Cited by 9
Self-citationLearning from Mistakes via Cooperative Study Assistant for Large Language Models
Wang, D., & Li, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.659 · Cited by 17
The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study
Mouton, C., Lucas, C., & Guest, E. (2024) · RAND Corporation eBooks · doi:10.7249/rra2977-2 · Cited by 7
Evaluation of Medium-large Language Models at Zero-shot Closed Book Generative Question Answering
Peinl, R., & Wirth, J. (2024) · doi:10.5121/csit.2024.140106 · Cited by 8
An astronomical question answering dataset for evaluating large language models
Li, J., Zhao, F., Chen, P., Xie, J., Zhang, X., Li, H., Chen, M., Wang, Y., & Zhu, M. (2025) · Scientific Data · doi:10.1038/s41597-025-04613-9 · Cited by 10
clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents
Chalamalasetti, K., Götze, J., Hakimov, S., Madureira, B., Sadler, P., & Schlangen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.689 · Cited by 9
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6
Self-citationDissociating Artificial Intelligence from Artificial Consciousness
Findlay, G., Marshall, W., Albantakis, L., David, I., Mayner, W. G., Koch, C., & Tononi, G. (2024) · arXiv · doi:10.48550/arxiv.2412.04571 · Cited by 41
Data Similarity is Not Enough to Explain Language Model Performance
Yauney, G., Reif, E., & Mimno, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.695 · Cited by 13
Maslinski, J., Grasfield, R. H., Awasthi, R., Mishra, S. V., Mahapatra, D., & Mathur, P. (2025) · Cureus · doi:10.7759/cureus.82397 · Cited by 6
Self-citationSemantic Search of FDA Guidance Documents Using Generative AI
Proestel, S., Jeng, L. J. B., Smith, C., Deady, M., Amer, O., Ahmed, M. A., & Rodgers, S. (2025) · Therapeutic Innovation & Regulatory Science · doi:10.1007/s43441-025-00798-8 · Cited by 6
Large Language Models as SocioTechnical Systems
Dhole, K. (2023) · doi:10.18653/v1/2023.bigpicture-1.6 · Cited by 7
Self-citationRILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation
Yang, Z., Lin, J., Chen, P., Cherian, A., Marks, T. K., Roux, J. L., & Gan, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01538 · Cited by 8
Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models
Waldis, A., Waldis, A., Perlitz, Y., Choshen, L., Choshen, L., Hou, Y., & Gurevych, I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00718 · Cited by 23
Self-citationAstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy
Pan, R., Nguyen, T. D., Arora, H., Accomazzi, A., Ghosal, T., & Ting, Y.-S. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00019 · Cited by 10
Martínez, J., Llinás, B., Botello, J. G., Padilla, J. J., & Frydenlund, E. F. (2024) · Winter Simulation Conference · doi:10.1109/wsc63780.2024.10838967 · Cited by 9
XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations
Zhang, Y., Wang, J., Wang, Z., & Zhang, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.887 · Cited by 6
Self-citationBeyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models
Zhang, Y., Yasunaga, M., Zhou, Z., HaoChen, J. Z., Zou, J., Liang, P., & Yeung, S. Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.472 · Cited by 17
Self-citationA Universal Discriminator for Zero-Shot Generalization
Xu, H., Lin, Z., Zhou, J., Zheng, Y., & Yang, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.589 · Cited by 12
LLM-driven Instruction Following: Progresses and Concerns
Yin, W., Ye, Q., Liu, P., Ren, X., & Schütze, H. (2023) · doi:10.18653/v1/2023.emnlp-tutorial.4 · Cited by 8
Self-citationAttanasio, G., Plaza-del-Arco, F. M., Nozza, D., & Lauscher, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.243 · Cited by 8
Li, S., Zhou, G., Li, Z., Lu, J., & Huang, N. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00032 · Cited by 7
Joint Prediction and Denoising for Large-Scale Multilingual Self-Supervised Learning
Chen, W., Shi, J., Yan, B., Berrebbi, D., Zhang, W., Peng, Y., Chang, X., Maiti, S., & Watanabe, S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389735 · Cited by 7
Self-citationRelational Programming with Foundational Models
Li, Z., Huang, J., Liu, J., Zhu, F., Zhao, E., Dodds, W., Velingker, N., Alur, R., & Naik, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28934 · Cited by 14
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0005 · Cited by 6
Bridging Information-Theoretic and Geometric Compression in Language Models
Cheng, E., Kervadec, C., & Baroni, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.762 · Cited by 17
Systematic Human Learning and Generalization From a Brief Tutorial With Explanatory Feedback
Nam, A., & McClelland, J. L. (2024) · Open Mind · doi:10.1162/opmi_a_00123 · Cited by 12
Complex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 32
How Ethical Should AI Be? How AI Alignment Shapes the Risk Preferences of LLMsPreprint
Ouyang, S., Yun, H., & Zheng, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4851711 · Cited by 6
Mei, Y., & Duan, Y. (2025) · Applied Sciences · doi:10.3390/app15031103 · Cited by 5
ElecBench: A large language model benchmark in electric power domain
Zhang, S., Huang, Q., Zhang, Q., Liang, X. M., Liu, W., Gao, K., Zhou, F., & Shi, C. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112310 · Cited by 7
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark
Hardalov, M., Atanasova, P., Mihaylov, T., Angelova, G., Simov, K., Osenova, P., Stoyanov, V., Koychev, I. K., Nakov, P., & Radev, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.487 · Cited by 7
LLM-Generated Word Association Norms
Abramski, K., Lavorati, C., Rossetti, G., & Stella, M. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240177 · Cited by 5
An Evaluation System for Large Language Models based on Open-Ended Questions
Cao, Z., Ma, Z., & Chen, M. (2024) · IEEE International Conference on Cyber Security and Cloud Computing (CSCloud) · doi:10.1109/cscloud62866.2024.00019 · Cited by 5
HiBenchLLM: Historical Inquiry Benchmarking for Large Language Models
Chartier, M. A., Dakkoune, N., Bourgeois, G., & Jean, S. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102383 · Cited by 5
Deep Learning Strategies for Identifying Machine-Generated Text
Yadagiri, A., & Pakray, P. (2025) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1109/imcom64595.2025.10857545 · Cited by 5
Ciapparelli, M., Zarbo, C., & Marelli, M. (2025) · Cognitive Science · doi:10.1111/cogs.70048 · Cited by 5
Self-citationNarrative coherence in neural language models
Acciai, A., Guerrisi, L., Perconti, P., Plebe, A., Suriano, R., & Velardi, A. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1572076 · Cited by 5
The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition
Chochlakis, G., Potamianos, A., Lerman, K., & Narayanan, S. S. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00041 · Cited by 11
SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
Bashlovkina, V., Matthews, R., Kuang, Z., Baumgartner, S., & Bendersky, M. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599907 · Cited by 8
ALERT: Adapt Language Models to Reasoning Tasks
Yu, P., Wang, T., Golovneva, O., AlKhamissi, B., Verma, S. S., Jin, Z., Ghosh, G., Diab, M., & Çelikyılmaz, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.60 · Cited by 25
Self-citationJackson, D., & Courneya, M. (2023) · Brazilian Creative Industries Journal · doi:10.25112/bcij.v3i2.3540 · Cited by 5
Who Knows the Answer? Finding the Best Model and Prompt for Each Query Using Confidence-Based Search
Gerych, W., Rizk, Y., Isahagian, V., Muthusamy, V., Duesterwald, E., & Venkateswaran, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29763 · Cited by 5
Automatic Prompt Selection for Large Language Models
Do, V.-T., Nguyen, X.-Q., Hoang, V.-K., Nguyen, D., Sabahi, S., Yang, J., Hotta, H., Nguyen, M.-T., & Le, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8180-8_8 · Cited by 20
True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4
Del, M., & Fishel, M. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.28 · Cited by 22
Challenges and Opportunities in Neuro-Symbolic Composition of Foundation Models
Jha, S., Roy, A., Cobb, A. D., Berenbeim, A., & Bastian, N. D. (2023) · IEEE Military Communications Conference (MILCOM) · doi:10.1109/milcom58377.2023.10356344 · Cited by 6
Semeraro, A., Vilella, S., Mohammad, S. M., Ruffo, G., & Stella, M. (2023) · Research Square · doi:10.21203/rs.3.rs-2428155/v1 · Cited by 8
Self-citationGentopia.AI: A Collaborative Platform for Tool-Augmented LLMs
Xu, B., Liu, X., Shen, H., Han, Z., Li, Y., Yue, M., Peng, Z., Liu, Y., Yao, Z., & Xu, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.20 · Cited by 4
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. L., Reinecke, K., & Sap, M. (2023) · arXiv · doi:10.48550/arxiv.2306.01943 · Cited by 31
Self-citationAnchor Points: Benchmarking Models with Much Fewer Examples
Vivek, R., Ethayarajh, K., Yang, D., & Kiela, D. (2023) · arXiv · doi:10.48550/arxiv.2309.08638 · Cited by 4
Self-citationAnchor Points: Benchmarking Models with Much Fewer Examples
Vivek, R., Ethayarajh, K., Yang, D., & Kiela, D. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.95 · Cited by 4
Self-citationThompson, K. N., Chandler, K. L., Morgan, C., Khashabi, D., Delinski, E. A., & Durme, B. V. (2025) · Journal of Advanced Academics · doi:10.1177/1932202x251356323 · Cited by 5
Self-citation
Metadata
- Authors
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A., La, A., Lampinen, A., Zou, A., Jiang, A., Chen, A., Vuong, A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabassum, A., Menezes, A., Kirubarajan, A., Mullokandov, A., Sabharwal, A., Herrick, A., Efrat, A., Erdem, A., Karakaş, A., Roberts, B. R., Loe, B. S., Zoph, B., Bojanowski, B., Özyurt, B., Hedayatnia, B., Neyshabur, B., Inden, B., Stein, B., Ekmekci, B., Lin, B. Y., Howald, B., Orinion, B., Diao, C., Dour, C., Stinson, C., Argueta, C., Ramírez, C. F., Singh, C., Rathkopf, C., Meng, C., Baral, C., Wu, C., Callison-Burch, C., Waites, C., Voigt, C., Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C., Ashcraft, C., Garbacea, C., Sileo, D., Garrette, D., Hendrycks, D., Kilman, D., Roth, D., Freeman, D., Khashabi, D., Levy, D., González, D. M., Perszyk, D., Hernandez, D., Chen, D., Ippolito, D., Gilboa, D., Dohan, D., Drakard, D., Jurgens, D., Datta, D., Ganguli, D., Emelin, D., Kleyko, D., Yuret, D., Chen, D., Tam, D., Hupkes, D., Misra, D., Buzan, D., Mollo, D. C., Yang, D., Lee, D.-H., Schrader, D., Shutova, E., Cubuk, E. D., Segal, E., Hagerman, E., Barnes, E., Donoway, E., Pavlick, E., Rodola, E., Lam, E., Chu, E., Tang, E., Erdem, E., Chang, E., Chi, E. A., Dyer, E., Jerzak, E., Kim, E., Manyasi, E. E., Zheltonozhskii, E., Xia, F., Siar, F., Martínez-Plumed, F., Happé, F., Chollet, F., Rong, F., Mishra, G., Winata, G. I., de Melo, G., Kruszewski, G., Parascandolo, G., Mariani, G., Wang, G., Jaimovitch-López, G., Betz, G., Gur-Ari, G., Galijasevic, H., Kim, H., Rashkin, H., Hajishirzi, H., Mehta, H., Bogar, H., Shevlin, H., Schütze, H., Yakura, H., Zhang, H., Wong, H. M., Ng, I., Noble, I., Jumelet, J., Geissinger, J., Kernion, J., Hilton, J., Lee, J., Fisac, J. F., Simon, J. B., Koppel, J., Zheng, J., Zou, J., Kocoń, J., Thompson, J., Wingfield, J., Kaplan, J., Radom, J., Sohl-Dickstein, J., Phang, J., Wei, J., Yosinski, J., Novikova, J., Bosscher, J., Marsh, J., Kim, J., Taal, J., Engel, J., Alabi, J., Xu, J., Song, J., Tang, J., Waweru, J., Burden, J., Miller, J., Balis, J. U., Batchelder, J., Berant, J., Frohberg, J., Rozen, J., Hernandez-Orallo, J., Boudeman, J., Guerr, J., Jones, J., Tenenbaum, J. B., Rule, J. S., Chua, J., Kanclerz, K., Livescu, K., Krauth, K., Gopalakrishnan, K., Ignatyeva, K., Markert, K., Dhole, K. D., Gimpel, K., Omondi, K., Mathewson, K., Chiafullo, K., Shkaruta, K., Shridhar, K., McDonell, K., Richardson, K., Reynolds, L., Gao, L., Zhang, L., Dugan, L., Qin, L., Contreras-Ochando, L., Morency, L.-P., Moschella, L., Lam, L., Noble, L., Schmidt, L., He, L., Colón, L. O., Metz, L., Şenel, L. K., Bosma, M., Sap, M., ter Hoeve, M., Farooqi, M., Faruqui, M., Mazeika, M., Baturan, M., Marelli, M., Maru, M., Quintana, M. J. R., Tolkiehn, M., Giulianelli, M., Lewis, M., Potthast, M., Leavitt, M. L., Hagen, M., Schubert, M., Baitemirova, M. O., Arnaud, M., McElrath, M., Yee, M. A., Cohen, M., Gu, M., Ivanitskiy, M., Starritt, M., Strube, M., Swędrowski, M., Bevilacqua, M., Yasunaga, M., Kale, M., Cain, M., Xu, M., Suzgun, M., Walker, M., Tiwari, M., Bansal, M., Aminnaseri, M., Geva, M., Gheini, M., T, M. V., Peng, N., Chi, N. A., Lee, N., Krakover, N. G.-A., Cameron, N., Roberts, N., Doiron, N., Martinez, N., Nangia, N., Deckers, N., Muennighoff, N., Keskar, N. S., Iyer, N. S., Constant, N., Fiedel, N., Wen, N., Zhang, O., Agha, O., Elbaghdadi, O., Levy, O., Evans, O., Casares, P. A. M., Doshi, P., Fung, P., Liang, P. P., Vicol, P., Alipoormolabashi, P., Liao, P., Liang, P., Chang, P., Eckersley, P., Htut, P. M., Hwang, P., Miłkowski, P., Patil, P., Pezeshkpour, P., Oli, P., Mei, Q., Lyu, Q., Chen, Q., Banjade, R., Rudolph, R. E., Gabriel, R., Habacker, R., Risco, R., Millière, R., Garg, R., Barnes, R., Saurous, R. A., Arakawa, R., Raymaekers, R., Frank, R., Sikand, R., Novak, R., Sitelew, R., LeBras, R., Liu, R., Jacobs, R., Zhang, R., Salakhutdinov, R., Chi, R., Lee, R., Stovall, R., Teehan, R., Yang, R., Singh, S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff, S., Gehrmann, S., Schuster, S., Sadeghi, S., Hamdan, S., Zhou, S., Srivastava, S., Shi, S., Singh, S., Asaadi, S., Gu, S. S., Pachchigar, S., Toshniwal, S., Upadhyay, S., Shyamolima, Debnath, Shakeri, S., Thormeyer, S., Melzi, S., Reddy, S., Makini, S. P., Lee, S.-H., Torene, S., Hatwar, S., Dehaene, S., Divic, S., Ermon, S., Biderman, S., Lin, S., Prasad, S., Piantadosi, S. T., Shieber, S. M., Misherghi, S., Kiritchenko, S., Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L., Desbordes, T., Rothschild, T., Phan, T., Wang, T., Nkinyili, T., Schick, T., Kornev, T., Tunduny, T., Gerstenberg, T., Chang, T., Neeraj, T., Khot, T., Shultz, T., Shaham, U., Misra, V., Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V., Prabhu, V. U., Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, Y., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z., Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z., & Wu, Z.
- Affiliations
- Shammie
- ORCID
- Abid, Abubakar
- Brown, Adam R.
- Gupta, Aditya
- Power, Alethea
- Warstadt, Alex
- Kocurek, Alexander W.
- Safaya, Ali
- Tazarv, Ali
- Xiang, Alice
- Parrish, Alicia
- Nie, Allen
- Dsouza, Amanda
- Rahane, Ameet
- Andreassen, Anders
- Madotto, Andrea
- Santilli, Andrea
- Dai, Andrew
- Lampinen, Andrew
- Gupta, Animesh
- Norelli, Antonio
- Venkatesh, Anu
- Mullokandov, Asher
- Erdem, Aykut
- Loe, Bao Sheng
- Bojanowski, Bartłomiej
- Inden, Benjamin
- Stein, Benno
- Howald, Blake
- Stinson, Catherine
- Singh, Chandan
- Rathkopf, Charles
- Meng, Chenlin
- Baral, Chitta
- Callison-Burch, Chris
- Voigt, Christian
- Manning, Christopher D.
- Potts, Christopher
- Siro, Clemencia
- Raffel, Colin
- Sileo, Damien
- Roth, Dan
- Khashabi, Daniel
- Chen, Danqi
- Ippolito, Daphne
- Jurgens, David
- Datta, Debajyoti
- Ganguli, Deep
- Kleyko, Denis
- Chen, Derek
- Misra, Diganta
- Mollo, Dimitri Coelho
- Yang, Diyi
- Lee, Dong-Ho
- Shutova, Ekaterina
- Cubuk, Ekin Dogus
- Segal, Elad
- Barnes, Elizabeth
- Donoway, Elizabeth
- Pavlick, Ellie
- Erdem, Erkut
- Dyer, Ethan
- Jerzak, Ethan
- Kim, Ethan
- Zheltonozhskii, Evgenii
- Mishra, Gaurav
- Betz, Gregor
- Kim, Hannah
- Hajishirzi, Hannaneh
- Mehta, Harsh
- Shevlin, Henry
- Schütze, Hinrich
- Yakura, Hiromu
- Zhang, Hongming
- Geissinger, Jack
- Lee, Jaehoon
- Fisac, Jaime Fernández
- Simon, James B.
- Koppel, James
- Zou, James
- Kocoń, Jan
- Phang, Jason
- Novikova, Jekaterina
- Kim, Jeremy
- Xu, Jiacheng
- Song, Jiaming
- Burden, John
- Rozen, Jos
- Tenenbaum, Joshua B.
- Rule, Joshua S.
- Kanclerz, Kamil
- Livescu, Karen
- Krauth, Karl
- Dhole, Kaustubh D.
- Mathewson, Kory
- Richardson, Kyle
- Zhang, Li
- Dugan, Liam
- Contreras-Ochando, Lidia
- Morency, Louis-Philippe
- Moschella, Luca
- Lam, Lucas
- Şenel, Lütfi Kerem
- Sap, Maarten
- Farooqi, Maheen
- Marelli, Marco
- Tolkiehn, Marie
- Giulianelli, Mario
- Lewis, Martha
- Potthast, Martin
- Leavitt, Matthew L.
- Hagen, Matthias
- Yee, Michael A.
- Ivanitskiy, Michael
- Strube, Michael
- Swędrowski, Michał
- Yasunaga, Michihiro
- Bansal, Mohit
- Geva, Mor
- T, Mukund Varma
- Peng, Nanyun
- Chi, Nathan A.
- Lee, Nayeon
- Deckers, Niklas
- Muennighoff, Niklas
- Keskar, Nitish Shirish
- Iyer, Niveditha S.
- Wen, Nuan
- Zhang, Oliver
- Levy, Omer
- Casares, Pablo Antonio Moreno
- Fung, Pascale
- Liang, Paul Pu
- Liao, Peiyuan
- Liang, Percy
- Miłkowski, Piotr
- Pezeshkpour, Pouya
- Oli, Priti
- Mei, Qiaozhu
- Banjade, Rabin
- Habacker, Rahel
- Millière, Raphaël
- Saurous, Rif A.
- Arakawa, Riku
- Frank, Robert
- Zhang, Rui
- Salakhutdinov, Ruslan
- Teehan, Ryan
- Singh, Sahib
- Mohammad, Saif M.
- Anand, Sajant
- Dillavou, Sam
- Wiseman, Sam
- Gruetter, Samuel
- Schoenholz, Samuel S.
- Rous, Sarah A.
- Ghosh, Sayan
- Casey, Sean
- Bischoff, Sebastian
- Gehrmann, Sebastian
- Schuster, Sebastian
- Hamdan, Shadi
- Zhou, Sharon
- Srivastava, Shashank
- Shi, Sherry
- Singh, Shikhar
- Asaadi, Shima
- Gu, Shixiang Shane
- Melzi, Simone
- Reddy, Siva
- Makini, Sneha Priscilla
- Lee, Soo-Hwan
- Dehaene, Stanislas
- Divic, Stefan
- Ermon, Stefano
- Biderman, Stella
- Lin, Stephanie
- Piantadosi, Steven T.
- Shieber, Stuart M.
- Kiritchenko, Svetlana
- Mishra, Swaroop
- Linzen, Tal
- Schuster, Tal
- Li, Tao
- Wu, Te-Lin
- Desbordes, Théo
- Wang, Tianle
- Nkinyili, Tiberius
- Tunduny, Titus
- Gerstenberg, Tobias
- Chang, Trenton
- Shaham, Uri
- Demberg, Vera
- Ramasesh, Vinay
- Prabhu, Vinay Uday
- Padmakumar, Vishakh
- Srikumar, Vivek
- Zhang, William
- Vossen, Wout
- Ren, Xiang
- Zhao, Xinran
- Wu, Xinyi
- Shen, Xudong
- Yaghoobzadeh, Yadollah
- Lakretz, Yair
- Song, Yangqiu
- Choi, Yejin
- Yang, Yichi
- Hao, Yiding
- Chen, Yifu
- Belinkov, Yonatan
- Hou, Yu
- Hou, Yufang
- Bai, Yuntao
- Zhao, Zhuoye
- Wang, Zijian
- Wang, Zijie J.
- Wu, Ziyi
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2206.04615
- OpenAlex
- W4281690148
- Semantic Scholar
- bd1331b233e84bab7eba503abc60b31ac08e7881
- Cited by
- 569
- References
- 0
- Citations by year
- 2026: 29 · 2025: 114 · 2024: 169 · 2023: 232 · 2022: 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- model scaling, sparse transformer, GPT, model calibration, social bias, commonsense reasoning, mathematical reasoning, prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
