The DOI matches its DataCite record; 527 of the 528 references it cites were found in a registry or an index.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Open accessAbstract
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even performing some traditionally symbolic tasks well with only a few training examples [61].”
From Symbols and grounding in large language models (2023) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The evaluation of large language models has historically relied on benchmarks that emphasize static, non-interactive tasks”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Srivastava et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many benchmark sets do not allow for arbitrary generation of novel questions, or worse, draw data from publicly available sources-the same sources LLM trainers have access to [51, 39].”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [39]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Surprisingly, we observe that using training FLOPs leads to reasonable predictions of LM performance with and without CoT on BBH tasks, possibly due to the denoising effect of aggregation over all tasks.”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In order to standardize our evaluation, we use the same prompt for all models evaluated, following the convention established by [83–85].”
From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [85]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One might thus posit, with some justification from the above-mentioned studies, that the emergence of skills arises from training on related tasks that were implicitly solved while solving next-word prediction in the training dataset.This is indeed our starting point.”
From A Theory for Emergence of Complex Skills in Language Models (2023) · cites this paper as Srivastava et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prominent among these are multi-task benchmarks such as MMLU [325] and BIG-bench [326], which evaluate knowledge and skills across diverse domains and scenarios.”
From A Survey of Large Language Models (2026) · cites this paper as [326]
The cited source says what the claim attributes to it.
In the citing paper
“To holistically assess the capabilities of LLMs, several comprehensive benchmarks have been developed [325-327].”
From A Survey of Large Language Models (2026) · cites this paper as [326]
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to prior benchmarks (e.g., SuperGLUE, 9 EleutherAI LM Harness, 10 and BIG-Bench 11 ), which enumerate a set of scenarios and metrics, we situate our scenarios in a larger taxonomy to make explicit what is currently missing.”
From Holistic Evaluation of Language Models (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“BIG-Bench Hard (BBH; Suzgun et al., 2022) is a set of 23 multiple-choice tasks from the BIG-Bench benchmark (Srivastava et al., 2022), on which current state-of-the-art models do substantially worse than the average human rater.”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“BIG-bench [182] introduces a diverse collection of 204 challenging tasks contributed by 450 authors from 132 institutions.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [182]
The cited source says what the claim attributes to it.
In the citing paper
“Popular tests, such as MMLU (137), BIG-Bench (138), BIG-Bench Hard (BBH) (139), or AGIEval (140) are typically based on human ability tasks (137) but test for knowledge recall or problem-solving (141), often by comparing performance against human experts doing the same tasks (138).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [138]
The cited source says what the claim attributes to it.
In the citing paper
“We show that when we use a format with visible lettered answer options, large models are very well calibrated on diverse multiple choice questions (e.g. from BIG Bench [Srivastava et al., 2022], MMLU[Hendrycks et al., 2021], and many other evaluations); see Figures 4, 5, and 6.”
From Language Models (Mostly) Know What They Know (2022) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Although biases already pose a challenge for conventional AI in health, they are of particular relevance for GMAI as a recent large-scale evaluation showed that social bias can increase with model scale 46 .”
From Foundation models for generalist medical artificial intelligence (2023) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Our experiments validate the effectiveness of this approach on a range of complex reasoning tasks, including several challenging Big-Bench tasks (Srivastava et al., 2022).”
From Large language models as tool makers (2023) · cites this paper as Srivastava et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“We note that our evaluation is in no means comprehensive and a larger benchmarking akin to Srivastava et al. (2022) would be helpful.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Srivastava et al. (2022)
The cited source says what the claim attributes to it.
In the citing paper
“Large language models have demonstrated remarkable reasoning ability on a variety of reasoning tasks26–29.”
From Solving olympiad geometry without human demonstrations (2024) · cites this paper as [28]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Furthermore, using PC measures accurately captures the scaling trends in both setups, even when using training FLOPs leads to less tight captures in the”
From Observational Scaling Laws and the Predictability of Language Model Performance (2024) · cites this paper as [83]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 140
- Scott Alexander. A very unlikely chess game, 2020. URL https://slatestarcodex.com/2020/01/06/a-very-unlikely-chessgame/ . (cited on pp. 29 and 30)
- Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 245-256. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/alon20a.html . (cited on p. 37)
- Miriam Amin and Manuel Burghardt. A survey on approaches to computational humor generation. In Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature , pp. 29-41, Online, December 2020. International Committee on Computational Linguistics. URL https://aclanthology. org/2020.latechclfl-1.4 . (cited on p. 38)
- Brandon Amos and J. Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks, 2017. URL https: //arxiv.org/abs/1703.00443 . (cited on p. 38)
- Salvatore Attardo. Humor in language. In Oxford Research Encyclopedia of Linguistics . Oxford University Press, 2017. doi: 10. 1093/acrefore/9780199384655.013.342. URL https://oxfordre.com/linguistics/view/10.1093/acrefore/9780199384655. 001.0001/acrefore-9780199384655-e-342 . (cited on p. 38) · DOI
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: //arxiv.org/abs/2108.07732 . (cited on p. 4)
- Satanjeev Banerjee and Ted Pedersen. Extended gloss overlaps as a measure of semantic relatedness. In IJCAI'03: Proceedings of the 18th International Joint Conference on Artificial Intelligence , pp. 805-810, San Francisco, 2003. Morgan Kaufmann. doi: 10.5555/1630659.1630775. URL https://dl.acm.org/doi/10.5555/1630659.1630775 . (cited on p. 36) · DOI
- Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review , 104(3):671-732, 2016. URL http://www.jstor.org/stable/24758720 . (cited on p. 31)
- Gregor Betz, Christian Voigt, and Kyle Richardson. Critical thinking for language models, 2020. URL https://arxiv.org/abs/ 2009.07185 . (cited on p. 33)
- Stella Biderman and Edward Raff. Neural language models are effective plagiarists, 2022. URL https://arxiv.org/abs/2201. 07406 . (cited on p. 4)
- Julia Birke and Anoop Sarkar. A clustering approach for nearly unsupervised recognition of nonliteral language. In 11th Conference of the European Chapter of the Association for Computational Linguistics , pp. 329-336, Trento, Italy, April 2006. Association for Computational Linguistics. URL https://aclanthology.org/E06-1042 . (cited on p. 35)
- Nicholas Boillot. Vector forms as a foreign language, 24 June 2019. URL https://www.fluate.net/en/travaux/vectoglyph . (cited on pp. 31 and 33)
- Paul F. Boller, Jr. and John George. They Never Said It: A Book of Fake Quotes, Misquotes, and Misleading Attributions . Oxford University Press, Oxford, 1989. (cited on p. 35)
- Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings. neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html . (cited on pp. 17, 29, 31, and 33)
- Bozhidar Bozhanov and Ivan Derzhanski. Rosetta stone linguistic problems. In Proceedings of the Fourth Workshop on Teaching NLP and CL , pp. 1-8, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https: //aclanthology.org/W13-3401 . (cited on p. 34)
- Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable Forth interpreter. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 547-556, 2017. URL https: //arxiv.org/abs/1605.06640 . (cited on p. 34)
- Gwern Branwen. GPT-3 creative fiction. Gwern.net , June 2020. URL https://www.gwern.net/GPT-3 . (cited on pp. 19, 38, and 39)
- Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review , 78(1):1-3, 1950. (cited on pp. 8 and 11)
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html . (cited on pp. 4, 7, 8, 10, 11, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40, and 41)
- Corrado Böhm. On a family of Turing machines and the related programming language. ICC Bulletin , 3:187-194, 1964. (cited on p. 38)
- Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pp. 2633-2650. USENIX Association, August 2021. URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting . (cited on p. 19)
- Nathanael Chambers. Labeling documents with timestamps: Learning from their time expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 98-106, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1011 . (cited on p. 31)
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 1691-1703. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/chen20s.html . (cited on p. 29)
- Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. https://openreview.net/pdf?id= H1gfOiAqYm , 2019b. (cited on p. 34)
- Kate Crawford. The trouble with bias. https://www.youtube.com/watch?v=fMym\_BKWQzk , 2017. Keynote address, NIPS 2017, Long Beach CA. Dec. 5, 2017. (cited on p. 29)
- Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 933-941, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305478. URL https://dl.acm.org/doi/10.5555/3305381. 3305478 . (cited on p. 10) · DOI
- Wayne Davis. Implicature. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Fall 2019 edition, 2019. (cited on p. 33)
- Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. Finding contradictions in text. In Proceedings of ACL-08: HLT , pp. 1039-1047, Columbus, Ohio, June 2008. Association for Computational Linguistics. URL https: //aclanthology.org/P08-1118 . (cited on p. 32)
- Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. RobustFill: Neural program learning under noisy I/O. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pp. 990-998, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.5555/3305381.3305484. URL https://dl.acm.org/doi/10.5555/3305381.3305484 . (cited on p. 34) · DOI
- Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. Beyond English-centric multilingual machine translation. Journal of Machine Learning Research , 22(107): 1-48, 2021. URL http://jmlr.org/papers/v22/20-1307.html . (cited on p. 20)
- Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database . MIT Press, Cambridge, MA, 1998. (cited on p. 33)
- Jerry A. Fodor. The Language of Thought . Harvard University Press, Cambridge, MA, 1975. (cited on p. 30)
- Evgeniy Gabrilovich and Shaul Markovitch. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In IJCAI'07: Proceedings of the 20th International Joint Conference on Artifical Intelligence , pp. 1606-1611, San Francisco, 2007. Morgan Kaufmann. doi: 10.5555/1625275.1625535. URL https://dl.acm.org/doi/10.5555/1625275.1625535 . (cited on p. 36) · DOI
- Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics , 9: 346-361, 04 2021. doi: 10.1162/tacl\_a\_00370. URL https://doi.org/10.1162/tacl\_a\_00370 . (cited on pp. 29 and 39) · DOI
- Arthur S. Goldberger. Structural equation methods in the social sciences. Econometrica , 40(6):979-1001, 1972. URL http://www.jstor.org/stable/1913851 . (cited on p. 32)
- Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pp. 581-586, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https://aclanthology.org/P11-2102 . (cited on p. 39)
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pp. 1321-1330. PMLR, 06-11 Aug. 2017. URL https://proceedings.mlr.press/v70/guo17a. html . (cited on p. 11)
- Samuel Gyasi Obeng. The proverb as a mitigating and politeness strategy in Akan discourse. Anthropological Linguistics , 38(3): 521-549, 1996. URL http://www.jstor.org/stable/30028601 . (cited on p. 32)
- Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171, Dec. 2020. doi: 10.1162/tacl\_a\_00306. URL https://doi.org/10.1162/tacl\_a\_00306 . (cited on p. 31) · DOI
- Joseph Y. Halpern. Actual causality . MIT Press, Cambridge, MA, 2016. (cited on p. 29)
- Maria Hanzén. When in Rome, do as the Romans do: Proverbs as a part of EFL teaching. Master's thesis, Jönköping University, School of Education and Communication, Jönköping, 2007. URL http://www.diva-portal.org/smash/get/diva2: 3499/fulltext01.pdf . (cited on p. 40)
- Irene Heim. On the projection problem for presuppositions. In Paul Portner and Barbara H. Partee (eds.), Formal Semantics The Essential Readings , pp. 249-260. Blackwell, Oxford, 1983. (cited on p. 37)
- Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Hkg4TI9xl . (cited on pp. 11 and 12)
- Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 2712-2721. PMLR, 09-15 June 2019. URL https: //proceedings.mlr.press/v97/hendrycks19a.html . (cited on p. 11)
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS, 2021a. URL https: //arxiv.org/abs/2105.09938 . (cited on pp. 4, 37, and 41)
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021b. URL https: //openreview.net/forum?id=d7KBjmI3GmQ . (cited on pp. 25, 31, 35, 38, and 40)
- Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips. cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html . (cited on pp. 5 and 32)
- Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URL https: //arxiv.org/abs/2102.01293 . (cited on p. 41)
- Keith J. Holyoak. Analogy and relational reasoning. In Keith J. Holyoak and Robert G. Morrison (eds.), The Oxford Handbook of Thinking and Reasoning . Oxford University Press, Oxford, 2012. URL https://www.oxfordhandbooks.com/view/10.1093/ oxfordhb/9780199734689.001.0001/oxfordhb-9780199734689-e-13 . (cited on p. 28)
- Yufang Hou. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 1428-1438, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.132. URL https://aclanthology.org/2020.acl-main.132 . (cited on p. 29)
- Yufang Hou, Katja Markert, and Michael Strube. Global inference for bridging anaphora resolution. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 907-917, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology.org/N131111 . (cited on p. 29)
- Hrisztalina Hrisztova-Gotthardt and Melita Aleksa Varga. Introduction to Paremiology: A Comprehensive Guide to Proverb Studies . De Gruyter Open, Warsaw, 2015. URL https://www.degruyter.com/document/doi/10.2478/9783110410167/html . (cited on p. 32) · DOI
- Thad Hughes and Daniel Ramage. Lexical semantic relatedness with random graph walks. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , pp. 581-589, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/D07-1061 . (cited on p. 36)
- Jay J. Jiang and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th Research on Computational Linguistics International Conference , pp. 19-33, Taipei, Taiwan, August 1997. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). URL https://aclanthology.org/O97-1002 . (cited on p. 36)
- Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438, 2020. doi: 10.1162/tacl\_a\_00324. URL https://doi.org/10. 1162/tacl\_a\_00324 . (cited on pp. 11 and 12) · DOI
- Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS'15 , volume 1, pp. 190-198, Cambridge, MA, USA, 2015. MIT Press. doi: 10.5555/2969239.2969261. URL https://dl.acm.org/doi/10.5555/2969239.2969261 . (cited on p. 34) · DOI
- Lauri Karttunen. Simple and phrasal implicatives. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pp. 124-131, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1020 . (cited on p. 32)
- Faisal Khan, Bilge Mutlu, and Jerry Zhu. How do humans teach: On curriculum learning and teaching dimension. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper/2011/file/ f9028faec74be6ec9b852b0a542e2f39-Paper.pdf . (cited on p. 38)
- Milton King and Paul Cook. Evaluating approaches to personalizing language models. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 2461-2469, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.299 . (cited on p. 38)
- Christo Kirov and Ryan Cotterell. Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate. Transactions of the Association for Computational Linguistics , 6:651-665, 12 2018. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00247. URL https://doi.org/10.1162/tacl\_a\_00247 . (cited on p. 34) · DOI
- Joshua Knobe. Intentional action and side effects in ordinary language. Analysis , 63, 07 2003. doi: 10.1111/ 1467-8284.00419. URL https://www.researchgate.net/publication/28763794\_Intentional\_Action\_and\_Side\_Effects\_ in\_Ordinary\_Language . (cited on p. 29)
- Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics , 6:317-328, 2018. doi: 10.1162/tacl\_a\_00023. URL https://aclanthology.org/Q18-1023 . (cited on p. 35) · DOI
- Alexander W. Kocurek, Ethan Jerzak, and Rachel Etta Rudolph. Against conventional wisdom. Philosophers' Imprint , 20(22): 1-27, 2020. URL http://hdl.handle.net/2027/spo.3521354.0020.022 . (cited on p. 29)
- Jarmo Korhonen. Sprichwörter und zweisprachige lexikographie: Deutsch-schwedische und deutsch-finnische wörtebücher im vergleich. In C. Földes (ed.), Phraseologie disziplinär und interdisziplinär , pp. 537-549. Gunter Narr Verlag, 2009. (cited on p. 40)
- Mahnaz Koupaee and William Yang Wang. WikiHow: A large scale text summarization dataset, 2018. URL https://arxiv. org/abs/1810.09305 . (cited on p. 5)
- Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics , 10:50-72, 01 2022. doi: 10.1162/tacl\_a\_00447. URL https://doi.org/10.1162/tacl\_a\_00447 . (cited on p. 20) · DOI
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 08 2019. doi: 10.1162/tacl\_a\_00276. URL https://doi.org/10.1162/tacl\_a\_00276 . (cited on pp. 5 and 37) · DOI
- Kevin Lacker. Giving GPT-3 a Turing test. Kevin Lacker's blog , July 2020. URL https://lacker.io/ai/2020/07/06/givinggpt-3-a-turing-test.html . (cited on pp. 33 and 41)
- George Lakoff and Mark Johnson. Metaphors We Live By . University of Chicago Press, Chicago, 2008. (cited on pp. 33 and 35)
- Guillaume Lample and François Charton. Deep learning for symbolic mathematics, 2019. URL https://arxiv.org/abs/1912. 01412 . (cited on p. 38)
- Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 986-995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL https://aclanthology.org/I17-1099 . (cited on p. 40)
- Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics , 4:521-535, 12 2016. doi: 10.1162/tacl\_a\_00115. URL https://doi.org/10.1162/tacl\_a\_00115 . (cited on p. 39) · DOI
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019. URL https://arxiv.org/abs/1907. 11692 . (cited on pp. 39 and 41)
- Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. In Vivek Nigam, Tajana Ban Kirigin, Carolyn Talcott, Joshua Guttman, Stepan Kuznetsov, Boon Thau Loo, and Mitsuhiro Okada (eds.), Logic, Language, and Security . Springer, Cham, 2020. URL https://www.springerprofessional. de/en/gender-bias-in-neural-natural-language-processing/18531692 . (cited on p. 33)
- Jihang Mao and Wanli Liu. A BERT-based approach for automatic humor detection and scoring. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019) , pp. 197-202, 2019. URL http://ceur-ws.org/Vol-2421/HAHA\_paper\_8.pdf . (cited on p. 39)
- Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence, 2020. URL https://arxiv.org/abs/ 2002.06177 . (cited on p. 30)
- Katja Markert, Yufang Hou, and Michael Strube. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 795-804, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/P12-1084 . (cited on p. 29)
- R. Thomas McCoy, Robert Frank, and Tal Linzen. Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks. Transactions of the Association for Computational Linguistics , 8:125-140, 01 2020. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00304. URL https://doi.org/10.1162/tacl\_a\_00304 . (cited on p. 34) · DOI
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https: //arxiv.org/abs/1609.07843 . (cited on p. 31)
- John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 6905-6916. PMLR, 13-18 July 2020. URL https://proceedings.mlr.press/v119/miller20a.html . (cited on p. 39)
- David Milne and Ian H. Witten. An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: An Evolving Synergy , pp. 25-30, Menlo Park, 2008. Association for the Advancement of Artificial Intelligence. URL https://www.aaai.org/Papers/Workshops/2008/WS08-15/WS08-15-005.pdf . (cited on p. 36)
- Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Natural reference to objects in a visual domain. In Proceedings of the 6th International Natural Language Generation Conference . Association for Computational Linguistics, July 2010. URL https://aclanthology.org/W10-4210 . (cited on p. 38)
- Margaret Mitchell, Kees van Deemter, and Ehud Reiter. Generating expressions that refer to visible objects. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 1174-1184, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https:// aclanthology.org/N13-1137 . (cited on p. 38)
- Elham Mohammadi, Hessam Amini, and Leila Kosseim. CLaC at CLPsych 2019: Fusion of neural features and predicted class probabilities for suicide risk assessment based on online posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology , pp. 34-38, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-3004. URL https://aclanthology.org/W19-3004 . (cited on p. 39) · DOI
- Jane Morris and Graeme Hirst. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics , 17(1):21-48, 1991. URL https://aclanthology.org/J91-1002 . (cited on p. 36)
- Karl Mulligan, Robert Frank, and Tal Linzen. Structure here, bias there: Hierarchical generalization by jointly learning syntactic transformations. In Proceedings of the Society for Computation in Linguistics 2021 , pp. 125-135, Online, February 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.scil-1.12 . (cited on p. 34)
- Gregory L. Murphy. Comprehending complex concepts. Cognitive Science , 12(4):529-562, 1988. doi: https://doi.org/10.1207/ s15516709cog1204\_2. URL https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1204\_2 . (cited on p. 30) · DOI
- Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 8558cb408c1d76621371888657d2eb1d-Abstract.html . (cited on p. 11)
- Anthony M. Paul. Figurative language. Philosophy & Rhetoric , 3(4):225-248, 1970. URL http://www.jstor.org/stable/ 40237206 . (cited on p. 35)
- Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann, San Francisco, 1988. (cited on p. 32)
- Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, Cambridge, 2000. (cited on p. 30)
- Tony A. Plate. Distributed representations and nested compositional structure . PhD thesis, University of Toronto, Toronto, 1994. (cited on p. 29)
- Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving, 2020. URL https://arxiv. org/abs/2009.03393 . (cited on pp. 34 and 38)
- Willard V.O. Quine. Main trends in recent philosophy: Two dogmas of empiricism. The Philosophical Review , 60(1):20-43, 1951. URL http://www.jstor.org/stable/2181906 . (cited on p. 29)
- Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/languagemodels.pdf . (cited on pp. 10, 30, 33, 35, 37, 38, and 39)
- Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pp. 777-789, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology. org/D12-1071 . (cited on p. 41)
- Ian Ravenscroft. Folk psychology as a theory. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Summer 2019 edition, 2019. (cited on p. 32)
- Joseph Reisinger and Raymond J. Mooney. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 109-117, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology. org/N10-1013 . (cited on p. 36)
- Philip Resnik. Using information content to evaluate semantic similarity in a taxonomy. In IJCAI'95: Proceedings of the 14th International Joint Conference on Artificial Intelligence, Volume 1 , pp. 448-453, San Francisco, 1995. Morgan Kaufmann. doi: 10.5555/1625855.1625914. URL https://dl.acm.org/doi/10.5555/1625855.1625914 . (cited on p. 36) · DOI
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. AAAI Spring Symposium , 2011. URL http://commonsensereasoning.org/2011/papers/Roemmele.pdf . (cited on p. 32)
- Joshua S. Rule. The child as hacker: Building more human-like models of learning . PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2020. URL https://hdl.handle.net/1721.1/129232 . (cited on p. 34)
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 . (cited on pp. 9 and 25)
- Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 624-643, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.52. URL https://aclanthology.org/2021.naacl-main.52 . (cited on p. 41) · DOI
- Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021b. URL https://openreview.net/ forum?id=fe\_hCc4RBrg . (cited on p. 4)
- Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. Diagram understanding in geometry questions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, Menlo Park, CA, Jun. 2014. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9146 . (cited on p. 34)
- Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL http: //arxiv.org/abs/1804.04235 . (cited on p. 43)
- Ekaterina Shutova. Automatic metaphor interpretation as a paraphrasing task. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pp. 1029-1037, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https://aclanthology.org/N10-1147 . (cited on p. 35)
- Ekaterina Shutova and Simone Teufel. Metaphor corpus annotated for source-target domain mappings. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) , Valletta, Malta, May 2010. European Language Resources Association. URL http://www.lrec-conf.org/proceedings/lrec2010/pdf/612\_Paper.pdf . (cited on p. 35)
- Damien Sileo, Tim Van de Cruys, Camille Pradel, and Philippe Muller. DiscSense: Automated semantic analysis of discourse markers. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 991-999, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.125 . (cited on p. 31)
- Gurdeep Singh, Kaustubh Dhole, Priyadarshini P. Pai, and Sukanta Mondal. SPRINGS: Prediction of protein-protein interaction sites using artificial neural networks. Journal of Proteomics & Computational Biology , 1:7, 2014. URL https: //www.avensonline.org/fulltextarticles/JPCB-2572-8679-01-0001.html . (cited on p. 37)
- Douglas R. Smith. The synthesis of LISP programs from examples: A survey. In Alan W. Biermann, Gerhard Guiho, and Yves Kodratoff (eds.), Automatic Program Construction Techniques , pp. 307-324. Macmillan, New York, 1984. (cited on p. 34)
- Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences , 11(1):1-23, 1988. doi: 10.1017/ S0140525X00052432. (cited on p. 30)
- Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://aclanthology.org/D13-1170 . (cited on p. 31)
- Shashank Srivastava, Snigdha Chaturvedi, and Tom Mitchell. Inferring interpersonal relations in narrative summaries. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16 , pp. 2807-2813, Menlo Park, CA, 2016. Association for the Advancement of Artificial Intelligence. doi: 10.5555/3016100.3016294. URL https://dl.acm.org/doi/10. 5555/3016100.3016294 . (cited on p. 34) · DOI
- Robert Stalnaker. Assertion. In P. Cole (ed.), Pragmatics , Syntax and Semantics 9, pp. 315-332. Brill, Leiden, 1978. (cited on p. 32)
- Bernd Steinbach and Roman Kohut. Neural networks - a model of boolean functions. 5th International Workshop on Boolean Problems, Freiburg, Sept. 2002. , 2002. URL https://www.researchgate.net/publication/246931125\_Neural\_Networks\_-\_A\_Model\_of\_Boolean\_Functions . (cited on p. 29)
- Andreas Stöckl. Watching a language model learning chess. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) , pp. 1369-1379, Held Online, September 2021. INCOMA Ltd. URL https://aclanthology.org/2021.ranlp-1.153 . (cited on p. 4)
- Kevin Stowe, Leonardo Ribeiro, and Iryna Gurevych. Metaphoric paraphrase generation, 2020. URL https://arxiv.org/abs/ 2002.12854 . (cited on p. 35)
- Michael Strube and Simone Paolo Ponzetto. Wikirelate! Computing semantic relatedness using Wikipedia. In AAAI'06: Proceedings of the 21st National Conference on Artificial Intelligence , volume 2, pp. 1419-1424. Association for the Advancement of Artificial Intelligence, 2006. doi: 10.5555/1597348.1597414. URL https://dl.acm.org/doi/10.5555/1597348. 1597414 . (cited on p. 36) · DOI
- Saku Sugawara, Hikaru Yokono, and Akiko Aizawa. Prerequisite skills for reading comprehension: Multi-perspective analysis of MCTest datasets and systems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, Menlo Park, CA, Feb. 2017. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/ view/10957 . (cited on p. 35)
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks, 2014. URL https: //arxiv.org/abs/1409.3215 . (cited on p. 4)
- Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Tunuguntla Deepak, and Oskar van der Wal. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models , 2022. URL https://openreview.net/forum?id=rK-7NhfSIW5 . (cited on pp. 18 and 20)
- Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. Learning to recommend quotes for writing. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15 , pp. 2453-2459, Menlo Park, CA, 2015. Association for the Advancement of Artificial Intelligence. URL https://dl.acm.org/doi/10.5555/2886521.2886662 . (cited on p. 28) · DOI
- Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15 , pp. 1923-1929, 2015. URL https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/view/10957/10931 . (cited on p. 38)
- George Tsatsaronis, Iraklis Varlamis, Michalis Vazirgiannis, and Kjetil Nørvåg. Omiotis: A thesaurus-based measure of text relatedness. In Wray Buntine, Marko Grobelnik, Dunja Mladenić, and John Shawe-Taylor (eds.), Machine Learning and Knowledge Discovery in Databases , pp. 742-745, Berlin, 2009. Springer. (cited on p. 36)
- Shikhar Vashishth, Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Dating documents using graph convolution networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1605-1615, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ v1/P18-1149. URL https://aclanthology.org/P18-1149 . (cited on p. 31)
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 30, pp. 5998--6008. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf . (cited on pp. 10 and 38)
- Jette Viethen and Robert Dale. The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference , pp. 59-67, Salt Fork, Ohio, USA, June 2008. Association for Computational Linguistics. URL https://aclanthology.org/W08-1109 . (cited on p. 38)
- Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers , pp. 176-187, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-1017 . (cited on p. 35)
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019a. URL https://proceedings.neurips.cc/paper/2019/hash/ 4496bf24afe7fab6f046bf4923da8de6-Abstract.html . (cited on pp. 5, 7, 29, 32, and 41)
- David Wechsler. Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) . Pearson, San Antonio, 2008. (cited on p. 38)
- Ludwig Wittgenstein. Philosophical investigations . Basil Blackwell, Oxford, 1953. (cited on p. 29)
- Shijie Wu, Ryan Cotterell, and Mans Hulden. Applying the transformer to character-level transduction, 2020. URL https: //arxiv.org/abs/2005.10213 . (cited on p. 41)
- Scott Cheng-Hsin Yang and Patrick Shafto. Explainable artificial intelligence via Bayesian teaching. Workshop on Teaching Machines, Robots, and Humans, NIPS 2017 , 2017. URL http://shaftolab.com/assets/papers/yangShafto\_NIPS\_2017\_ machine\_teaching.pdf . (cited on p. 38)
- Eric Yeh, Daniel Ramage, Christopher D. Manning, Eneko Agirre, and Aitor Soroa. WikiWalk: Random walks on Wikipedia for semantic relatedness. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4) , pp. 41-49, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https: //aclanthology.org/W09-3206 . (cited on p. 36)
- Wenpeng Yin and Yulong Pei. Optimizing sentence modeling and selection for document summarization. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI-15 , pp. 1383-1389, 2015. doi: 10.5555/2832415.2832442. URL https://dl.acm.org/doi/10.5555/2832415.2832442 . (cited on p. 32) · DOI
- Eliezer Yudkowsky. Artificial intelligence as a positive and negative factor in global risk. In Nick Bostrom and Milan M. Ćirković (eds.), Global Catastrophic Risks , pp. 308-345. Oxford University Press, Oxford, 2008. URL https://web.archive.org/web/ 20210125025955/https://intelligence.org/files/AIPosNegFactor.pdf . (cited on p. 38)
- Omnia Zayed, John Philip McCrae, and Paul Buitelaar. Figure me out: A gold standard dataset for metaphor interpretation. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 5810-5819, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020.lrec-1.712 . (cited on p. 35)
- Meishan Zhang, Yue Zhang, and Guohong Fu. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pp. 2449-2460, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https://aclanthology.org/C16-1231 . (cited on p. 39)
- Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, Menlo Park, CA, Mar. 2015. Association for the Advancement of Artificial Intelligence. URL https://ojs.aaai.org/index.php/AAAI/article/view/9761 . (cited on p. 38)
Cited by
Sorted by citations
Items 401–500 of 681
Vervoort, L., & Nikolaev, V. (2025) · Journal for General Philosophy of Science · doi:10.1007/s10838-025-09735-7 · Cited by 4
Is AI at human parity yet? A case study on speech recognition
Beaver, I. (2022) · AI Magazine · doi:10.1002/aaai.12071 · Cited by 4
After Computational Reproducibility: Scientific Reproducibility and Trustworthy AI
Yu, B. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.ea5e6f9a · Cited by 4
Prompt Engineering ChatGPT for Codenames
Sidji, M., & Stephenson, M. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645591 · Cited by 4
Loconte, R., Orrù, G., Tribastone, M., Pietrini, P., & Sartori, G. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e38911 · Cited by 9
Du, Y., & Chilton, L. B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.171 · Cited by 8
Wicke, P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.302 · Cited by 6
Towards an Exhaustive Evaluation of Vision-Language Foundation Models
Salin, E., Ayache, S., & Favre, B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00041 · Cited by 4
Weissweiler, L., Hofmann, V., Köksal, A., & Schütze, H. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1225791 · Cited by 4
Self-citationCRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
Ismayilzada, M., Paul, D., Montariol, S., Geva, M., & Bosselut, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.607 · Cited by 14
Self-citationYe, S., Jang, J., Kim, D., Jo, Y., & Seo, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.822 · Cited by 10
Is It Possible to Truly Understand Performance in LLMs?
Greengard, S. (2024) · Communications of the ACM · doi:10.1145/3695860 · Cited by 4
Analysing the Predictability of Language Model Performance
Schellaert, W., Martínez‐Plumed, F., & Hernández‐Orallo, J. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3706118 · Cited by 7
Self-citationTrippas, J. R. (2024) · ACM eBooks · doi:10.1145/3674127.3674135 · Cited by 5
Liu, P., Chen, Z., Li, Y., & Wong, W. E. (2025) · International Symposium on System Security, Safety, and Reliability (ISSSR) · doi:10.1109/isssr65654.2025.00047 · Cited by 4
Zhou, S., Li, T., Wang, K., Huang, Y., Shi, L., Liu, Y., & Wang, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00209 · Cited by 18
Self-citationA chit-chat between Llama 2 and ChatGPT for the automated creation of exploits
Caturano, F., Ciotola, J., Romano, S. P., & Varlese, M. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111501 · Cited by 4
The Computational Evolution of Cognitive Architectures
Kotseruba, I., & Tsotsos, J. K. (2025) · doi:10.1093/oso/9780192844835.001.0001 · Cited by 4
Bommasani, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.522 · Cited by 7
Probing neural language models for understanding of words of estimative probability
Sileo, D., & Moens, M. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.41 · Cited by 10
Self-citationCoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification
Kim, S., Joo, S. J., Jang, Y., Chae, H., & Yeo, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.23 · Cited by 5
The Path to AGI Goes through Embodiment
Tan, C., & Jaiswal, S. (2023) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v1i1.27485 · Cited by 4
An Investigation of LLMs’ Inefficacy in Understanding Converse Relations
Qi, C., Li, B., Hui, B., Wang, B., Li, J., Wu, J., & Laili, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.429 · Cited by 7
Genetic Algorithm for Prompt Engineering with Novel Genetic Operators
Tanaka, H., Mori, N., & Okada, M. (2023) · IIAI International Congress on Advanced Applied Informatics (IIAI-AAI) · doi:10.1109/iiai-aai-winter61682.2023.00047 · Cited by 4
Automated Theorem Provers Help Improve Large Language Model Reasoning
McGinness, L. P., & Baumgartner, P. (2024) · EPiC series in computing · doi:10.29007/2n9m · Cited by 9
Establishing vocabulary tests as a benchmark for evaluating large language models
Martínez, G., Conde, J., Merino-Gómez, E., Margaretto, B. B., Hernández, J. A., Reviriego, P., & Brysbaert, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308259 · Cited by 6
Abramski, K., Citraro, S., Lombardi, L., Rossetti, G., & Stella, M. (2023) · doi:10.31234/osf.io/27u6z · Cited by 4
Semeraro, A., Vilella, S., Improta, R., Duro, E. S. D., Mohammad, S. M., Ruffo, G., & Stella, M. (2024) · doi:10.31234/osf.io/mnvy3 · Cited by 4
Self-citationLangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases
Bouchard, D., Chauhan, M. S., Skarbrevik, D., Bajaj, V., & Ahmad, Z. (2025) · The Journal of Open Source Software · doi:10.21105/joss.07570 · Cited by 9
Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational CompliancePreprint
Imperial, J. M., Jones, M. D., & Madabushi, H. T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5146161 · Cited by 4
Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models
Lyu, C., Wu, M., & Aji, A. F. (2024) · arXiv · doi:10.48550/arxiv.2402.13887 · Cited by 4
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3
Self-citationOn the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
“Many benchmark sets do not allow for arbitrary generation of novel questions, or worse, draw data from publicly available sources–the same sources LLM trainers have access to [36, 29].”
BackgroundReliable Reasoning Beyond Natural Language
Borazjanizadeh, N., & Piantadosi, S. T. (2024) · arXiv · doi:10.48550/arxiv.2407.11373 · Cited by 3
“…mathematical tasks [27], and the simple and repetitive pattern of reasoning required to solve the problems of the current reasoning benchmarks [11, 32, 28, 10, 19], we present a new dataset that (i) is certainly outside of current models’ training sets, and (ii) each problem necessitates a unique…”
Self-citationNear to Mid-term Risks and Opportunities of Open-Source Generative AI
Eiras, F., Petrov, A., Vidgen, B., de Witt, C. S., Pizzati, F., Elkins, K., Mukhopadhyay, S., Bibi, A., Csaba, B., Steibel, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.17047 · Cited by 2
Shinwari, H. U. K., & Usama, M. (2025) · arXiv · doi:10.48550/arxiv.2506.18267 · Cited by 3
“Out-of-distribution (OOD) sig-nifies performance on related, unseen datasets (MMLU-tuned model on BigBench–Hard [31]).”
Agarwal, N. S., & Behera, R. K. (2024) · International Conference on Computational Intelligence and Networks (CINE) · doi:10.1109/cine63708.2024.10881793 · Cited by 3
Relins, S., Birks, D., & Lloyd, C. (2025) · Journal of Quantitative Criminology · doi:10.1007/s10940-025-09611-z · Cited by 5
A review on recent advancements of ChatGPT and datafication in healthcare applications
Jagatheesaperumal, S. K., Pandiyarajan, A., Prabadevi, B., Deepa, N., Barreto, A. G., & de Albuquerque, V. H. C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110885 · Cited by 6
Active use of latent tree-structured sentence representation in humans and large language models
Liu, W., Xiang, M., & Ding, N. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02297-0 · Cited by 8
Understanding Domain Learning in Language Models Through Subpopulation Analysis
Zhao, Z., Ziser, Y., & Cohen, S. B. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.16 · Cited by 4
Self-citationDynamic Voting for Efficient Reasoning in Large Language Models
Xue, M., Liu, D., Lei, W., Ren, X., Yang, B., Xie, J., Zhang, Y., Peng, D., & Lv, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.203 · Cited by 10
Self-citationLearning Preference Model for LLMs via Automatic Preference Data Generation
Huang, S., Zhao, J., Li, Y., & Wang, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.570 · Cited by 4
Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study
Li, Y., Wang, H., & Zhang, C. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.4 · Cited by 11
TibetanQA2.0: Dataset with Unanswerable Questions for Tibetan Machine Reading Comprehension
Dan, Z., & Sun, Y. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0029 · Cited by 3
On the attribution of confidence to large language models
Keeling, G., & Street, W. (2025) · Inquiry · doi:10.1080/0020174x.2025.2450598 · Cited by 12
Emerging Trends in LLM Benchmarking
Desai, A. P., Prajapati, R., Ravi, T., Luqman, M., & Yadav, P. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825845 · Cited by 4
Specialized or general AI? a comparative evaluation of LLMs’ performance in legal tasks
Guo, X., Huang, Y., Wei, B., Kuang, K., Wu, Y., Gan, L., Huang, X., & Dong, X. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09460-y · Cited by 4
Benchmarking AI: Toward Inclusive Evaluation of Language Models
Jovanovic, M., & Campbell, M. (2025) · Computer · doi:10.1109/mc.2025.3575880 · Cited by 3
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Hlavnova, E., & Ruder, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.396 · Cited by 3
INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models
Renduchintala, H. S. V. N. S. K., Killamsetty, K., Bhatia, S., Aggarwal, M., Ramakrishnan, G., Iyer, R., & Krishnamurthy, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.445 · Cited by 4
XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models
Lee, D., Kadakia, A., Joshi, B., Chan, A., Liu, Z., Narahari, K., Shibuya, T., Mitani, R., Sekiya, T., Pujara, J., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.25 · Cited by 4
Self-citationModel Criticism for Long-Form Text Generation
Deng, Y., Kuleshov, V., & Rush, A. M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.815 · Cited by 11
Studying and improving reasoning in humans and machinesPreprint
Palminteri, S., Yax, N., & Anlló, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3124634/v1 · Cited by 10
Evaluating Language Model Agency through NegotiationsPreprint
Davidson, T. R., Veselovsky, V., Josifoski, M., Peyrard, M., Bosselut, A., Kosiński, M., & West, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.04536 · Cited by 39
A Causal Framework for AI Regulation and AuditingPreprint
Sharkey, L., Ghuidhir, C. N., Braun, D., Scheurer, J., Balesni, M., Bushnaq, L., Stix, C., & Hobbhahn, M. (2024) · Preprints.org · doi:10.20944/preprints202401.1424.v1 · Cited by 9
Kewenig, V. N., Lampinen, A. K., Nastase, S. A., Edwards, C., D'Elascombe, Q., Rechardt, A., Skipper, J. I., & Vigliocco, G. (2024) · Research Square · doi:10.21203/rs.3.rs-3913308/v1 · Cited by 4
Self-citationZhang, Y., & He, Z. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.691 · Cited by 3
Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal Judgment
Chang, E. Y. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.427 · Cited by 2
Self-citationEvidence from formal logical reasoning reveals that the language of thought is not natural language
Kean, H., Fung, A., Jaggers, P., Chen, J., Rule, J. S., Benn, Y., Tenenbaum, J. B., Piantadosi, S. T., Varley, R. A., & Fedorenko, E. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2520095123 · Cited by 2
Self-citationFeedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3
“These behaviors have motivated benchmarking of LLM behavior (Srivastava et al., 2022; Liang et al., 2022), automated detection of harmful outputs (Ganguli et al., 2022; Perez et al., 2022b) and methods towards reducing their harms (Bai et al., 2022).”
BackgroundBALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Kuciński, Ł., Pinto, L., Fergus, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.13543 · Cited by 2
“BigBench (Srivastava et al., 2022) further expands the scope by including a diverse set of linguistic and cognitive challenges.”
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · arXiv · Cited by 2
“For reasoning and in-context learning, we use GSM8K (Cobbe et al., 2021) and Big Bench Hard (BBH) (bench authors, 2023), with evaluation metrics consistent with LLMLin-gua (Jiang et al., 2023a).”
MethodInfluentialSelf-citationQuantifying Variance in Evaluation Benchmarks
Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., & Hupkes, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10229 · Cited by 2
“…large-scale well-established NLP benchmarks: AGIEval (Zhong et al., 2023), AI2 Reasoning Challenge (ARC-C) (Clark et al., 2018), BIG Bench (Hard) (Srivastava et al., 2022; Suzgun et al., 2022), COPA (Roemmele et al., 2011), GSM8k (Cobbe et al., 2021), Hellaswag (Zellers et al., 2019), HumanEval…”
Self-citationObservational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3
Self-citationThe Role of AI Language Assistants in Dialogic Education for Collective Intelligence
Casebourne, I., & Wegerif, R. (2024) · Intelligent systems reference library · doi:10.1007/978-3-031-71232-6_7 · Cited by 4
Temporal Extraction and Retrieval
Alonso, O., & Spaniol, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674137 · Cited by 2
Tonellotto, N. (2024) · ACM eBooks · doi:10.1145/3674127.3674130 · Cited by 3
Révauger, G., & Rident, F. (2025) · Alsic · doi:10.4000/137dy · Cited by 3
Towards Explainable Renewable Energy Communities Operations Using Generative AI
Nammouchi, A., Chaabani, C., Theocharis, A. D., & Kassler, A. (2024) · IEEE PES Innovative Smart Grid Technologies Europe · doi:10.1109/isgteurope62998.2024.10863790 · Cited by 2
Making sense of transformer success
Angius, N., Perconti, P., Plebe, A., & Acciai, A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1509338 · Cited by 2
Udrescu, A., & Popovici, D.-M. (2025) · IEEE Access · doi:10.1109/access.2025.3568028 · Cited by 2
Neural language models as content analysis tools in psychology
Acciai, A., Guerrisi, L., Plebe, A., & Suriano, R. (2025) · Philosophical Psychology · doi:10.1080/09515089.2025.2508942 · Cited by 4
In the eyes of a language model: A comprehensive examination through eye-tracking data
Dini, L., Moroni, L., Brunato⋄, D., & Dell’Orletta⋄, F. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130617 · Cited by 3
Fajčík, M., Dočekal, M., Doležal, J., Ondrej, K., Beneš, K., Kapsa, J., Smrž, P., Polok, A., Hradiš, M., Nevěřilová, Z., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.32 · Cited by 14
When it was 2024 – Generative AI in the field of digital scholarly editions
Pollin, C., Fischer, F., Sahle, P., Scholger, M., & Vogeler, G. (2025) · ARCA (Università Ca' Foscari Venezia) · doi:10.17175/2025_008 · Cited by 2
Cimiano, P., Brazda, N., Härtung, M., Starke-Knäusel, C., Schmidt, A. L., Vuono, M. C. D., Tyagi, A. S., Gottowik, J., Rodriguez‐Esteban, R., Collins, B., et al. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2025.1703923 · Cited by 2
Li, J., Deng, H., Zhang, Y., Zhang, K., Shao, T., Zhao, T., Wang, W., Jin, Z. G., Li, G., Liu, Y., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797106 · Cited by 5
Equilibration of Coordinating Imitation and Best-Response Dynamics
Hasheminejad, N., & Ramazi, P. (2023) · IEEE Control Systems Letters · doi:10.1109/lcsys.2023.3292049 · Cited by 2
Evaluation of Large Language Models Using an Indian Language LGBTI+ Lexicon
Joshi, A., & Rawat, S. M. (2023) · AI Ethics Journal · doi:10.47289/aiej20231109 · Cited by 3
Bootstrapping a Conversational Guide for Colonoscopy Prep
Arya, P., Bloomquist, M., Chakraborty, S., Perrault, A., Schuler, W., Fosler‐Lussier, E., & White, M. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.38 · Cited by 1
Evaluation of Medium-Sized Language Models in German and English Language
Peinl, R., & Wirth, J. (2024) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2024.13101 · Cited by 2
Beyond the hype: A critical look at the motivations driving automated driving systems research
Almlöf, E. (2024) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2024.101075 · Cited by 2
Modeling Patterns for Neural-Symbolic Reasoning Using Energy-based Models
Dickens, C., Pryor, C., & Getoor, L. C. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31187 · Cited by 2
Ashihara, T., Delcroix, M., Ijima, Y., & Kashino, M. (2024) · IEEE Access · doi:10.1109/access.2024.3428364 · Cited by 3
Research on Alignment and Evaluation Methods for Large Language Models
Li, Y., Yan, Q., Ma, N., & Wan, F. (2024) · Guangdong-Hong Kong-Macao Greater Bay Area International Conference on Digital Economy and Artificial Intelligence (DEAI) · doi:10.1145/3675417.3675536 · Cited by 3
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 5
Ferro, N., & Maistro, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674132 · Cited by 3
Privacy in Information Retrieval
Soboroff, I. M. (2024) · ACM eBooks · doi:10.1145/3674127.3674139 · Cited by 2
Villuri, G., & Doboli, A. (2024) · IEEE Conference on Information Technology and Data Science (CITDS) · doi:10.1109/citds62610.2024.10791359 · Cited by 2
Understanding generative AI to harness its potentials and mini- mize risks: A perspective
Caselli, T. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.111951 · Cited by 2
Coevolution of large language models with physical models boosts advanced battery research
Li, J.-H., Hu, Y., Xia, G., Mo, W., Li, B., Jia, Y., Gao, Y., Xuan, F., Liu, H., & Lian, C. (2025) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2025.102553 · Cited by 2
The state of hybrid artificial intelligence for interstellar missions
Ellery, A. A. (2025) · Progress in Aerospace Sciences · doi:10.1016/j.paerosci.2025.101100 · Cited by 2
SGEU: enhancing LLM reasoning via backward exemplar generation and verification
Wang, Z., Zhou, X., Yang, Y., Ma, B., Wang, L., & Dong, R. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06529-8 · Cited by 1
Self-citationCan a machine talk the talk though not climb the rock? A Turing Test on rock climbing
Segersven, O., & Arminen, I. (2025) · Discourse Context & Media · doi:10.1016/j.dcm.2025.100915 · Cited by 2
A Report on the FigLang 2022 Shared Task on Understanding Figurative Language
Saakyan, A., Chakrabarty, T., Ghosh, D., & Muresan, S. (2022) · Workshop on Figurative Language Processing (FigLang) · doi:10.18653/v1/2022.flp-1.26 · Cited by 2
On General Language Understanding
Schlangen, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.591 · Cited by 3
Estimating Large Language Model Capabilities without Labeled Test Data
Fu, H. Y., Ye, Q., Xu, A., Ren, X., & Jia, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.639 · Cited by 6
Self-citationExpNote: Black-box Large Language Models are better Task Solvers with Experience Notebook
Sun, W., Yu, X., He, S., Zhao, J., & Liu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1034 · Cited by 4
Metadata
- Authors
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A., La, A., Lampinen, A., Zou, A., Jiang, A., Chen, A., Vuong, A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabassum, A., Menezes, A., Kirubarajan, A., Mullokandov, A., Sabharwal, A., Herrick, A., Efrat, A., Erdem, A., Karakaş, A., Roberts, B. R., Loe, B. S., Zoph, B., Bojanowski, B., Özyurt, B., Hedayatnia, B., Neyshabur, B., Inden, B., Stein, B., Ekmekci, B., Lin, B. Y., Howald, B., Orinion, B., Diao, C., Dour, C., Stinson, C., Argueta, C., Ramírez, C. F., Singh, C., Rathkopf, C., Meng, C., Baral, C., Wu, C., Callison-Burch, C., Waites, C., Voigt, C., Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C., Ashcraft, C., Garbacea, C., Sileo, D., Garrette, D., Hendrycks, D., Kilman, D., Roth, D., Freeman, D., Khashabi, D., Levy, D., González, D. M., Perszyk, D., Hernandez, D., Chen, D., Ippolito, D., Gilboa, D., Dohan, D., Drakard, D., Jurgens, D., Datta, D., Ganguli, D., Emelin, D., Kleyko, D., Yuret, D., Chen, D., Tam, D., Hupkes, D., Misra, D., Buzan, D., Mollo, D. C., Yang, D., Lee, D.-H., Schrader, D., Shutova, E., Cubuk, E. D., Segal, E., Hagerman, E., Barnes, E., Donoway, E., Pavlick, E., Rodola, E., Lam, E., Chu, E., Tang, E., Erdem, E., Chang, E., Chi, E. A., Dyer, E., Jerzak, E., Kim, E., Manyasi, E. E., Zheltonozhskii, E., Xia, F., Siar, F., Martínez-Plumed, F., Happé, F., Chollet, F., Rong, F., Mishra, G., Winata, G. I., de Melo, G., Kruszewski, G., Parascandolo, G., Mariani, G., Wang, G., Jaimovitch-López, G., Betz, G., Gur-Ari, G., Galijasevic, H., Kim, H., Rashkin, H., Hajishirzi, H., Mehta, H., Bogar, H., Shevlin, H., Schütze, H., Yakura, H., Zhang, H., Wong, H. M., Ng, I., Noble, I., Jumelet, J., Geissinger, J., Kernion, J., Hilton, J., Lee, J., Fisac, J. F., Simon, J. B., Koppel, J., Zheng, J., Zou, J., Kocoń, J., Thompson, J., Wingfield, J., Kaplan, J., Radom, J., Sohl-Dickstein, J., Phang, J., Wei, J., Yosinski, J., Novikova, J., Bosscher, J., Marsh, J., Kim, J., Taal, J., Engel, J., Alabi, J., Xu, J., Song, J., Tang, J., Waweru, J., Burden, J., Miller, J., Balis, J. U., Batchelder, J., Berant, J., Frohberg, J., Rozen, J., Hernandez-Orallo, J., Boudeman, J., Guerr, J., Jones, J., Tenenbaum, J. B., Rule, J. S., Chua, J., Kanclerz, K., Livescu, K., Krauth, K., Gopalakrishnan, K., Ignatyeva, K., Markert, K., Dhole, K. D., Gimpel, K., Omondi, K., Mathewson, K., Chiafullo, K., Shkaruta, K., Shridhar, K., McDonell, K., Richardson, K., Reynolds, L., Gao, L., Zhang, L., Dugan, L., Qin, L., Contreras-Ochando, L., Morency, L.-P., Moschella, L., Lam, L., Noble, L., Schmidt, L., He, L., Colón, L. O., Metz, L., Şenel, L. K., Bosma, M., Sap, M., ter Hoeve, M., Farooqi, M., Faruqui, M., Mazeika, M., Baturan, M., Marelli, M., Maru, M., Quintana, M. J. R., Tolkiehn, M., Giulianelli, M., Lewis, M., Potthast, M., Leavitt, M. L., Hagen, M., Schubert, M., Baitemirova, M. O., Arnaud, M., McElrath, M., Yee, M. A., Cohen, M., Gu, M., Ivanitskiy, M., Starritt, M., Strube, M., Swędrowski, M., Bevilacqua, M., Yasunaga, M., Kale, M., Cain, M., Xu, M., Suzgun, M., Walker, M., Tiwari, M., Bansal, M., Aminnaseri, M., Geva, M., Gheini, M., T, M. V., Peng, N., Chi, N. A., Lee, N., Krakover, N. G.-A., Cameron, N., Roberts, N., Doiron, N., Martinez, N., Nangia, N., Deckers, N., Muennighoff, N., Keskar, N. S., Iyer, N. S., Constant, N., Fiedel, N., Wen, N., Zhang, O., Agha, O., Elbaghdadi, O., Levy, O., Evans, O., Casares, P. A. M., Doshi, P., Fung, P., Liang, P. P., Vicol, P., Alipoormolabashi, P., Liao, P., Liang, P., Chang, P., Eckersley, P., Htut, P. M., Hwang, P., Miłkowski, P., Patil, P., Pezeshkpour, P., Oli, P., Mei, Q., Lyu, Q., Chen, Q., Banjade, R., Rudolph, R. E., Gabriel, R., Habacker, R., Risco, R., Millière, R., Garg, R., Barnes, R., Saurous, R. A., Arakawa, R., Raymaekers, R., Frank, R., Sikand, R., Novak, R., Sitelew, R., LeBras, R., Liu, R., Jacobs, R., Zhang, R., Salakhutdinov, R., Chi, R., Lee, R., Stovall, R., Teehan, R., Yang, R., Singh, S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff, S., Gehrmann, S., Schuster, S., Sadeghi, S., Hamdan, S., Zhou, S., Srivastava, S., Shi, S., Singh, S., Asaadi, S., Gu, S. S., Pachchigar, S., Toshniwal, S., Upadhyay, S., Shyamolima, Debnath, Shakeri, S., Thormeyer, S., Melzi, S., Reddy, S., Makini, S. P., Lee, S.-H., Torene, S., Hatwar, S., Dehaene, S., Divic, S., Ermon, S., Biderman, S., Lin, S., Prasad, S., Piantadosi, S. T., Shieber, S. M., Misherghi, S., Kiritchenko, S., Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L., Desbordes, T., Rothschild, T., Phan, T., Wang, T., Nkinyili, T., Schick, T., Kornev, T., Tunduny, T., Gerstenberg, T., Chang, T., Neeraj, T., Khot, T., Shultz, T., Shaham, U., Misra, V., Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V., Prabhu, V. U., Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, Y., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z., Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z., & Wu, Z.
- Affiliations
- Shammie
- ORCID
- Abid, Abubakar
- Brown, Adam R.
- Gupta, Aditya
- Power, Alethea
- Warstadt, Alex
- Kocurek, Alexander W.
- Safaya, Ali
- Tazarv, Ali
- Xiang, Alice
- Parrish, Alicia
- Nie, Allen
- Dsouza, Amanda
- Rahane, Ameet
- Andreassen, Anders
- Madotto, Andrea
- Santilli, Andrea
- Dai, Andrew
- Lampinen, Andrew
- Gupta, Animesh
- Norelli, Antonio
- Venkatesh, Anu
- Mullokandov, Asher
- Erdem, Aykut
- Loe, Bao Sheng
- Bojanowski, Bartłomiej
- Inden, Benjamin
- Stein, Benno
- Howald, Blake
- Stinson, Catherine
- Singh, Chandan
- Rathkopf, Charles
- Meng, Chenlin
- Baral, Chitta
- Callison-Burch, Chris
- Voigt, Christian
- Manning, Christopher D.
- Potts, Christopher
- Siro, Clemencia
- Raffel, Colin
- Sileo, Damien
- Roth, Dan
- Khashabi, Daniel
- Chen, Danqi
- Ippolito, Daphne
- Jurgens, David
- Datta, Debajyoti
- Ganguli, Deep
- Kleyko, Denis
- Chen, Derek
- Misra, Diganta
- Mollo, Dimitri Coelho
- Yang, Diyi
- Lee, Dong-Ho
- Shutova, Ekaterina
- Cubuk, Ekin Dogus
- Segal, Elad
- Barnes, Elizabeth
- Donoway, Elizabeth
- Pavlick, Ellie
- Erdem, Erkut
- Dyer, Ethan
- Jerzak, Ethan
- Kim, Ethan
- Zheltonozhskii, Evgenii
- Mishra, Gaurav
- Betz, Gregor
- Kim, Hannah
- Hajishirzi, Hannaneh
- Mehta, Harsh
- Shevlin, Henry
- Schütze, Hinrich
- Yakura, Hiromu
- Zhang, Hongming
- Geissinger, Jack
- Lee, Jaehoon
- Fisac, Jaime Fernández
- Simon, James B.
- Koppel, James
- Zou, James
- Kocoń, Jan
- Phang, Jason
- Novikova, Jekaterina
- Kim, Jeremy
- Xu, Jiacheng
- Song, Jiaming
- Burden, John
- Rozen, Jos
- Tenenbaum, Joshua B.
- Rule, Joshua S.
- Kanclerz, Kamil
- Livescu, Karen
- Krauth, Karl
- Dhole, Kaustubh D.
- Mathewson, Kory
- Richardson, Kyle
- Zhang, Li
- Dugan, Liam
- Contreras-Ochando, Lidia
- Morency, Louis-Philippe
- Moschella, Luca
- Lam, Lucas
- Şenel, Lütfi Kerem
- Sap, Maarten
- Farooqi, Maheen
- Marelli, Marco
- Tolkiehn, Marie
- Giulianelli, Mario
- Lewis, Martha
- Potthast, Martin
- Leavitt, Matthew L.
- Hagen, Matthias
- Yee, Michael A.
- Ivanitskiy, Michael
- Strube, Michael
- Swędrowski, Michał
- Yasunaga, Michihiro
- Bansal, Mohit
- Geva, Mor
- T, Mukund Varma
- Peng, Nanyun
- Chi, Nathan A.
- Lee, Nayeon
- Deckers, Niklas
- Muennighoff, Niklas
- Keskar, Nitish Shirish
- Iyer, Niveditha S.
- Wen, Nuan
- Zhang, Oliver
- Levy, Omer
- Casares, Pablo Antonio Moreno
- Fung, Pascale
- Liang, Paul Pu
- Liao, Peiyuan
- Liang, Percy
- Miłkowski, Piotr
- Pezeshkpour, Pouya
- Oli, Priti
- Mei, Qiaozhu
- Banjade, Rabin
- Habacker, Rahel
- Millière, Raphaël
- Saurous, Rif A.
- Arakawa, Riku
- Frank, Robert
- Zhang, Rui
- Salakhutdinov, Ruslan
- Teehan, Ryan
- Singh, Sahib
- Mohammad, Saif M.
- Anand, Sajant
- Dillavou, Sam
- Wiseman, Sam
- Gruetter, Samuel
- Schoenholz, Samuel S.
- Rous, Sarah A.
- Ghosh, Sayan
- Casey, Sean
- Bischoff, Sebastian
- Gehrmann, Sebastian
- Schuster, Sebastian
- Hamdan, Shadi
- Zhou, Sharon
- Srivastava, Shashank
- Shi, Sherry
- Singh, Shikhar
- Asaadi, Shima
- Gu, Shixiang Shane
- Melzi, Simone
- Reddy, Siva
- Makini, Sneha Priscilla
- Lee, Soo-Hwan
- Dehaene, Stanislas
- Divic, Stefan
- Ermon, Stefano
- Biderman, Stella
- Lin, Stephanie
- Piantadosi, Steven T.
- Shieber, Stuart M.
- Kiritchenko, Svetlana
- Mishra, Swaroop
- Linzen, Tal
- Schuster, Tal
- Li, Tao
- Wu, Te-Lin
- Desbordes, Théo
- Wang, Tianle
- Nkinyili, Tiberius
- Tunduny, Titus
- Gerstenberg, Tobias
- Chang, Trenton
- Shaham, Uri
- Demberg, Vera
- Ramasesh, Vinay
- Prabhu, Vinay Uday
- Padmakumar, Vishakh
- Srikumar, Vivek
- Zhang, William
- Vossen, Wout
- Ren, Xiang
- Zhao, Xinran
- Wu, Xinyi
- Shen, Xudong
- Yaghoobzadeh, Yadollah
- Lakretz, Yair
- Song, Yangqiu
- Choi, Yejin
- Yang, Yichi
- Hao, Yiding
- Chen, Yifu
- Belinkov, Yonatan
- Hou, Yu
- Hou, Yufang
- Bai, Yuntao
- Zhao, Zhuoye
- Wang, Zijian
- Wang, Zijie J.
- Wu, Ziyi
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2206.04615
- OpenAlex
- W4281690148
- Semantic Scholar
- bd1331b233e84bab7eba503abc60b31ac08e7881
- Cited by
- 569
- References
- 0
- Citations by year
- 2026: 29 · 2025: 114 · 2024: 169 · 2023: 232 · 2022: 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- model scaling, sparse transformer, GPT, model calibration, social bias, commonsense reasoning, mathematical reasoning, prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
