Embers of autoregression show how large language models are shaped by the problem they are trained to solve
Open accessThe DOI matches its Crossref record; 97 of the 99 references it cites were found in a registry or an index; 15 of the 34 sources its claims cite were checked for support.
Abstract
The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that to develop a holistic understanding of these systems, we must consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts, we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. Using this approach—which we call the teleological approach—we identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. To test our predictions, we evaluate five LLMs (GPT-3.5, GPT-4, Claude 3, Llama 3, and Gemini 1.0) on 11 tasks, and we find robust evidence that LLMs are influenced by probability in the hypothesized ways. Many of the experiments reveal surprising failure modes. For instance, GPT-4’s accuracy at decoding a simple cipher is 51% when the output is a high-probability sentence but only 13% when it is low-probability, even though this task is a deterministic one for which probability should not matter. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system—one that has been shaped by its own particular set of pressures.
Overview
CoChat Summary
TL;DR
The paper argues that large language models can be understood teleologically as statistical next-word prediction systems shaped by Internet text, and it tests three predicted sensitivities: output probability, input probability, and task frequency. Across five LLMs and eleven tasks, accuracy is often higher for high-probability outputs and common task variants, and sometimes for high-probability inputs, even on deterministic tasks.
Key findings
- Output probability: significant effect in all cases except Gemini 1.0 decoding shift ciphers (0% accuracy) and Claude 3 forming acronyms (P=0.38). GPT-4’s article-swapping accuracy ranged from 0.02 on low-probability examples to 0.83 on high-probability examples. Counting accuracy spiked at multiples of 10; for GPT-4, accuracy correlated more with number frequency than magnitude (absolute coefficients 0.84 vs. 0.70), and for Claude 3, 0.85 vs. 0.66. In shift-cipher errors, GPT-3.5 and GPT-4 produced a regularized high-probability sentence more often than the correct answer.
- Input probability: little or no effect on most tasks, but the birthday task showed a large effect (P<10−15 for all models). GPT-4 had higher accuracy when shift-cipher inputs were high-probability than low-probability; GPT-3.5 showed no significant input-probability effect.
- Task probability: across seven common-vs-rare task pairs, models performed substantially better on common variants. For shift ciphers, all models except Gemini 1.0 spiked at rot-13; GPT-4 scored ≥0.50 on the three highest-frequency shifts (1, 3, 13) but <0.03 on all other shifts, and rot-13 beat rot-12 (P<0.01) for GPT-3.5, GPT-4, and Claude 3. Pig Latin encoding was better for the common variant (P<0.01) for GPT-3.5, GPT-4, and Claude 3; decoding favored Pig Latin for all five models but was significant only for GPT-4 and Llama 3 (P<0.05). First-letter acronyms beat second-letter acronyms for all five models (P<10−4). The common linear function f(x)=(9/5)x+32 beat the rare f(x)=(7/5)x+31 for GPT-3.5, GPT-4, and Claude 3 (P<0.01; Llama 3 P=0.22). Alphabetical sorting beat reverse alphabetical sorting for all five models (P<10−5), paralleling a 150× corpus-frequency difference.
- Overall: chain-of-thought and step-by-step prompting, scaling, and few-shot learning can raise overall accuracy, but the qualitative sensitivities to task frequency and example probability remained.
Methods
- The paper uses a teleological/Bayesian analysis of next-word prediction to derive predictions about output probability, input probability, and task probability. It evaluated five models: gpt-3.5-turbo-0613, gpt-4-0613, claude-3-opus-20240229, gemini-1.0-pro-001, and llama-3-70b-chat-hf, accessed via APIs at temperature 0.0. Preliminary tests with OLMo and Llama 2 were too low to yield trends (0% on most tasks).
- Eleven tasks were used: article swapping, reversal, counting, acronyms, linear function, multiplication, sorting, keyboard cipher, shift cipher, Pig Latin, and birthdays. High-probability sentences came from Global Voices; medium-probability sentences used RoBERTa word replacements; low-probability sentences shuffled medium-probability examples except first and last words. Words came from the CMU Pronouncing Dictionary; acronym lists had 7 words, sorting lists 10–20 words, counting lists 1–100 words, linear-function inputs were 0–999, multiplication inputs 100–999, and birthdays came from WikiBio. Sample sizes were 1,000 per condition for acronyms, 30 per number for counting, and 100 per condition for other tasks. Models received basic prompts without tuning.
- Statistical tests used logistic regressions with correctness as the outcome; predictors generally included input probability, output probability, input length, output length, and task variant. Output probabilities were estimated using GPT-2, and task frequencies were estimated from C4 and the Pile.
Limitations
- Paper states: the analysis focuses only on next-word prediction and leaves instruction tuning for future work; it considers only text-processing LLMs, not augmented models with image or code-execution abilities; it uses basic prompting, though chain-of-thought and step-by-step prompting can improve performance while preserving the reported trends; it does not make strong claims about qualitative human differences because there is no fair human comparison; and it does not claim LLMs cannot perform the tasks, only that some tasks and examples are harder than others.
- Paper states/evident: tasks were chosen adversarially to push models into low-probability situations and ensure nonnegligible error rates, so the results are not a general capability benchmark. Some models showed floor effects (e.g., Gemini 1.0 at 0% on shift ciphers; OLMo and Llama 2 at 0% on most tasks), and corpus-frequency estimates are proxies for training data. Sample sizes also varied, including only 30 examples per number in the counting task.
- Evident: the study covers only five API models at specific versions and temperature 0.0, does not separate instruction-tuned from pretrained behavior, lacks a human baseline, and uses a limited set of eleven tasks; some significance tests excluded models because of floor effects.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
11 Valid · 3 Not valid · 8 Not checked · 107 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Note that many recent LLMs are not solely trained on next-word prediction but also go through a training phase based on instruction tuning (22), which aims to align model behavior with human preferences.”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > A Teleological Approach to Understanding LLMs > What Problem Do LLMs Solve?.
What the sources say
- Supportedcites [22] L. Ouyang, Training language models to follow instructions with human feedback. NeurIPS 35, 27730– 27744 (2022). source
The Introduction (§1) describes fine-tuning GPT-3 to follow instructions using human preferences as a reward signal, rather than relying only on next-token prediction. Related Work (§2) also describes a range of studies that fine-tune language models with instructions. The paper qualifies its own alignment result as matching the stated preferences of its labelers and researchers, not human values broadly. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“they found that LLMs performed better on the default version of a task (e.g., executing Python code under the true assumption that Python uses 0-based indexing) than on a counterfactual version of the task (e.g., falsely assuming that Python uses 1-based indexing).”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > Sensitivity to Task Probability > Summary of Task Probability Effects.
What the sources say
- Supportedcites [42] Z. Wu et al., “Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks” in Proceedings of NAACL (2024), pp. 1819–1862. source
The Results (§4) state that language models consistently performed substantially worse on counterfactual task variants than on default variants. Appendix A.2 confirms that the programming task compared ordinary 0-based indexing with counterfactual 1-based indexing. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Based on an analysis of the C4 corpus (36), we found that the three most common shift levels are rot-1, rot-3, and rot-13”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > Sensitivity to Task Probability > Shift Ciphers.
What the sources say
- Supportedcites [36] C. Raffel, Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 5485– 5551 (2020). source
Section 2.2 (p. 6) identifies the dataset as the Colossal Clean Crawled Corpus, or C4. The citing authors attribute the shift-frequency finding to their own analysis; this citation identifies the corpus they analyzed, not the source of that finding. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Based on an analysis of the C4 corpus, we estimate that alphabetical order is 150 times more common than reverse alphabetical order in Internet text.”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > Sensitivity to Task Probability > Sorting.
What the sources say
- Supportedcites [36] C. Raffel, Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 5485– 5551 (2020). source
Section 2.2 (p. 6) identifies C4 as the authors’ corpus of cleaned web text. The citing sentence presents the 150-times estimate as its own analysis, not as a result reported by Raffel et al.; this source supports the citation’s identification of C4, but does not verify the estimate. (LLM, conf=0.88)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“a human’s ease of processing a sentence is influenced by that sentence’s probability (66–68)”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > Discussion > Comparing Models to Humans.
What the sources say
- Not checked — no full text of the sourcecites [66] S. F. Ehrlich, K. Rayner, Contextual effects on word perception and eye movements during reading. J. Verb. Learn. Verb. Behav. 20, 641– 655 (1981). source
- Partly supportedcites [67] N. J. Smith, R. Levy, The effect of word predictability on reading time is logarithmic. Cognition 128, 302– 319 (2013). 23747651 PMC3709001 sourceExperimental — accuracy not yet measured
The Discussion reports that word predictability affects word comprehension in context, and the paper relates that effect to reading time. It does not establish that the probability of an entire sentence affects how easily a human processes that sentence; its measured probabilities are for words in context. (LLM, conf=0.86)
- Partly supportedcites [68] E. Gibson, L. Bergen, S. T. Piantadosi, Rational integration of noisy evidence and prior semantic expectations in sentence interpretation. Proc. Natl. Acad. Sci. U.S.A. 110, 8051– 8056 (2013). 23637344 PMC3657782 sourceExperimental — accuracy not yet measured
The paper’s Discussion says that comprehenders use expectations about likely sentences to arrive at an interpretation, supporting the claim that sentence probability influences comprehension. But the Results measure which interpretation participants chose, not how easy a sentence was to process; the claim’s processing-ease component is not established. (LLM, conf=0.83)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
65 more claims
Create a free account to see all 129 claims and every source verdict.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 42
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, models and humans also have different capabilities and consequently different constraints for some tasks (Griffiths, 2020;Shiffrin & Mitchell, 2023;McCoy et al., 2024).”
From Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse (2024) · cites this paper as McCoy et al., 2024
In the citing paper
“The disparity between LLM performance on mARC-QA and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities19,54–57.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [54]
In the citing paper
“This interpretation is reinforced by studies that have shown LLMs’ limited robustness and increased hallucination rates in low-probability contexts, where reliance on surface-level statistical correlations proves insufficient19,54,58.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [54]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
The spandrels of San Marco and the Panglossian paradigm: a critique of the adaptationist programme
Gould, S. J., & Lewontin, R. C. (1979) · Proceedings of the Royal Society of London. Series B. Biological Sciences · doi:10.1098/rspb.1979.0086 · Cited by 7,770
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
On the proper treatment of connectionism
Smolensky, P. (1988) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00052432 · Cited by 2,441
Toward a Universal Law of Generalization for Psychological Science
Shepard, R. N. (1987) · Science · doi:10.1126/science.3629243 · Cited by 2,416
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Expectation-based syntactic comprehension
Levy, R. (2008) · Cognition · doi:10.1016/j.cognition.2007.05.006 · Cited by 2,202
The Adaptive Character of Thought
Massaro, D. W., Friedman, D., & Anderson, J. R. (1991) · The American Journal of Psychology · doi:10.2307/1423253 · Cited by 2,174
Self-citationLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Cognition, Evolution, and Behavior
Shettleworth, S. J. (2009) · doi:10.1093/oso/9780195319842.001.0001 · Cited by 1,453
Distributed representations, simple recurrent networks, and grammatical structure
Elman, J. L. (1991) · Machine Learning · doi:10.1007/bf00114844 · Cited by 1,354
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Self-citationA probabilistic earley parser as a psycholinguistic model
Hale, J. (2001) · Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001 - NAACL '01 · doi:10.3115/1073336.1073357 · Cited by 1,150
Mechanization in problem solving: The effect of Einstellung.
Luchins, A. S. (1942) · Psychological Monographs · doi:10.1037/h0093502 · Cited by 1,143
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Self-citationBeyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Are We Smart Enough to Know How Smart Animals Are
de Waal, F. Β. Μ. (2016) · Medical Entomology and Zoology · Cited by 876
Contextual effects on word perception and eye movements during reading
Ehrlich, S. F., & Rayner, K. (1981) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(81)90220-6 · Cited by 796
The effect of word predictability on reading time is logarithmic
Smith, N. J., & Levy, R. (2013) · Cognition · doi:10.1016/j.cognition.2013.02.013 · Cited by 725
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559
The neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480
Cognition, Evolution, and Behavior
Emery, N. J. (2010) · Animal Behaviour · doi:10.1016/j.anbehav.2010.05.017 · Cited by 465
Neural Text Generation from Structured Data with Application to the Biography Domain
Lebret, R., Grangier, D., & Auli, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1128 · Cited by 444
Rational integration of noisy evidence and prior semantic expectations in sentence interpretation
Gibson, E., Bergen, L., & Piantadosi, S. T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1216438110 · Cited by 407
Emergent analogical reasoning in large language models
Webb, T., Holyoak, K. J., & Lu, H. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01659-w · Cited by 404
Talking about Large Language Models
Shanahan, M. (2024) · Communications of the ACM · doi:10.1145/3624724 · Cited by 381
Rethinking Eliminative Connectionism
Marcus, G. F. (1998) · Cognitive Psychology · doi:10.1006/cogp.1998.0694 · Cited by 348
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
The debate over understanding in AI’s large language models
Mitchell, M., & Krakauer, D. C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2215907120 · Cited by 340
Emergent linguistic structure in artificial neural networks trained by self-supervision
Manning, C. D., Clark, K., Hewitt, J., Khandelwal, U., & Levy, O. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907367117 · Cited by 339
Faith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3081 · Cited by 335
Functional explanation and the function of explanation
LOMBROZO, T., & CAREY, S. (2006) · Cognition · doi:10.1016/j.cognition.2004.12.009 · Cited by 300
Godfrey-Smith, P. (2001) · Adaptationism and Optimality · doi:10.1017/cbo9780511609084.012 · Cited by 259
Compositionality Decomposed: How do Neural Networks Generalise?
Hupkes, D., Dankers, V., Mul, M., & Bruni, E. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11674 · Cited by 194
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2304.05335 · Cited by 178
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01373 · Cited by 160
Entity Tracking in Language Models
Kim, N., & Schuster, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.213 · Cited by 159
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
Kim, N., & Linzen, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.731 · Cited by 153
Performance vs. competence in human–machine comparisons
Firestone, C. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905334117 · Cited by 150
Large Language Models Struggle to Learn Long-Tail Knowledge
Kandpal, N., Deng, H., Roberts, A., Wallace, E., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2211.08411 · Cited by 143
Language Models are General-Purpose InterfacesPreprint
Hao, Y., Song, H., Li, D., Huang, S., Chi, Z., Wang, W., Ma, S., & Wei, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.06336 · Cited by 136
AI and the Everything in the Whole Wide World Benchmark
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021) · arXiv · doi:10.48550/arxiv.2111.15366 · Cited by 127
Faith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. (2023) · arXiv · Cited by 124
Krogman, W. M. (1951) · Scientific American · doi:10.1038/scientificamerican1251-54 · Cited by 101
Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning TasksPreprint
Mitchell, M., Palmarini, A. B., & Moskvichev, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.09247 · Cited by 100
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89
Understanding Human Intelligence through Human Limitations
Griffiths, T. L. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.09.001 · Cited by 88
Self-citationDistributed Representations, Simple Recurrent Networks, And Grammatical Structure
Elman, J. L. (1991) · Machine Learning · doi:10.1023/a:1022699029236 · Cited by 84
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2023) · arXiv · doi:10.48550/arxiv.2301.06627 · Cited by 80
Language Models are Open Knowledge Graphs
Wang, C., Liu, X., & Song, D. (2020) · arXiv · doi:10.48550/arxiv.2010.11967 · Cited by 79
Lake, B. M., & Baroni, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00350 · Cited by 76
How do we know how smart AI systems are?
Mitchell, M. (2023) · Science · doi:10.1126/science.adj5957 · Cited by 71
Probing the psychology of AI models
Shiffrin, R., & Mitchell, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2300963120 · Cited by 70
OLMo: Accelerating the Science of Language Models
Groeneveld, D., Beltagy, I., Walsh, E., Bhagia, A., Kinney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.841 · Cited by 70
Language models, like humans, show content effects on reasoning tasks
Lampinen, A. K., Dasgupta, I., Chan, S. C. Y., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L., & Hill, F. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae233 · Cited by 69
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J., & Hill, F. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1371 · Cited by 69
Language Models are Multilingual Chain-of-Thought Reasoners
Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.03057 · Cited by 60
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58
What do tokens know about their characters and how do they know it?
Kaushal, A., & Mahowald, K. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.179 · Cited by 57
Language models show human-like content effects on reasoning tasks
Dasgupta, I., Lampinen, A. K., Chan, S. C. Y., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L., & Hill, F. (2022) · arXiv · doi:10.48550/arxiv.2207.07051 · Cited by 53
LMentry: A Language Model Benchmark of Elementary Language Tasks
Efrat, A., Honovich, O., & Levy, O. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.666 · Cited by 48
Coelho Mollo, D., & Millière, R. (2026) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2026.12307 · Cited by 48
Mollo, D. C., & Millière, R. (2023) · arXiv · doi:10.48550/arxiv.2304.01481 · Cited by 47
Editorial: The Perils of Being Bipedal
Latimer, B. (2005) · Annals of Biomedical Engineering · doi:10.1007/s10439-005-8957-8 · Cited by 43
Wong, L., Grand, G., Lew, A. K., Goodman, N. D., Mansinghka, V. K., Andreas, J., & Tenenbaum, J. B. (2023) · arXiv · doi:10.48550/arxiv.2306.12672 · Cited by 36
Lampinen, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00525 · Cited by 32
Lampinen, A. K. (2022) · arXiv · doi:10.48550/arxiv.2210.15303 · Cited by 32
Transferring Inductive Biases through Knowledge Distillation
Abnar, S., Dehghani, M., & Zuidema, W. (2020) · arXiv · doi:10.48550/arxiv.2006.00555 · Cited by 30
Mechanistic versus Functional Understanding
Lombrozo, T., & Wilkenfeld, D. (2019) · Varieties of Understanding · doi:10.1093/oso/9780190860974.003.0011 · Cited by 29
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens
Itzhak, I., & Levy, O. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.373 · Cited by 28
Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models
Blevins, T., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.233 · Cited by 27
Holtzman, A., West, P., & Zettlemoyer, L. (2023) · Journal of Social Computing · doi:10.48550/arxiv.2308.00189 · Cited by 24
Misra, K., Rayz, J., & Ettinger, A. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.213 · Cited by 15
Adversarial Policies Beat Superhuman Go AIs
Wang, T. T., Gleave, A., Tseng, T., Pelrine, K., Belrose, N., Miller, J., Dennis, M. D., Duan, Y., Pogrebniak, V., Levine, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.00241 · Cited by 12
Rational Simplification and Rigidity in Human Planning
Ho, M. K., Cohen, J. D., & Griffiths, T. L. (2023) · Psychological Science · doi:10.1177/09567976231200547 · Cited by 11
Self-citationOLMo: Accelerating the Science of Language Models
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.00838 · Cited by 10
Can You Hear Me Now? Sensitive Comparisons of Human and Machine Perception
Lepori, M. A., & Firestone, C. (2022) · Cognitive Science · doi:10.1111/cogs.13191 · Cited by 7
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Research works printed but not linked: 8
- Anthropic, Introducing the next generation of Claude. Anthropic [Preprint] (2024). https://www.anthropic.com/news/claude-3-family. Accessed 28 August 2024.
- D. Marr, Vision (W.H. Freeman, 1982).
- A. Radford et al., Language models are unsupervised multitask learners. OpenAI [Preprint] (2019). https://openai.com/index/better-language-models. Accessed 28 August 2024.
- Suetonius, The Twelve Caesars (R. Worthington, 1883), vol. 121, Translator: Alexander Thomson.
- D. Kahn, The Codebreakers (Scribner, 1967).
- S. Singh, The Code Book (Doubleday, 1999).
- B. Vaux, A. I. Nevins, “Underdetermination in language games: Survey & analysis of Pig Latin dialects” in 77th Annual Meeting of the LSA (Atlanta, GA, 2003), pp. 2–5.
- BIG-bench authors, Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res. 05/2023, 1–95 (2023).
Cited by
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2024) · arXiv · doi:10.48550/arxiv.2410.05229 · Cited by 714
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., & Murthy, A. (2024) · arXiv · doi:10.48550/arxiv.2402.01817 · Cited by 237
Theory Is All You Need: AI, Human Cognition, and Causal Reasoning
Felin, T., & Holweg, M. (2024) · doi:10.2139/ssrn.4737265 · Cited by 152
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
Metadata
- Authors
- McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L.
- Affiliations
- Princeton University (US)
- Corresponding author
- McCoy, R. Thomas
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2024-10-04
- Volume
- 121
- Issue
- 41
- Pages
- e2322420121-e2322420121
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC11474099
- OpenAlex
- W4403122979
- Cited by
- 96
- References
- 66
- Citations by year
- 2026: 48 · 2025: 39 · 2024: 9
- FWCI
- 14.11
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (2204152)
- Princeton University (N/A)
- U.S. Department of Defense (N/A)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, teleological approach, next-word prediction, LLM failure modes, LLM evaluation, autoregression, GPT-4, Claude 3, Llama 3
- MeSH terms
- Humans, Language, Models, Theoretical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.