The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 15 of the 34 sources its claims cite were checked for support.
Embers of autoregression show how large language models are shaped by the problem they are trained to solve
Open accessAbstract
The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that to develop a holistic understanding of these systems, we must consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts, we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. Using this approach—which we call the teleological approach—we identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. To test our predictions, we evaluate five LLMs (GPT-3.5, GPT-4, Claude 3, Llama 3, and Gemini 1.0) on 11 tasks, and we find robust evidence that LLMs are influenced by probability in the hypothesized ways. Many of the experiments reveal surprising failure modes. For instance, GPT-4’s accuracy at decoding a simple cipher is 51% when the output is a high-probability sentence but only 13% when it is low-probability, even though this task is a deterministic one for which probability should not matter. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system—one that has been shaped by its own particular set of pressures.
Overview
CoChat Summary
TL;DR
The paper argues that large language models can be understood teleologically as statistical next-word prediction systems shaped by Internet text, and it tests three predicted sensitivities: output probability, input probability, and task frequency. Across five LLMs and eleven tasks, accuracy is often higher for high-probability outputs and common task variants, and sometimes for high-probability inputs, even on deterministic tasks.
Key findings
- Output probability: significant effect in all cases except Gemini 1.0 decoding shift ciphers (0% accuracy) and Claude 3 forming acronyms (P=0.38). GPT-4’s article-swapping accuracy ranged from 0.02 on low-probability examples to 0.83 on high-probability examples. Counting accuracy spiked at multiples of 10; for GPT-4, accuracy correlated more with number frequency than magnitude (absolute coefficients 0.84 vs. 0.70), and for Claude 3, 0.85 vs. 0.66. In shift-cipher errors, GPT-3.5 and GPT-4 produced a regularized high-probability sentence more often than the correct answer.
- Input probability: little or no effect on most tasks, but the birthday task showed a large effect (P<10−15 for all models). GPT-4 had higher accuracy when shift-cipher inputs were high-probability than low-probability; GPT-3.5 showed no significant input-probability effect.
- Task probability: across seven common-vs-rare task pairs, models performed substantially better on common variants. For shift ciphers, all models except Gemini 1.0 spiked at rot-13; GPT-4 scored ≥0.50 on the three highest-frequency shifts (1, 3, 13) but <0.03 on all other shifts, and rot-13 beat rot-12 (P<0.01) for GPT-3.5, GPT-4, and Claude 3. Pig Latin encoding was better for the common variant (P<0.01) for GPT-3.5, GPT-4, and Claude 3; decoding favored Pig Latin for all five models but was significant only for GPT-4 and Llama 3 (P<0.05). First-letter acronyms beat second-letter acronyms for all five models (P<10−4). The common linear function f(x)=(9/5)x+32 beat the rare f(x)=(7/5)x+31 for GPT-3.5, GPT-4, and Claude 3 (P<0.01; Llama 3 P=0.22). Alphabetical sorting beat reverse alphabetical sorting for all five models (P<10−5), paralleling a 150× corpus-frequency difference.
- Overall: chain-of-thought and step-by-step prompting, scaling, and few-shot learning can raise overall accuracy, but the qualitative sensitivities to task frequency and example probability remained.
Methods
- The paper uses a teleological/Bayesian analysis of next-word prediction to derive predictions about output probability, input probability, and task probability. It evaluated five models: gpt-3.5-turbo-0613, gpt-4-0613, claude-3-opus-20240229, gemini-1.0-pro-001, and llama-3-70b-chat-hf, accessed via APIs at temperature 0.0. Preliminary tests with OLMo and Llama 2 were too low to yield trends (0% on most tasks).
- Eleven tasks were used: article swapping, reversal, counting, acronyms, linear function, multiplication, sorting, keyboard cipher, shift cipher, Pig Latin, and birthdays. High-probability sentences came from Global Voices; medium-probability sentences used RoBERTa word replacements; low-probability sentences shuffled medium-probability examples except first and last words. Words came from the CMU Pronouncing Dictionary; acronym lists had 7 words, sorting lists 10–20 words, counting lists 1–100 words, linear-function inputs were 0–999, multiplication inputs 100–999, and birthdays came from WikiBio. Sample sizes were 1,000 per condition for acronyms, 30 per number for counting, and 100 per condition for other tasks. Models received basic prompts without tuning.
- Statistical tests used logistic regressions with correctness as the outcome; predictors generally included input probability, output probability, input length, output length, and task variant. Output probabilities were estimated using GPT-2, and task frequencies were estimated from C4 and the Pile.
Limitations
- Paper states: the analysis focuses only on next-word prediction and leaves instruction tuning for future work; it considers only text-processing LLMs, not augmented models with image or code-execution abilities; it uses basic prompting, though chain-of-thought and step-by-step prompting can improve performance while preserving the reported trends; it does not make strong claims about qualitative human differences because there is no fair human comparison; and it does not claim LLMs cannot perform the tasks, only that some tasks and examples are harder than others.
- Paper states/evident: tasks were chosen adversarially to push models into low-probability situations and ensure nonnegligible error rates, so the results are not a general capability benchmark. Some models showed floor effects (e.g., Gemini 1.0 at 0% on shift ciphers; OLMo and Llama 2 at 0% on most tasks), and corpus-frequency estimates are proxies for training data. Sample sizes also varied, including only 30 examples per number in the counting task.
- Evident: the study covers only five API models at specific versions and temperature 0.0, does not separate instruction-tuned from pretrained behavior, lacks a human baseline, and uses a limited set of eleven tasks; some significance tests excluded models because of floor effects.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
11 Valid · 3 Not valid · 8 Not checked · 107 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Note that many recent LLMs are not solely trained on next-word prediction but also go through a training phase based on instruction tuning (22), which aims to align model behavior with human preferences.”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > A Teleological Approach to Understanding LLMs > What Problem Do LLMs Solve?.
What the sources say
- Supportedcites [22] L. Ouyang, Training language models to follow instructions with human feedback. NeurIPS 35, 27730– 27744 (2022). source
The Introduction (§1) describes fine-tuning GPT-3 to follow instructions using human preferences as a reward signal, rather than relying only on next-token prediction. Related Work (§2) also describes a range of studies that fine-tune language models with instructions. The paper qualifies its own alignment result as matching the stated preferences of its labelers and researchers, not human values broadly. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“they found that LLMs performed better on the default version of a task (e.g., executing Python code under the true assumption that Python uses 0-based indexing) than on a counterfactual version of the task (e.g., falsely assuming that Python uses 1-based indexing).”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > Sensitivity to Task Probability > Summary of Task Probability Effects.
What the sources say
- Supportedcites [42] Z. Wu et al., “Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks” in Proceedings of NAACL (2024), pp. 1819–1862. source
The Results (§4) state that language models consistently performed substantially worse on counterfactual task variants than on default variants. Appendix A.2 confirms that the programming task compared ordinary 0-based indexing with counterfactual 1-based indexing. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Based on an analysis of the C4 corpus (36), we found that the three most common shift levels are rot-1, rot-3, and rot-13”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > Sensitivity to Task Probability > Shift Ciphers.
What the sources say
- Supportedcites [36] C. Raffel, Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 5485– 5551 (2020). source
Section 2.2 (p. 6) identifies the dataset as the Colossal Clean Crawled Corpus, or C4. The citing authors attribute the shift-frequency finding to their own analysis; this citation identifies the corpus they analyzed, not the source of that finding. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Based on an analysis of the C4 corpus, we estimate that alphabetical order is 150 times more common than reverse alphabetical order in Internet text.”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > Sensitivity to Task Probability > Sorting.
What the sources say
- Supportedcites [36] C. Raffel, Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 5485– 5551 (2020). source
Section 2.2 (p. 6) identifies C4 as the authors’ corpus of cleaned web text. The citing sentence presents the 150-times estimate as its own analysis, not as a result reported by Raffel et al.; this source supports the citation’s identification of C4, but does not verify the estimate. (LLM, conf=0.88)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“a human’s ease of processing a sentence is influenced by that sentence’s probability (66–68)”
§ Embers of autoregression show how large language models are shaped by the problem they are trained to solve > Discussion > Comparing Models to Humans.
What the sources say
- Not checked — no full text of the sourcecites [66] S. F. Ehrlich, K. Rayner, Contextual effects on word perception and eye movements during reading. J. Verb. Learn. Verb. Behav. 20, 641– 655 (1981). source
- Partly supportedcites [67] N. J. Smith, R. Levy, The effect of word predictability on reading time is logarithmic. Cognition 128, 302– 319 (2013). 23747651 PMC3709001 sourceExperimental — accuracy not yet measured
The Discussion reports that word predictability affects word comprehension in context, and the paper relates that effect to reading time. It does not establish that the probability of an entire sentence affects how easily a human processes that sentence; its measured probabilities are for words in context. (LLM, conf=0.86)
- Partly supportedcites [68] E. Gibson, L. Bergen, S. T. Piantadosi, Rational integration of noisy evidence and prior semantic expectations in sentence interpretation. Proc. Natl. Acad. Sci. U.S.A. 110, 8051– 8056 (2013). 23637344 PMC3657782 sourceExperimental — accuracy not yet measured
The paper’s Discussion says that comprehenders use expectations about likely sentences to arrive at an interpretation, supporting the claim that sentence probability influences comprehension. But the Results measure which interpretation participants chose, not how easy a sentence was to process; the claim’s processing-ease component is not established. (LLM, conf=0.83)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
65 more claims
Create a free account to see all 129 claims and every source verdict.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 42
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, models and humans also have different capabilities and consequently different constraints for some tasks (Griffiths, 2020;Shiffrin & Mitchell, 2023;McCoy et al., 2024).”
From Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse (2024) · cites this paper as McCoy et al., 2024
In the citing paper
“The disparity between LLM performance on mARC-QA and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities19,54–57.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [54]
In the citing paper
“This interpretation is reinforced by studies that have shown LLMs’ limited robustness and increased hallucination rates in low-probability contexts, where reliance on surface-level statistical correlations proves insufficient19,54,58.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [54]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
The spandrels of San Marco and the Panglossian paradigm: a critique of the adaptationist programme
Gould, S. J., & Lewontin, R. C. (1979) · Proceedings of the Royal Society of London. Series B. Biological Sciences · doi:10.1098/rspb.1979.0086 · Cited by 7,770
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Research works printed but not linked: 8
- Anthropic Introducing the next generation of Claude. Anthropic [Preprint] (2024). https://www.anthropic.com/news/claude-3-family. Accessed 28 August 2024.
- D. Marr, Vision (W.H. Freeman, 1982).
- A. Radford et al. Language models are unsupervised multitask learners. OpenAI [Preprint] (2019). https://openai.com/index/better-language-models. Accessed 28 August 2024.
- Suetonius The Twelve Caesars (R. Worthington 1883) vol. 121 Translator: Alexander Thomson.
- D. Kahn, The Codebreakers (Scribner, 1967).
- S. Singh, The Code Book (Doubleday, 1999).
- B. Vaux A. I. Nevins “Underdetermination in language games: Survey & analysis of Pig Latin dialects” in 77th Annual Meeting of the LSA (Atlanta GA 2003) pp. 2–5.
- BIG-bench authors Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res. 05/2023 1–95 (2023).
Cited by
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2024) · arXiv · doi:10.48550/arxiv.2410.05229 · Cited by 714
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., & Murthy, A. (2024) · arXiv · doi:10.48550/arxiv.2402.01817 · Cited by 237
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
Theory Is All You Need: AI, Human Cognition, and Causal Reasoning
(2024) · doi:10.1287/stsc.2024.0189 · Cited by 151
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
Bean, A. M., Kearns, R. O., Romanou, A., Hafner, F. S., Mayne, H., Batzner, J., Foroutan Eghlidi, N., Schmitz, C., Korgul, K., Batra, H., et al. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-0590 · Cited by 110
Metadata
- Authors
- McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L.
- Affiliations
- Princeton University (US)
- Corresponding author
- McCoy, R. Thomas
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2024-10-04
- Volume
- 121
- Issue
- 41
- Pages
- e2322420121-e2322420121
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC11474099
- OpenAlex
- W4403122979
- Cited by
- 96
- References
- 66
- Citations by year
- 2026: 48 · 2025: 39 · 2024: 9
- FWCI
- 14.11
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (2204152)
- Princeton University (N/A)
- U.S. Department of Defense (N/A)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, teleological approach, next-word prediction, LLM failure modes, LLM evaluation, autoregression, GPT-4, Claude 3, Llama 3
- MeSH terms
- Humans, Language, Models, Theoretical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Embers of autoregression show how large language models are shaped by the problem they are trained to solve. https://app.cochat.ai/papers/10.1073/pnas.2322420121
- The paper
- McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024). Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2322420121
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.