Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 15 of the 34 sources its claims cite were checked for support.

Embers of autoregression show how large language models are shaped by the problem they are trained to solve

Open access
McCoy, R. Thomas, Yao, Shunyu, Friedman, Dan, Hardy, Mathew D., Griffiths, Thomas L. · 2024 · Proceedings of the National Academy of Sciences

Abstract

The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that to develop a holistic understanding of these systems, we must consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts, we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. Using this approach—which we call the teleological approach—we identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. To test our predictions, we evaluate five LLMs (GPT-3.5, GPT-4, Claude 3, Llama 3, and Gemini 1.0) on 11 tasks, and we find robust evidence that LLMs are influenced by probability in the hypothesized ways. Many of the experiments reveal surprising failure modes. For instance, GPT-4’s accuracy at decoding a simple cipher is 51% when the output is a high-probability sentence but only 13% when it is low-probability, even though this task is a deterministic one for which probability should not matter. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system—one that has been shaped by its own particular set of pressures.

Overview

CoChat Summary

TL;DR

The paper argues that large language models can be understood teleologically as statistical next-word prediction systems shaped by Internet text, and it tests three predicted sensitivities: output probability, input probability, and task frequency. Across five LLMs and eleven tasks, accuracy is often higher for high-probability outputs and common task variants, and sometimes for high-probability inputs, even on deterministic tasks.

Key findings

  • Output probability: significant effect in all cases except Gemini 1.0 decoding shift ciphers (0% accuracy) and Claude 3 forming acronyms (P=0.38). GPT-4’s article-swapping accuracy ranged from 0.02 on low-probability examples to 0.83 on high-probability examples. Counting accuracy spiked at multiples of 10; for GPT-4, accuracy correlated more with number frequency than magnitude (absolute coefficients 0.84 vs. 0.70), and for Claude 3, 0.85 vs. 0.66. In shift-cipher errors, GPT-3.5 and GPT-4 produced a regularized high-probability sentence more often than the correct answer.
  • Input probability: little or no effect on most tasks, but the birthday task showed a large effect (P<10−15 for all models). GPT-4 had higher accuracy when shift-cipher inputs were high-probability than low-probability; GPT-3.5 showed no significant input-probability effect.
  • Task probability: across seven common-vs-rare task pairs, models performed substantially better on common variants. For shift ciphers, all models except Gemini 1.0 spiked at rot-13; GPT-4 scored ≥0.50 on the three highest-frequency shifts (1, 3, 13) but <0.03 on all other shifts, and rot-13 beat rot-12 (P<0.01) for GPT-3.5, GPT-4, and Claude 3. Pig Latin encoding was better for the common variant (P<0.01) for GPT-3.5, GPT-4, and Claude 3; decoding favored Pig Latin for all five models but was significant only for GPT-4 and Llama 3 (P<0.05). First-letter acronyms beat second-letter acronyms for all five models (P<10−4). The common linear function f(x)=(9/5)x+32 beat the rare f(x)=(7/5)x+31 for GPT-3.5, GPT-4, and Claude 3 (P<0.01; Llama 3 P=0.22). Alphabetical sorting beat reverse alphabetical sorting for all five models (P<10−5), paralleling a 150× corpus-frequency difference.
  • Overall: chain-of-thought and step-by-step prompting, scaling, and few-shot learning can raise overall accuracy, but the qualitative sensitivities to task frequency and example probability remained.

Methods

  • The paper uses a teleological/Bayesian analysis of next-word prediction to derive predictions about output probability, input probability, and task probability. It evaluated five models: gpt-3.5-turbo-0613, gpt-4-0613, claude-3-opus-20240229, gemini-1.0-pro-001, and llama-3-70b-chat-hf, accessed via APIs at temperature 0.0. Preliminary tests with OLMo and Llama 2 were too low to yield trends (0% on most tasks).
  • Eleven tasks were used: article swapping, reversal, counting, acronyms, linear function, multiplication, sorting, keyboard cipher, shift cipher, Pig Latin, and birthdays. High-probability sentences came from Global Voices; medium-probability sentences used RoBERTa word replacements; low-probability sentences shuffled medium-probability examples except first and last words. Words came from the CMU Pronouncing Dictionary; acronym lists had 7 words, sorting lists 10–20 words, counting lists 1–100 words, linear-function inputs were 0–999, multiplication inputs 100–999, and birthdays came from WikiBio. Sample sizes were 1,000 per condition for acronyms, 30 per number for counting, and 100 per condition for other tasks. Models received basic prompts without tuning.
  • Statistical tests used logistic regressions with correctness as the outcome; predictors generally included input probability, output probability, input length, output length, and task variant. Output probabilities were estimated using GPT-2, and task frequencies were estimated from C4 and the Pile.

Limitations

  • Paper states: the analysis focuses only on next-word prediction and leaves instruction tuning for future work; it considers only text-processing LLMs, not augmented models with image or code-execution abilities; it uses basic prompting, though chain-of-thought and step-by-step prompting can improve performance while preserving the reported trends; it does not make strong claims about qualitative human differences because there is no fair human comparison; and it does not claim LLMs cannot perform the tasks, only that some tasks and examples are harder than others.
  • Paper states/evident: tasks were chosen adversarially to push models into low-probability situations and ensure nonnegligible error rates, so the results are not a general capability benchmark. Some models showed floor effects (e.g., Gemini 1.0 at 0% on shift ciphers; OLMo and Llama 2 at 0% on most tasks), and corpus-frequency estimates are proxies for training data. Sample sizes also varied, including only 30 examples per number in the counting task.
  • Evident: the study covers only five API models at specific versions and temperature 0.0, does not separate instruction-tuned from pretrained behavior, lacks a human baseline, and uses a limited set of eleven tasks; some significance tests excluded models because of floor effects.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.2322420121

Cites

91 of 91 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1. Humans and LLMs have some shared properties and some properties that differ. If we analyze LLMs using tests designed for humans, we risk identifying only the shared properties, missing the properties that are unique to LLMs (the dotted region of the diagram). We argue that to identify the properties in the dotted region, we must approach LLMs on their own terms by considering the problem that they were trained to solve: next-word prediction over Internet text.
Fig. 1. Humans and LLMs have some shared properties and some properties that differ. If we analyze LLMs using tests designed for humans, we risk identifying only the shared properties, missing the properties that are unique to LLMs (the dotted region of the diagram). We argue that to identify the properties in the dotted region, we must approach LLMs on their own terms by considering the problem that they were trained to solve: next-word prediction over Internet text.CC BY · PubMed Central
Fig. 2. GPT-4 struggles on some seemingly simple tasks such as counting, article swapping, shift ciphers, and linear functions. In the counting and article swapping examples, GPT-4 fails in the cases where the correct output is a low-probability piece of text (for the counting example, we refer to 29 as low-probability because it occurs much less frequently in natural corpora than 30 does). In the shift cipher and linear function examples, GPT-4 performs well on the task variants that are common in Internet text but poorly on the variants that are rare (note that the shift cipher with a shift of 13 is over 100 times more common in Internet text than the shift cipher with a shift of 12; and the linear function f(x)=(9/5)x+32 is common because it is the Celsius-to-Fahrenheit conversion, while the other linear function has no special significance). The GPT-4 predictions were obtained using gpt-4-0613 on the OpenAI API; other model versions (e.g., the online chat interface) may give different predictions.
Fig. 2. GPT-4 struggles on some seemingly simple tasks such as counting, article swapping, shift ciphers, and linear functions. In the counting and article swapping examples, GPT-4 fails in the cases where the correct output is a low-probability piece of text (for the counting example, we refer to 29 as low-probability because it occurs much less frequently in natural corpora than 30 does). In the shift cipher and linear function examples, GPT-4 performs well on the task variants that are common in Internet text but poorly on the variants that are rare (note that the shift cipher with a shift of 13 is over 100 times more common in Internet text than the shift cipher with a shift of 12; and the linear function f(x)=(9/5)x+32 is common because it is the Celsius-to-Fahrenheit conversion, while the other linear function has no special significance). The GPT-4 predictions were obtained using gpt-4-0613 on the OpenAI API; other model versions (e.g., the online chat interface) may give different predictions.CC BY · PubMed Central
Fig. 3. When processing shift ciphers, LLMs are highly sensitive to probability, even though shift ciphers are deterministic. Top Left: Effect of output probability. When decoding text written in the rot-13 cipher, both models score better when the answer is high-probability than when it is low-probability. Top Middle: Effect of input probability. As predicted, input probability does not show as strong an effect as output probability. Top Right: Effect of task probability. Both models score better on a frequently occurring shift cipher (rot-13) than on a rare one (rot-12). All error intervals show one SE. Bottom: Examples for output and input probability; see Fig. 2 for task probability examples.
Fig. 3. When processing shift ciphers, LLMs are highly sensitive to probability, even though shift ciphers are deterministic. Top Left: Effect of output probability. When decoding text written in the rot-13 cipher, both models score better when the answer is high-probability than when it is low-probability. Top Middle: Effect of input probability. As predicted, input probability does not show as strong an effect as output probability. Top Right: Effect of task probability. Both models score better on a frequently occurring shift cipher (rot-13) than on a rare one (rot-12). All error intervals show one SE. Bottom: Examples for output and input probability; see Fig. 2 for task probability examples.CC BY · PubMed Central
Fig. 4. The effect of output probability on model accuracy across five tasks. The intervals around the lines show one SE.
Fig. 4. The effect of output probability on model accuracy across five tasks. The intervals around the lines show one SE.CC BY · PubMed Central
Fig. 5. Model performance on counting a list of words, as a function of the length of the list. The intervals around the lines show one SE.
Fig. 5. Model performance on counting a list of words, as a function of the length of the list. The intervals around the lines show one SE.CC BY · PubMed Central
Fig. 6. The effect of input probability on model accuracy across five tasks. The intervals around the lines show one SE.
Fig. 6. The effect of input probability on model accuracy across five tasks. The intervals around the lines show one SE.CC BY · PubMed Central
Fig. 7. Analysis of shift ciphers with shift levels varying from 1 to 25. Top: Number of occurrences of each shift level in the C4 corpus. Bottom: Accuracy on decoding shift ciphers with various shifts. Most models display a spike in accuracy at a shift of 13, which the top plot shows to be the most common shift level.
Fig. 7. Analysis of shift ciphers with shift levels varying from 1 to 25. Top: Number of occurrences of each shift level in the C4 corpus. Bottom: Accuracy on decoding shift ciphers with various shifts. Most models display a spike in accuracy at a shift of 13, which the top plot shows to be the most common shift level.CC BY · PubMed Central
Fig. 8. Accuracy on Pig Latin variants when encoding (Left) or decoding (Right), as a function of the variant’s frequency. The shaded intervals show one SE.
Fig. 8. Accuracy on Pig Latin variants when encoding (Left) or decoding (Right), as a function of the variant’s frequency. The shaded intervals show one SE.CC BY · PubMed Central
Fig. 9. Comparing performance on common vs. rare versions of tasks. For shift ciphers, the common variant that we illustrate is rot-13, and the rare variant is rot-12. For Pig Latin, the common variant is Pig Latin, and the rare variant is Boar Etruscan (see text). For acronyms, the common variant is acronyms formed from the first letters of words, while the rare variant is acronyms formed from the second letters of words. The common linear function is f(x)=(9/5)x+32 (which is the Celsius-to-Fahrenheit conversion), and the rare linear function is f(x)=(7/5)x+31. The common version of sorting is sorting in alphabetical order, while the rare variant uses reverse alphabetical order. Error bars show one SE.
Fig. 9. Comparing performance on common vs. rare versions of tasks. For shift ciphers, the common variant that we illustrate is rot-13, and the rare variant is rot-12. For Pig Latin, the common variant is Pig Latin, and the rare variant is Boar Etruscan (see text). For acronyms, the common variant is acronyms formed from the first letters of words, while the rare variant is acronyms formed from the second letters of words. The common linear function is f(x)=(9/5)x+32 (which is the Celsius-to-Fahrenheit conversion), and the rare linear function is f(x)=(7/5)x+31. The common version of sorting is sorting in alphabetical order, while the rare variant uses reverse alphabetical order. Error bars show one SE.CC BY · PubMed Central

Verification

Claims the paper makes

11 Valid · 3 Not valid · 8 Not checked · 107 Own results and interpretations

65 more claims

Create a free account to see all 129 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 23
  • Probably not a substantial claim. 42

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 66 references

Research works printed but not linked: 8

  1. Anthropic Introducing the next generation of Claude. Anthropic [Preprint] (2024). https://www.anthropic.com/news/claude-3-family. Accessed 28 August 2024.
  2. D. Marr, Vision (W.H. Freeman, 1982).
  3. A. Radford et al. Language models are unsupervised multitask learners. OpenAI [Preprint] (2019). https://openai.com/index/better-language-models. Accessed 28 August 2024.
  4. Suetonius The Twelve Caesars (R. Worthington 1883) vol. 121 Translator: Alexander Thomson.
  5. D. Kahn, The Codebreakers (Scribner, 1967).
  6. S. Singh, The Code Book (Doubleday, 1999).
  7. B. Vaux A. I. Nevins “Underdetermination in language games: Survey & analysis of Pig Latin dialects” in 77th Annual Meeting of the LSA (Atlanta GA 2003) pp. 2–5.
  8. BIG-bench authors Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res. 05/2023 1–95 (2023).

Cited by

See all 109

Metadata

Authors
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L.
Affiliations
Princeton University (US)
Corresponding author
McCoy, R. Thomas
Venue
Proceedings of the National Academy of Sciences
Published
2024-10-04
Volume
121
Issue
41
Pages
e2322420121-e2322420121
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4403122979
Cited by
96
References
66
Citations by year
2026: 48 · 2025: 39 · 2024: 9
FWCI
14.11
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • National Science Foundation (2204152)
  • Princeton University (N/A)
  • U.S. Department of Defense (N/A)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, teleological approach, next-word prediction, LLM failure modes, LLM evaluation, autoregression, GPT-4, Claude 3, Llama 3
MeSH terms
Humans, Language, Models, Theoretical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Embers of autoregression show how large language models are shaped by the problem they are trained to solve. https://app.cochat.ai/papers/10.1073/pnas.2322420121
The paper
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024). Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2322420121

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.