LLMs are not just next token predictors
Open accessAbstract
LLMs are statistical models of language learning through stochastic gradient descent with a next token prediction objective. Prompting a popular view among AI modelers: LLMs are just next token predictors. While LLMs are engineered using next token prediction, and trained based on their success at this task, our view is that a reduction to just next token predictor sells LLMs short. Moreover, there are important explanations of LLM behavior and capabilities that are lost when we engage in this kind of reduction. In order to draw this out, we will make an analogy with a once prominent research program in biology explaining evolution and development from the gene's eye view.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs are fundamentally agnostic to empirical reality.
“While some, like Downes et al. (33), challenge this view, suggesting that LLMs can produce sensible answers by leveraging higher-level structural information inherent in their design, the fact remains that LLMs remain fundamentally agnostic to empirical reality.”
From Why we need to be careful with LLMs in medicine (2024) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“The term comes from Bender et al. (2021) and aims to capture the following thought: much like a parrot who says, “Polly wants a cracker,” LLMs produce strings that humans can interpret, but (if the objection is correct), the LLM itself has no knowledge of the meaning of the string it produces.”
Adaptation and Natural Selection
Williams, G. (2018) · Journal of Animal Ecology · doi:10.2307/3103 · Cited by 4,539 · cited 1× · see citing works
“The so-called ‘ gene ’ s eye view ’ of evolution, fi rst articulated by George Williams (1966) and popularized by Richard Dawkins (1976) and Daniel Dennett (see especially 1995), claims that genes and only genes are the relevant units of selection.”
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B. K., Pearce, A., Citro, C., Ameisen, E., Jones, A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.29358 · Cited by 915 · cited 1×
“This is the sort of information Anthropic (Templeton et al. 2024) is identifying in the LLMs when they are ‘ mapping the mind ’ and revealing concept clusters and other abstract features encoded in the model.”
Launer, J. (2002) · QJM : monthly journal of the Association of Physicians · doi:10.1093/qjmed/95.3.195 · Cited by 848 · cited 1× · see citing works
“The so-called ‘ gene ’ s eye view ’ of evolution, fi rst articulated by George Williams (1966) and popularized by Richard Dawkins (1976) and Daniel Dennett (see especially 1995), claims that genes and only genes are the relevant units of selection.”
Evolution and the Levels of Selection
Okasha, S. (2007) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199267972.001.0001 · Cited by 741 · see citing works
Talking about Large Language Models
Shanahan, M. (2024) · Communications of the ACM · doi:10.1145/3624724 · Cited by 381 · cited 1× · see citing works
“So , LLMs are ‘just next token predictors’, a popular view among AI modelers, explicitly laid out by Shanahan (2024): “A great many tasks that demand intelligence in humans can be reduced to next-token prediction with a sufficiently performant model” (2024, 68), and “surely what they are doing is…”
Science and Selection: Essays on Biological Evolution and the Philosophy of Science
Hull, D. (2000) · Choice Reviews Online · doi:10.5860/choice.39-0297 · Cited by 135 · cited 1× · see citing works
“One early and valuable proposal, put forward by the biologist and philosopher David Hull (2001), involves a distinction between replicators and interactors.”
Does ChatGPT have semantic understanding? A problem with the statistics-of-occurrence strategy
Titus, L. M. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101174 · Cited by 82 · cited 1× · see citing works
“The present paper is not focused on whether LLMs are meaning users in the sense of being sensitive to the semantics of the strings they take as input or give as output (see Titus 2024 for further discussion).”
The Generational Cycle of State Spaces and Adequate Genetical Representation*
Lloyd, E., Lewontin, R., & Feldman, M. (2008) · Philosophia Scientiæ · doi:10.1086/590196 · Cited by 23
Butlin, P., & Viebahn, E. (2025) · Ergo an Open Access Journal of Philosophy · doi:10.3998/ergo.7960 · Cited by 19 · see citing works
Still "Talking About Large Language Models": Some Clarifications
Shanahan, M. (2024) · arXiv.org · doi:10.48550/arxiv.2412.10291 · Cited by 5
Flagged first, then by citations
2 of the 13 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Agren, A. (2021). *The Gene's-Eye View of Evolution*. Oxford University Press.
- Dawkins, R. (1976). *The Selfish Gene*. Oxford University Press, Oxford, UK.
- Oyama, S. (2000). *The Ontogeny of Information*. Duke University Press.
Cited by
Sorted by citations
Why we need to be careful with LLMs in medicine
Bélisle-Pipon, J.-C. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1495582 · Cited by 91 · see citing works
- Authors
- Downes, S. M., Forber, P., & Grzankowski, A.
- Affiliations
- University of Utah (US)
- Tufts University (US)
- University of London (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2408.04666
- OpenAlex
- W4402385988
- Cited by
- 2
- References
- 13
- Citations by year
- 2025: 1 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- large language models, next-token prediction, reductionism, language learning, stochastic gradient descent, evolution and development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
