CoChat
Sign Up
PreprintCited by 2Computability, Logic, AI Algorithmssemigroups and automata theoryArtificial Intelligence Applications

Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesis

Open access
Zenil, Hector, Uthamacumaran, Abicumaran, Ozelim, Luan·2026·arXiv
DOI 10.48550/arxiv.2601.05280arXiv 2601.05280
Open this paper in CoChat — free accountOpen PDF

Abstract

We connect two questions in Algorithmic Information Theory (AIT), Machine Learning (ML) and Artificial General Intelligence (AGI): whether LLMs estimate Solomonoff induction, and whether they can self-improve towards an AI Singularity. We provide theoretical, methodological and empirical answers in the negative but show how limits can be circumvented. Cross-entropy, negative log-likelihood and related next-token objectives cannot alone implement Solomonoff induction: they fit supplied conditionals rather than a program-weighted universal mixture. More computation can improve fit within a fixed objective but cannot change its inductive principle without external hyperparameter or architectural tuning; they alone do not deliver Solomonoff-Levin optimal prediction. For finite learners and observers, theoretical boundaries become less decisive and approaches diverge. Resource-bounded estimators are finite mechanism-search tools whose divergence does not violate algorithmic information conservation. All 26 served language-model checkpoints across five pre-training families, 0.8-35 billion parameters and 1.9-8.5 bits per weight, evaluated at their commitments over a closed alphabet, violate the dominance guarantee defining a universal mixture. Against a 3.32-bit bound attained by a genuine mixture, the best model trails a Krichevsky-Trofimov code by 4.5 bits, the median by 36 and the worst by 128; excess grows to every stream's end rather than settling to a constant. Served conditionals fail to form a mixture over the declared class in 79 of 91 checkpoint-designs; neither scale nor post-training closes the gap. Frontier developers adopt neurosymbolic approaches, including Fable and Astra, incorporating model synthesis via neurosymbolic computation. They are no longer purely statistical LLMs, making them better, though still limited, candidates for higher forms of induction & model synthesis.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2601.05280

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • If the fraction of exogenous, externally grounded signal vanishes asymptotically, degenerative dynamics such as entropy decay and variance amplification follow.

    “if the fraction of exogenous, externally grounded signal vanishes asymptotically, degenerative dynamics (entropy decay, variance amplification) follow [146].”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [146]

    Section 4 states that a grounding fraction can vanish while cumulative correction still brings the model toward the external distribution. The paper also gives a finite-sample counterexample and explicitly says that a vanishing grounding fraction does not necessarily cause collapse. It therefore rejects the claimed implication, although it describes diversity loss under repeated finite resampling without a restoring target. (LLM, conf=0.92)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Zenil's impossibility result argues LLM-style self-training cannot be unboundedly self-improving without symbolic model synthesis or an unvanishing stream of external signal.

    “Zenil's impossibility result [146] argues LLM-style self-training cannot be unboundedly self-improving without symbolic model synthesis or an unvanishing stream of external signal.”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [146]

    The Introduction (page not provided) says closed-loop distribution matching is not itself an improvement operator and that credible recursive self-improvement requires candidate changes to be evaluated against independent criteria. But it explicitly calls its thesis narrower than a universal impossibility theorem. Sections 4 and 9 (pages not provided) also say an external-data fraction can vanish while cumulative correction remains effective, so an unvanishing stream of external signal is not established as a necessary alternative. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Self-training without external signal degrades rather than explodes.

    “self-training without external signal degrades rather than explodes [146, 144]”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [146]

    The Conclusion says finite repeated resampling can reduce diversity and introduce stochastic distortion, supporting the possibility of degradation rather than self-improvement. But Section 4 says exact self-training reproduces the current distribution; without an external signal, degradation is not inevitable. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • In the extreme, recursive training on generated data can lead to model collapse, where distribution tails disappear and outputs become increasingly repetitive, which would be especially damaging for scientific novelty.

    “In the extreme, recursive training on generated data can lead to model collapse , where distribution tails disappear and outputs become increasingly repetitive, which would be especially damaging for scientific novelty [10].”

    From Are Researchers Being Replaced by Artificial Intelligence? (2026) · cites this paper as [10]

    The Introduction (section 1; page unavailable) says recursive replacement with synthetic data can lose rare events, and section 4 says repeated finite replacement reduces diversity. These support model collapse, tail loss, and increasingly repetitive outputs as a paraphrase. The paper does not specifically establish that this collapse is especially damaging for scientific novelty; it discusses scientific progress separately and cautions that novel output is not the same as useful discovery. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Uploading submissions to AI tools in peer review for conferences and funding agencies remains largely prohibited, with exception of the 'la Caixa' Foundation.

    “However, uploading submissions to AI tools in peer review for conferences and funding agencies remains largely prohibited, with exception of the 'la Caixa' Foundation. 10”

    From Are Researchers Being Replaced by Artificial Intelligence? (2026) · cites this paper as [10]

    The paper’s ‘Use of AI tools disclosure statement’ describes the author’s use of generative AI for manuscript preparation, not rules for peer reviewers. Searches for the ‘la Caixa’ Foundation and for restrictions on uploading conference or funding submissions found no passage addressing the claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 28

  1. I. J. Good. Speculations concerning the first ultraintelligent machine. Advances in Com- puters, 6:31-88, 1966. doi:10.1016/S0065-2458(08)60418-0. · DOI · Paper page
  2. V. Vinge. The coming technological Singularity: How to survive in the post-human era. In Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace, NASA Conference Publication 10129, 1993. NASA Technical Reports Server record 19940022856.
  3. S. Alemohammad, J. Casco-Rodriguez, L. Luzi, A. I. Humayun, H. Babaei, D. LeJeune, A. Siahkoohi, and R. G. Baraniuk. Self-consuming generative models go MAD. International Conference on Learning Representations, 2024.
  4. I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal. AI mod- els collapse when trained on recursively generated data. Nature, 631:755-759, 2024. doi:10.1038/s41586-024-07566-y. · DOI · Paper page
  5. E. Dohmatob, Y. Feng, P. Yang, F. Charton, and J. Kempe. A tale of tails: Model collapse as a change of scaling laws. In Proceedings of the 41st International Conference on Machine Learning, PMLR 235:11165-11197, 2024.
  6. S. Fu, S. Zhang, Y. Wang, X. Tian, and D. Tao. Towards theoretical understandings of self-consuming generative models. In Proceedings of the 41st International Conference on Machine Learning, PMLR 235:14228-14255, 2024.
  7. M. Gerstgrasser, R. Schaeffer, A. Dey, R. Rafailov, H. Sleight, J. Hughes, T. Korbak, R. Agrawal, D. Pai, A. Gromov, D. A. Roberts, D. Yang, D. L. Donoho, and S. Koyejo. Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthetic data. arXiv:2404.01413, 2024. · arXiv
  8. N. Gillman, M. Freeman, D. Aggarwal, C.-H. Hsu, C. Luo, Y. Tian, and C. Sun. Self- correcting self-consuming loops for generative model training. In Proceedings of the 41st International Conference on Machine Learning, PMLR 235:15646-15677, 2024.
  9. J. Kazdan, R. Schaeffer, A. Dey, M. Gerstgrasser, R. Rafailov, D. L. Donoho, and S. Koyejo. Collapse or thrive: Perils and promises of synthetic data in a self-generating world. In Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:29469-29494, 2025.
  10. A. T. Suresh, A. Thangaraj, and A. N. K. Khandavally. Rate of model collapse in recursive training. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:1396-1404, 2025.
  11. X. Zhu, D. Cheng, H. Li, K. Zhang, E. Hua, X. Lv, N. Ding, Z. Lin, Z. Zheng, and B. Zhou. How to synthesize text data without model collapse? In Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:79746-79771, 2025.
  12. T. M. Cover and J. A. Thomas. Elements of Information Theory, 2nd edition. Wiley, 2006. doi:10.1002/047174882X. · DOI · Paper page
  13. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, 2017.
  14. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, 2020.
  15. J. Grau-Moya, T. Genewein, M. Hutter, L. Orseau, G. Deletang, E. Catt, A. Ruoss, L. K. Wenliang, C. Mattern, M. Aitchison, and J. Veness. Learning universal predictors. In Proceedings of the 41st International Conference on Machine Learning, PMLR 235:16178- 16205, 2024.
  16. N. Young and M. Witbrock. Transformers as approximations of Solomonoff induction. arXiv:2408.12065, 2024. · arXiv
  17. J. Wan and L. Mei. Large language models as computable approximations to Solomonoff induction. arXiv:2505.15784, 2025. · arXiv
  18. L. Mei and J. Wan. Large language models as a computable surrogate to Solomonoff induction. ICLR 2026 submission, OpenReview, 2026. https://openreview.net/forum? id=psDyG1FDzX.
  19. R. J. Solomonoff. A formal theory of inductive inference, Parts I and II. Information and Control, 7:1-22 and 224-254, 1964. Part I doi:10.1016/S0019-9958(64)90223-2; Part II doi:10.1016/S0019-9958(64)90131-7. · DOI · Paper page
  20. L. A. Levin. Laws of information conservation (non-growth) and aspects of the foundation of probability theory. Problems of Information Transmission, 10(3):206-210, 1974.
  21. M. Li and P. Vitányi. An Introduction to Kolmogorov Complexity and Its Applications, 4th edition. Springer, 2019. doi:10.1007/978-3-030-11298-1. · DOI
  22. M. Hutter. Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability. Springer, 2005. doi:10.1007/b138233. · DOI · Paper page
  23. J.-P. Delahaye and H. Zenil. Numerical evaluation of algorithmic complexity for short strings: A glance into the innermost structure of randomness. Applied Mathematics and Computation, 219(1):63-77, 2012. doi:10.1016/j.amc.2011.10.006. · DOI
  24. H. Zenil, S. Hernández-Orozco, N. A. Kiani, F. Soler-Toscano, A. Rueda-Toicen, and J. Tegnér. A decomposition method for global evaluation of Shannon entropy and local estimations of algorithmic complexity. Entropy, 20(8):605, 2018. doi:10.3390/e20080605. · DOI
  25. H. Zenil, N. A. Kiani, F. Marabita, Y. Deng, S. Elias, A. Schmidt, G. Ball, and J. Tegnér. An algorithmic information calculus for causal discovery and reprogram ming systems. iScience, 19:1160-1172, 2019. doi:10.1016/j.isci.2019.07.043. · DOI
  26. I. Kant. Critique of Pure Reason. Edited and translated by P. Guyer and A. W. Wood. Cambridge University Press, 1998. doi:10.1017/CBO9780511804649. · DOI
  27. J. Pearl. Causality: Models, Reasoning, and Inference, 2nd edition. Cambridge University Press, 2009. doi:10.1017/CBO9780511803161. · DOI · Paper page
  28. H. Zenil, N. A. Kiani, and J. Tegnér. Algorithmic Information Dynamics: A Computational Approach to Causality with Applications to Living Systems. Cambridge University Press, 2023. doi:10.1017/9781108596619. · DOI

Cited by

Sorted by citations

Authors
Zenil, H., Uthamacumaran, A., & Ozelim, L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2601.05280
arXiv
2601.05280
OpenAlex
W7123279256
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Computability, Logic, AI Algorithms
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › semigroups and automata theory
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
large language models, Solomonoff induction, algorithmic information theory, technological singularity, universal prediction, data processing inequality, cross-entropy optimization, negative log-likelihood, next-token prediction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review