CoChat
Sign Up
PreprintCited by 2Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

A Statistical Physics of Language Model Reasoning

Open access
Carson, Jack David, Reisizadeh, Amir·2025·arXiv
DOI 10.48550/arxiv.2506.04374arXiv 2506.04374
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformer LMs show emergent reasoning that resists mechanistic understanding. We offer a statistical physics framework for continuous-time chain-of-thought reasoning dynamics. We model sentence-level hidden state trajectories as a stochastic dynamical system on a lower-dimensional manifold. This drift-diffusion system uses latent regime switching to capture diverse reasoning phases, including misaligned states or failures. Empirical trajectories (8 models, 7 benchmarks) show a rank-40 projection (balancing variance capture and feasibility) explains ~50% variance. We find four latent reasoning regimes. An SLDS model is formulated and validated to capture these features. The framework enables low-cost reasoning simulation, offering tools to study and predict critical transitions like misaligned states or other LM failures.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.04374

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 34

  1. Abdin et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.14219 , Apr 2024. URL https://arxiv. org/abs/2404.14219 . · arXiv
  2. Allen-Zhu et al. Physics of language models: Part 1, learning hierarchical language structures. arXiv preprint arXiv:2305.13673 , 2023. · arXiv
  3. Bai et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , Sep 2023. URL https://arxiv. org/abs/2309.16609 . · arXiv
  4. Brown et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 , pp. 1877-1901, 2020.
  5. Chaudhuri et al. Computational principles of memory. Nature Neuroscience , 19(3):394-403, 2016. doi: 10.1038/ nn.4237.
  6. Clark et al. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1090. URL https://aclanthology.org/N19-1090 . · DOI
  7. Cobbe et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , Oct 2021. URL https://arxiv.org/abs/2110.14168 . · arXiv
  8. Davis et al. The rotation of eigenvectors by a perturbation. III. SIAM Journal on Numerical Analysis , 7(1):1-46, 1970. doi: 10.1137/0707001. · DOI
  9. DeepSeek-AI et al. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 , Jan 2024. URL https://arxiv. org/abs/2401.02954 . · arXiv
  10. Dempster et al. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological) , 39(1):1-38, 1977. doi: 10.1111/j.2517-6161.1977.tb01600.x. · DOI · Paper page
  11. Gemma Team et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 , Mar 2024. URL https://arxiv. org/abs/2403.08295 . · arXiv
  12. Geva et al. Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics (TACL) , 9:346-361, 2021. doi: 10. 1162/tacl\_a\_00370. URL https://aclanthology. org/2021.tacl-1.21 .
  13. Ghahramani et al. Variational learning for switching statespace models. Neural Computation , 12(4):831-864, 2000. doi: 10.1162/089976600300015619. · DOI
  14. Grönwall. Note on the derivatives with respect to a parameter of the solutions of a system of differential equations. Annals of Mathematics , 20(4):292-296, 1919. doi: 10.2307/1967124. · DOI
  15. Hamilton. A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica , 57(2):357-384, 1989.
  16. Hoerl et al. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics , 12(1):55-67, 1970. doi: 10.1080/00401706.1970.10488634. · DOI · Paper page
  17. Jolliffe. Principal Component Analysis . Springer Series in Statistics. Springer-Verlag, New York, second edition, 2002. ISBN 0-387-95442-2. doi: 10.1007/b98835. · DOI · Paper page
  18. Li et al. Emergent world representations: Exploring a sequence model trained on a synthetic task. In Proceedings of the International Conference on Learning Representations (ICLR) , 2023.
  19. (Volume 1: Long Papers) , pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https: //aclanthology.org/2022.acl-long.229 . · DOI · Paper page
  20. López-Otal et al. Linguistic interpretability of transformerbased language models: A systematic review. arXiv preprint arXiv:2404.08001 , 2024. · arXiv
  21. Mihaylov et al. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 2381-2391, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18-1260. URL https://aclanthology.org/ D18-1260 . · DOI
  22. Nanda et al. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941 , 2023. · arXiv
  23. Øksendal. Stochastic Differential Equations: An Introduction with Applications . Springer Science & Business Media, sixth edition, 2003. ISBN 978-3540047582.
  24. Olsson et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 , 2022. · arXiv
  25. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE , 77(2):257-286, 1989.
  26. Risken et al. The Fokker-Planck Equation: Methods of Solution and Applications , volume 18 of Springer Series in Synergetics . Springer, Berlin, Heidelberg, 2nd ed. 1989, corrected 2nd printing edition, 1996. ISBN 978-3-54061530-9. doi: 10.1007/978-3-642-61530-9. · DOI
  27. Schuecker et al. Optimal sequence memory in driven random networks. Physical Review X , 8(4):041029, 2018. doi: 10.1103/PhysRevX.8.041029. · DOI
  28. Talmor et al. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https: //aclanthology.org/N19-1421 . · DOI
  29. (eds.), Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS 2021) , December 2021. URL https: //datasets-benchmarks-proceedings. neurips.cc/paper/2021/hash/ 1f1baa5b8eddf7699957626905810290-Abstract-round2. html . arXiv:2201.05320. · arXiv
  30. Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , Jul 2023. URL https://arxiv.org/abs/2307.09288 . · arXiv
  31. Vaswani et al. Attention is all you need. In Advances in Neural Information Processing Systems 30 , pp. 59986008, 2017.
  32. Wang et al. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001 , 2023. · arXiv
  33. Wei et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. · arXiv
  34. Zellers et al. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 4799-4809, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472 . · DOI · Paper page

Cited by

Sorted by citations

See all 2

Metadata

Authors
Carson, J. D., & Reisizadeh, A.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.04374
arXiv
2506.04374
OpenAlex
W4416131823
Cited by
2
References
2
Citations by year
2026: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
language model reasoning, chain-of-thought, emergent reasoning, drift-diffusion model, stochastic dynamical systems, tipping points, low-rank projection, continuous-time dynamics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Statistical Physics of Language Model Reasoning. https://app.cochat.ai/papers/10.48550/arxiv.2506.04374
The paper
Carson, J. D., & Reisizadeh, A. (2025). A Statistical Physics of Language Model Reasoning. arXiv. https://doi.org/10.48550/arxiv.2506.04374