Progress measures for grokking via mechanistic interpretability
Open accessAbstract
Neural networks often exhibit emergent behavior, where qualitatively new capabilities arise from scaling up the amount of parameters, training data, or training steps. One approach to understanding emergence is to find continuous \textit{progress measures} that underlie the seemingly discontinuous qualitative changes. We argue that progress measures can be found via mechanistic interpretability: reverse-engineering learned behaviors into their individual components. As a case study, we investigate the recently-discovered phenomenon of ``grokking'' exhibited by small transformers trained on modular addition tasks. We fully reverse engineer the algorithm learned by these networks, which uses discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle. We confirm the algorithm by analyzing the activations and weights and by performing ablations in Fourier space. Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup. Our results show that grokking, rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is no clear understanding of how emergent properties awaken from the mathematical operations within artificial neural networks; this is currently an object of research in Mechanistic Interpretability.
“This is still poorly understood since to date there is yet no clear understanding how these emergent properties awaken from the mathematical operations within the artificial neural networks, which is currently the object of research in a discipline called Mechanistic Interpretability”
From Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education (2024) · cites this paper as Nanda et al., 2023
Cites
Still being checked.
Cited by
Sorted by citations
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113 · see citing works
“…is yet no clear understanding how these emergent properties awaken from the mathematical operations within the artificial neural networks, which is currently the object of research in a discipline called Mechanistic Interpretability (Conmy et al., 2023; Nanda et al., 2023; Zimmermann et al., 2023).”
Longo, L., Brčić, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 648 · see citing works
Dissecting Recall of Factual Associations in Auto-Regressive Language Models
Geva, M., Bastings, J., Filippova, K., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.751 · Cited by 53 · see citing works
Explaining AI through mechanistic interpretability
Kästner, L., & Crook, B. (2024) · European Journal for Philosophy of Science · doi:10.1007/s13194-024-00614-4 · Cited by 48 · see citing works
Emergent Linear Representations in World Models of Self-Supervised Sequence Models
Nanda, N., Lee, A., & Wattenberg, M. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.2 · Cited by 55 · see citing works
Self-citationDomain-specific chatbots for science using embeddings
Yager, K. G. (2023) · Digital Discovery · doi:10.1039/d3dd00112a · Cited by 31 · see citing works
Is Ockham’s razor losing its edge? New perspectives on the principle of model parsimony
Dubova, M., Chandramouli, S., Gigerenzer, G., Grünwald, P., Holmes, W. R., Lombrozo, T., Marelli, M., Musslick, S., Nicenboim, B., Ross, L. N., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401230121 · Cited by 36 · see citing works
Shea, N. J. (2024) · doi:10.1093/9780191997167.001.0001 · Cited by 28 · see citing works
Stolfo, A., Belinkov, Y., & Sachan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.435 · Cited by 14 · see citing works
Longo, L., Brčić, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19775 · Cited by 14 · see citing works
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12 · see citing works
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10 · see citing works
“The line of work on mechanistic interpretability seeks to understand which algorithms (or “circuits”) Transformers internally learn to implement, by inspecting their internals (Olsson et al., 2022; Nanda et al., 2023; Zhong et al., 2023; Conmy et al., 2023; Zhang et al., 2023; Hanna et al., 2023).”
BackgroundCharacterizing Mechanisms for Factual Recall in Language Models
Yu, Q., Merullo, J., & Pavlick, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.615 · Cited by 9 · see citing works
IoT Software Vulnerability Detection Techniques through Large Language Model
Yang, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-7584-6_21 · Cited by 8 · see citing works
Branson, N., Cutillas, P. R., & Bessant, C. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf255 · Cited by 7 · see citing works
Arithmetic with language models: From memorization to computation
Maltoni, D., & Ferrara, M. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106550 · Cited by 7 · see citing works
Beyond the Black Box: Interpretability of LLMs in FinancePreprint
Tatsat, H., & Shater, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5263803 · Cited by 14 · see citing works
Kunz, J. (2024) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789180754712 · Cited by 6 · see citing works
Do Transformers Parse while Predicting the Masked Word?
Zhao, H., Panigrahi, A., Ge, R., & Arora, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1029 · Cited by 6 · see citing works
Anthropocentric Bias in Language Model Evaluation
Millière, R., & Rathkopf, C. (2026) · Computational Linguistics · doi:10.1162/coli.a.582 · Cited by 6 · see citing works
How Do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective
Liu, Q., Duan, H., Mao, J., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3774942 · Cited by 6 · see citing works
Sakarvadia, M., Ajith, A., Khan, A., Grzenda, D., Hudson, N., Bauer, A., Chard, K., & Foster, I. T. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.26 · Cited by 4 · see citing works
Electrochemical breath profiling for early thoracic malignancy screening
Paul, A., France, K. K., Bhatia, A., ABU-HIJLEH, M. A., Daescu, O., Thapa, R., Gordon, R. A., & Prasad, S. (2025) · Sensing and Bio-Sensing Research · doi:10.1016/j.sbsr.2025.100815 · Cited by 4 · see citing works
Aad, G., Aakvaag, E., Abbott, B., Abeling, K., Abicht, N. J., Abidi, S. H., Aboulhorma, A., Abramowicz, H., Abreu, H., Abulaiti, Y., et al. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad611e · Cited by 4 · see citing works
The complexity dynamics of grokking
DeMoss, B., Sapora, S., Foerster, J., Hawes, N., & Posner, I. (2025) · Physica D Nonlinear Phenomena · doi:10.1016/j.physd.2025.134859 · Cited by 5 · see citing works
Adversarial Attacks on the Interpretation of Neuron Activation Maximization
Nanfack, G., Fulleringer, A., Marty, J., Eickenberg, M., & Belilovsky, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28228 · Cited by 5 · see citing works
Improving Dictionary Learning with Gated Sparse Autoencoders
Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., & Nanda, N. (2024) · arXiv · doi:10.48550/arxiv.2404.16014 · Cited by 3 · see citing works
Self-citationGillgren, A., Fransson, E., Ludvig-Osipov, A., Enström, W., Flyckt, L., Green, M., Kvartsén, M., Liljegren, Y., Olsson, E. M. G. V. S., Orthag, A., et al. (2025) · Physics of Plasmas · doi:10.1063/5.0261456 · Cited by 2 · see citing works
Farooq, A., Anuragi, D., Li, Z., Ziat, M., Cooperstock, J. R., & Raisamo, R. (2025) · Human-Intelligent Systems Integration · doi:10.1007/s42454-025-00085-9 · Cited by 2 · see citing works
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
Hou, Y., Li, J., Fei, Y., Stolfo, A., Zhou, W., Zeng, G., Bosselut, A., & Sachan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.299 · Cited by 2 · see citing works
Dedieu, A., Lehrach, W., Zhou, G., George, D., & Lázaro-Gredilla, M. (2024) · arXiv · doi:10.48550/arxiv.2401.05946 · Cited by 2 · see citing works
“Interpretable circuits: Some works try to reverse-engineer LLMs to extract interpretable circuits, i.e., internal structures that drive certain model behaviors (Elhage et al., 2021; Olsson et al., 2022; Nanda et al., 2023).”
BackgroundFriedman, R., Mamatjan, Y., Pan, C., Silveira, P. P., & Zachariou, M. (2024) · Frontiers in Genetics · doi:10.3389/fgene.2024.1520148 · Cited by 1 · see citing works
Language models need inductive biases to count inductively
Chang, Y., & Bisk, Y. (2024) · arXiv · doi:10.48550/arxiv.2405.20131
“There is a possible connection between the bumps observed in some of our counting tasks to the grokking (Nanda et al., 2023) phenomena.”
BackgroundFrom Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task
Martorell, N. (2025) · arXiv · doi:10.48550/arxiv.2502.16690
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with FakepediaPreprint
Monea, G., Peyrard, M., Josifoski, M., Chaudhary, V., Eisner, J. M., Kıcıman, E., Palangi, H., Patra, B., & West, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.02073
Emergent Mental Lexicon Functions in ChatGPTPreprint
Kello, C. T., & Bruna, P. (2024) · doi:10.31234/osf.io/gka2j
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0005
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0009
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0001 · Cited by 1 · see citing works
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0010
The Frame Problem and the If-Then Problem
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0006
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0004
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0007
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.002.0005
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0002
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0008
Shea, N. J. (2024) · Oxford University Press eBooks · doi:10.1093/9780191997167.003.0003
Understanding Training Behaviour during Transfer Learning by drawing parallels to Grokking
Rath, K., & Gnanasigamani, L. J. (2024) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit61985.2024.10626430
How Do Transformers Model Physics? Investigating the Simple Harmonic Oscillator
Kantamneni, S., Liu, Z., & Tegmark, M. (2024) · Entropy · doi:10.3390/e26110997
Model Mapping: Can We Turn Transformers Into Programs?Preprint
Upadhyay, S., Keigwin, B., Wu, Y., Samkharadze, L., Hu, J., & Ginsburg, E. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5026576
A Non-Ergodic Framework for Understanding Emergent Capabilities in Large Language ModelsPreprint
Marin, J. (2025) · Qeios · doi:10.32388/00jqn6
Easing Optimization Paths: a Circuit PerspectivePreprint
Odonnat, A., Bouaziz, W., & Cabannes, V. (2025) · Qeios · doi:10.32388/2m0wcy
Transparent AI Without Deception, Limitation, and Guidance
Musker, S. (2025) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics65148.2025.11098293
Model Mapping: Can We Turn Transformers Into Programs?Preprint
Upadhyay, S., Keigwin, B., Bandi, C., Wu, Y., Samkharadze, L., Hu, J., & Ginsberg, E. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5361294
Do Llamas understand the periodic table?
Lei, G., & Cooper, S. J. (2025) · Digital Discovery · doi:10.1039/d5dd00374a
Crook, B., & Kästner, L. (2025) · Synthese Library/Synthese library · doi:10.1007/978-3-032-03083-2_20
Artificial Intelligence as an Emergent Order
Berg, C. (2026) · Palgrave studies in classical liberalism · doi:10.1007/978-3-032-22310-4_3
Model evaluation for extreme risks
Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.15324 · Cited by 66 · see citing works
“Another important tool is mechanistic analysis, i.e. studying the model’s weights and activations for understanding how it functions (Nanda et al., 2023; Olah et al., 2020).”
Method
- Authors
- Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.05217
- OpenAlex
- W4316135772
- Cited by
- 51
- References
- 0
- Citations by year
- 2026: 1 · 2025: 15 · 2024: 24 · 2023: 11
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- modular addition, transformer models, Lean 4 formal verification, Mathlib, dyadic rationals, model interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
