Tracr: Compiled Transformers as a Laboratory for Interpretability
Open accessAbstract
We show how to "compile" human-readable programs into standard decoder-only transformer models. Our compiler, Tracr, generates models with known structure. This structure can be used to design experiments. For example, we use it to study "superposition" in transformers that execute multi-step algorithms. Additionally, the known structure of Tracr-compiled models can serve as ground-truth for evaluating interpretability methods. Commonly, because the "programs" learned by transformers are unknown it is unclear whether an interpretation succeeded. We demonstrate our approach by implementing and examining programs including computing token frequencies, sorting, and parenthesis checking. We provide an open-source implementation of Tracr at https://github.com/google-deepmind/tracr.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RASP-L is a human-readable programming language whose programs can be compiled into Transformer weights, with each line compiling into at most one Transformer layer.
“RASP-L is a human-readable programming language which defines programs that can be compiled into Transformer weightsfoot_0 , such that each line of RASP-L compiles into at most one Transformer layer.”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Lindner et al., 2023
Section 2.2 describes RASP—not RASP-L—as human-readable, and the Machine code section says Tracr compiles RASP programs into Transformer weights. The paper does not establish a one-line-to-at-most-one-layer rule; Appendix D.2 describes a selector-width primitive that compiles into both an attention layer and an MLP layer. Page numbers were not available in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe Transformer implementation of selector width requires weights which grow linearly with sequence length.
“since the Transformer implementation of selector width requires weights which grow linearly with sequence length (Lindner et al., 2023).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Lindner et al., 2023
Appendix D.2 describes Tracr’s selector-width implementation using an attention layer and an MLP layer. It does not state that the implementation requires weights that grow linearly with sequence length; searches for weight scaling did not establish that part of the claim. (LLM, conf=0.81)
Experimental — accuracy not yet measuredLindner et al. (2023) provide an explicit compiler from RASP programs to Transformer weights.
“In factLindner et al. (2023) provide an explicit compiler from RASP programs to Transformer weights.”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Lindner et al. (2023)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 31
- J. Adebayo, M. Muelly, I. Liccardi, and B. Kim. Debugging tests for model explanations. In Advances in Neural Information Processing Systems, 2020.
- M. Aharon, M. Elad, and A. Bruckstein. K-SVD: An algorithm for designing overcomplete dic- tionaries for sparse representation. IEEE Transactions on signal processing, 54(11):4311-4322, 2006.
- E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in- context learning? Investigations with linear models. In International Conference on Learning Representations (ICLR), 2023.
- D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- Y. Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
- Y. Belinkov and J. Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72, 2019.
- N. Cammarata, S. Carter, G. Goh, C. Olah, M. Petrov, L. Schubert, C. Voss, B. Egan, and S. K. Lim. Thread: Circuits. Distill, 2020. URL https://distill.pub/2020/circuits.
- D. V. Carvalho, E. M. Pereira, and J. S. Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 8(8):832, 2019.
- A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, 2023.
- M. Danilevsky, K. Qian, R. Aharonov, Y. Katsis, B. Kawas, and P. Sen. A survey of the state of explainable AI for natural language processing. AACL-IJCNLP 2020, 2020.
- D. L. Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4):1289-1306, 2006.
- D. Friedman, A. Wettig, and D. Chen. Learning transformer programs. In Advances in Neural Information Processing Systems, 2023.
- A. Giannou, S. Rajput, J.-y. Sohn, K. Lee, J. D. Lee, and D. Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning (ICML), 2023.
- A. Jacovi and Y. Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
- M. L. Leavitt and A. Morcos. Towards falsifiable interpretability research. In NeurIPS Workshop: ML Retrospectives, Surveys & Meta-Analyses (ML-RSA), 2020.
- Z. C. Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57, 2018.
- K. Meng, D. Bau, A. J. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022.
- W. Merrill, A. Sabharwal, and N. A. Smith. Saturated transformers are constant-depth threshold circuits. Transactions of the Association for Computational Linguistics, 10:843-856, 2022.
- S. Mohseni, N. Zarei, and E. D. Ragan. A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Transactions on Interactive Intelligent Systems, 11 (3-4):1-45, 2021.
- N. Nanda, L. Chan, T. Liberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. In International Conference on Learning Representations (ICLR), 2023.
- I. E. Nielsen, D. Dera, G. Rasool, R. P. Ramachandran, and N. C. Bouaynaya. Robust explainability: A tutorial on gradient-based attribution methods for deep neural networks. IEEE Signal Processing Magazine, 39(4):73-84, 2022.
- T. Räukur, A. Ho, S. Casper, and D. Hadfield-Menell. Toward transparent AI: A survey on interpreting the inner structures of deep neural networks. In IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2023.
- A. Rogers, O. Kovaleva, and A. Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
- A. Scherlis, K. Sachan, A. S. Jermyn, J. Benton, and B. Shlegeris. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892, 2022. · arXiv
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
- K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations (ICLR), 2023.
- C. Wei, Y. Chen, and T. Ma. Statistically meaningful approximation: a case study on approximating Turing machines with transformers. In Advances in Neural Information Processing Systems, 2022.
- G. Weiss, Y. Goldberg, and E. Yahav. Thinking like transformers. In International Conference on Machine Learning (ICML), 2021.
- F. Yang, M. Du, and X. Hu. Evaluating explanation without ground truth in interpretable machine learning. arXiv preprint arXiv:1907.06831, 2019. · arXiv
- M. Yang and B. Kim. Benchmarking attribution methods with relative feature importance. arXiv preprint arXiv:1907.09701, 2019. · arXiv
- Y. Zhou, S. Booth, M. T. Ribeiro, and J. Shah. Do feature attribution methods correctly attribute features? In AAAI Conference on Artificial Intelligence, 2022.
Cited by
Sorted by citations
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10 · see citing works
“…work, and thus focus on the external performance of the trained models in the out-of-distribution setting; whether trained Transformers’ internals are at all similar to the internals of a compiled RASP program remains an open question (some such investigation was done in Lindner et al. (2023)).”
BackgroundLanguage models need inductive biases to count inductively
Chang, Y., & Bisk, Y. (2024) · arXiv · doi:10.48550/arxiv.2405.20131
“…on Transformer Expressivity Transformer expressivity can be formally analyzed from the perspectives of functional libraries (Weiss et al., 2021; Lindner et al., 2023; Zhou et al., 2023), boolean circuits (Cong et al., 1996; Yang & Chiang, 2024; Strobl et al., 2024; Merrill et al., 2022), or…”
BackgroundMethod
- Authors
- Lindner, D., Kramár, J., Farquhar, S., Rahtz, M., McGrath, T., & Mikulik, V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 37876-37899
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2301.05062
- OpenAlex
- W7133187725
- Cited by
- 8
- References
- 0
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 1
- FWCI
- 1.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- interpretability, transformer models, model compression, neural architecture search
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
