CoChat
Sign Up
Cited by 30Explainable Artificial Intelligence (XAI)Software Engineering ResearchEvolutionary Algorithms and Applications

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code

Open access
Michaud, Eric J., Liao, Isaac, Lad, Vedang, Liu, Ziming, Mudide, Anish, Loughridge, Chloe, Guo, Zifan Carl, Kheirkhah, Tara Rezaei, Vukelić, Mateja, Tegmark, Max·2024·Entropy
DOI 10.3390/e26121046PMC PMC11727218
Open this paper in CoChat — free account

Abstract

Can we turn AI black boxes into code? Although this mission sounds extremely challenging, we show that it is not entirely impossible by presenting a proof-of-concept method, MIPS, that can synthesize programs based on the automated mechanistic interpretability of neural networks trained to perform the desired task, auto-distilling the learned algorithm into Python code. We test MIPS on a benchmark of 62 algorithmic tasks that can be learned by an RNN and find it highly complementary to GPT-4: MIPS solves 32 of them, including 13 that are not solved by GPT-4 (which also solves 30). MIPS uses an integer autoencoder to convert the RNN into a finite state machine, then applies Boolean or integer symbolic regression to capture the learned algorithm. As opposed to large language models, this program synthesis technique makes no use of (and is therefore not limited by) human training data such as algorithms and code from GitHub. We discuss opportunities and challenges for scaling up this approach to make machine-learned models more interpretable and trustworthy.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/e26121046

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Program synthesis through mechanistic analysis offers an alternative approach by converting simple trained neural networks into more interpretable, controllable, and verifiable programs.

    “Program synthesis through mechanistic analysis offers an alternative approach by converting simple trained neural networks into more interpretable, controllable, and verifiable programs (Michaud et al., 2024).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Michaud et al., 2024

    The Introduction describes MIPS as distilling simple learned algorithms from neural networks into Python code. Section 3.3 says the resulting programs are more interpretable and verifiable, but our copy does not establish the additional claim that they are more controllable. No page numbers are provided in the indexed copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. Tegmark, M., and Omohundro, S. (2023). Provably safe systems: The only path to controllable agi. arXiv.
  2. Dalrymple, D., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., Omohundro, S., Szegedy, C., Goldhaber, B., and Ammann, N. (2024). Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems. arXiv.
  3. Zhou, 2023, Survey of intelligent program synthesis techniques, Proceedings of the International Conference on Algorithms, High Performance Computing, and Artificial Intelligence (AHPCAI 2023), Volume 12941, 1122
  4. Odena, A., Shi, K., Bieber, D., Singh, R., Sutton, C., and Dai, H. (2020). BUSTLE: Bottom-Up program synthesis through learning-guided exploration. arXiv.
  5. Wu, 2023, Programming by Example Made Easy, ACM Trans. Softw. Eng. Methodol., 33, 1
  6. Sobania, D., Briesch, M., and Rothlauf, F. (2022, January 9–13). Choose your programming copilot: A comparison of the program synthesis performance of github copilot and genetic programming. Proceedings of the Genetic and Evolutionary Computation Conference, Boston, MA, USA. · DOI
  7. Cammarata, N., Goh, G., Carter, S., Schubert, L., Petrov, M., and Olah, C. (2020). Curve Detectors. Distill, Available online: https://distill.pub/2020/circuits/curve-detectors. · DOI · Paper page
  8. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. (2022). Interpretability in the wild: A circuit for indirect object identification in gpt-2 small. arXiv.
  9. Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., and Olah, C. (2021). Multimodal Neurons in Artificial Neural Networks. Distill, Available online: https://distill.pub/2021/multimodal-neurons. · DOI · Paper page
  10. Gurnee, W., and Tegmark, M. (2023). Language models represent space and time. arXiv.
  11. Vafa, K., Chen, J.Y., Kleinberg, J., Mullainathan, S., and Rambachan, A. (2024). Evaluating the World Model Implicit in a Generative Model. arXiv.
  12. Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2022). Discovering latent knowledge in language models without supervision. arXiv.
  13. Marks, S., and Tegmark, M. (2023). The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv.
  14. McGrath, 2022, Acquisition of chess knowledge in alphazero, Proc. Natl. Acad. Sci. USA, 119, e2206625119, 10.1073/pnas.2206625119 · DOI · Paper page
  15. Toshniwal, S., Wiseman, S., Livescu, K., and Gimpel, K. (March, January 22). Chess as a testbed for language model state tracking. Proceedings of the AAAI Conference on Artificial Intelligence, Virtually.
  16. Li, K., Hopkins, A.K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. (2022). Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv.
  17. Nanda, N., Chan, L., Liberum, T., Smith, J., and Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv.
  18. Liu, Z., Kitouni, O., Nolte, N., Michaud, E.J., Tegmark, M., and Williams, M. (2022, January 28). Towards Understanding Grokking: An Effective Theory of Representation Learning. Proceedings of the Thirty-Sixth Conference on Neural Information Processing Systems, New Orleans, LA, USA.
  19. Zhong, Z., Liu, Z., Tegmark, M., and Andreas, J. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems: 37th Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, LA, USA, 10–16 December 2023.
  20. Quirke, P., and Barez, F. (2023). Understanding Addition in Transformers. arXiv.
  21. Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (2023, January 23–29). A Toy Model of Universality: Reverse Engineering how Networks Learn Group Operations. Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA. PMLR (Proceedings of Machine Learning Research) 2023.
  22. Hanna, M., Liu, O., and Variengien, A. (2023). How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. arXiv.
  23. Charton, F. (2023). Can transformers learn the greatest common divisor?. arXiv.
  24. Lindner, D., Kramár, J., Farquhar, S., Rahtz, M., McGrath, T., and Mikulik, V. (2023). Tracr: Compiled transformers as a laboratory for interpretability. arXiv.
  25. Friedman, 2023, Learning transformer programs, Adv. Neural Inf. Process. Syst., 36, 49044
  26. Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. (2024, November 26). Language Models Can Explain Neurons in Language Models. Available online: https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html.
  27. Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. (2023). Sparse autoencoders find highly interpretable features in language models. arXiv.
  28. Conmy, A., Mavor-Parker, A.N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. (2023). Towards automated circuit discovery for mechanistic interpretability. arXiv.
  29. Syed, A., Rager, C., and Conmy, A. (2023). Attribution Patching Outperforms Automated Circuit Discovery. arXiv. · DOI · Paper page
  30. Marks, S., Rager, C., Michaud, E.J., Belinkov, Y., Bau, D., and Mueller, A. (2024). Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv.
  31. Karpathy, A., Johnson, J., and Fei-Fei, L. (2015). Visualizing and understanding recurrent networks. arXiv.
  32. Strobelt, 2018, LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks, IEEE Trans. Vis. Comput. Graph., 24, 667, 10.1109/TVCG.2017.2744158 · DOI
  33. Giles, 1995, Learning a class of large finite state machines with a recurrent neural network, Neural Netw., 8, 1359, 10.1016/0893-6080(95)00041-0 · DOI
  34. Wang, Q., Zhang, K., Ororbia II, A.G., Xing, X., Liu, X., and Giles, C.L. (2017). An empirical evaluation of rule extraction from recurrent neural networks. arXiv. · DOI
  35. Weiss, G., Goldberg, Y., and Yahav, E. (2018, January 10–15). Extracting automata from recurrent neural networks using queries and counterexamples. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.
  36. Oliva, C., and Lago-Fernández, L.F. (2019, January 17–19). On the interpretation of recurrent neural networks as finite state machines. Proceedings of the Artificial Neural Networks and Machine Learning–ICANN 2019: Theoretical Neural Computation: 28th International Conference on Artificial Neural Networks, Munich, Germany. Proceedings, Part I 28. · DOI
  37. Muvskardin, E., Aichernig, B.K., Pill, I., and Tappler, M. (2022, January 7–10). Learning finite state models from recurrent neural networks. Proceedings of the International Conference on Integrated Formal Methods, Lugano, Switzerland. · DOI
  38. Udrescu, 2020, AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity, Adv. Neural Inf. Process. Syst., 33, 4860
  39. Cranmer, M. (2023). Interpretable machine learning for science with PySR and SymbolicRegression. jl. arXiv.
  40. Cranmer, 2020, Discovering symbolic models from deep learning with inductive biases, Adv. Neural Inf. Process. Syst., 33, 17429
  41. Ma, 2022, Evolving symbolic density functionals, Sci. Adv., 8, eabq0279, 10.1126/sciadv.abq0279 · DOI
  42. Gu, A., and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv.

Cited by

Sorted by citations

Metadata

Authors
Michaud, E. J., Liao, I., Lad, V., Liu, Z., Mudide, A., Loughridge, C., Guo, Z. C., Kheirkhah, T. R., Vukelić, M., & Tegmark, M.
Affiliations
  • The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (US)
  • Massachusetts Institute of Technology (US)
  • Harvard University (US)
Corresponding author
Tegmark, Max
ORCID
Venue
Entropy
Publisher
MDPI AG
Published
2024-12-02
Volume
26
Issue
12
Pages
1046
ISSN
1099-4300 (ISSN-L)
DOI
10.3390/e26121046
PMC
PMC11727218
OpenAlex
W4404928080
Cited by
30
References
2
Citations by year
2026: 9 · 2025: 11 · 2024: 10
FWCI
3.24
Type
journal article
Language
en
Open access
Licence
Locations
  • Entropy (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Erik Otto, Jaan Tallinn, the Rothberg Family Fund for Cognitive Science, the NSF Graduate Research Fellowship (2141064, PHY-2019786)
  • IAIFI through NSF (2141064, PHY-2019786)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
program synthesis, mechanistic interpretability, neural network interpretability, symbolic regression, finite automata, recurrent neural networks, GPT-4 comparison, model interpretability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review