CoChat
Sign Up
PreprintCited by 3Domain Adaptation and Few-Shot LearningReinforcement Learning in RoboticsAI-based Problem Solving and Planning

What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models

Open access
Vafa, Keyon, Chang, Peter G., Rambachan, Ashesh, Mullainathan, Sendhil·2025·arXiv
DOI 10.48550/arxiv.2507.06952arXiv 2507.06952
Open this paper in CoChat — free accountOpen PDF

Abstract

Foundation models are premised on the idea that sequence prediction can uncover deeper domain understanding, much like how Kepler's predictions of planetary motion later led to the discovery of Newtonian mechanics. However, evaluating whether these models truly capture deeper structure remains a challenge. We develop a technique for evaluating foundation models that examines how they adapt to synthetic datasets generated from some postulated world model. Our technique measures whether the foundation model's inductive bias aligns with the world model, and so we refer to it as an inductive bias probe. Across multiple domains, we find that foundation models can excel at their training tasks yet fail to develop inductive biases towards the underlying world model when adapted to new tasks. We particularly find that foundation models trained on orbital trajectories consistently fail to apply Newtonian mechanics when adapted to new physics tasks. Further analysis reveals that these models behave as if they develop task-specific heuristics that fail to generalize.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.06952

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 60

  1. Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., and Søgaard, A. Can language models encode perceptual structure without grounding? A case study in color. arXiv preprint arXiv:2109.06129 , 2021. · arXiv
  2. Achille, A., Paolini, G., Mbeng, G., and Soatto, S. The information complexity of learning tasks, their structure and their distance. Information and Inference: A Journal of the IMA , 10(1):51-72, 2021.
  3. Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. 2018. arXiv preprint arXiv:1610.01644 , 2018. · arXiv
  4. Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics , 48(1):207-219, 2022.
  5. Belyshev, A., Kovrigin, A., and Ustyuzhanin, A. Beyond dynamics: Learning to discover conservation principles. Machine Learning: Science and Technology , 5(2):025055, 2024.
  6. Benegas, G., Batra, S. S., and Song, Y. S. DNA language models are powerful predictors of genome-wide variant effects. Proceedings of the National Academy of Sciences , 120(44):e2311219120, 2023.
  7. Bengio, Y., Deleu, T., Rahaman, N., Ke, R., Lachapelle, S., Bilaniuk, O., Goyal, A., and Pal, C. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv:1901.10912 , 2019. · arXiv
  8. Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. Taken out of context: On measuring situational awareness in llms. arXiv preprint arXiv:2309.00667 , 2023. · arXiv
  9. Black, E., Raghavan, M., and Barocas, S. Model multiplicity: Opportunities, concerns, and solutions. In ACM Conference on Fairness, Accountability, and Transparency , pp. 850-863, 2022.
  10. Boiko, D. A., MacKnight, R., Kline, B., and Gomes, G. Autonomous chemical research with large language models. Nature , 624(7992):570-578, 2023.
  11. Breiman, L. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science , 16(3):199-231, 2001.
  12. Cao, S., Sanh, V., and Rush, A. M. Low-complexity probing via finding subnetworks. arXiv preprint arXiv:2104.03514 , 2021. · arXiv
  13. Casper, S., Bu, T., Li, Y., Li, J., Zhang, K., Hariharan, K., and Hadfield-Menell, D. Red teaming deep neural networks with feature synthesis tools. Advances in Neural Information Processing Systems , 36:80470-80516, 2023.
  14. Chen, B., Huang, K., Raghupathi, S., Chandratreya, I., Du, Q., and Lipson, H. Automated discovery of fundamental variables hidden in experimental data. Nature Computational Science , 2(7):433-442, 2022.
  15. Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., Sorger, P. K., and AlQuraishi, M. Single-sequence protein structure prediction using a language model and deep learning. Nature Biotechnology , 40(11):1617-1623, 2022.
  16. Cranmer, M. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01582 , 2023. · arXiv
  17. Cranmer, M., Sanchez Gonzalez, A., Battaglia, P., Xu, R., Cranmer, K., Spergel, D., and Ho, S. Discovering symbolic models from deep learning with inductive biases. Neural Information Processing Systems , 33:1742917442, 2020.
  18. Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 , 2023. · arXiv
  19. D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research , 23(226):1-61, 2022.
  20. Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060 , 2024. · arXiv
  21. Elman, J. L. Finding structure in time. Cognitive science , 14(2):179-211, 1990.
  22. Gingerich, O. The book nobody read: Chasing the revolutions of Nicolaus Copernicus . Bloomsbury Publishing USA, 2004.
  23. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023. · arXiv
  24. Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv preprint arXiv:2310.02207 , 2023. · arXiv
  25. Hao, Z., Liu, S., Zhang, Y., Ying, C., Feng, Y., Su, H., and Zhu, J. Physics-informed machine learning: A survey on problems, methods and applications. arXiv preprint arXiv:2211.08064 , 2022. · arXiv
  26. Hazineh, D. S., Zhang, Z., and Chiu, J. Linear latent world models in simple transformers: A case study on OthelloGPT. arXiv preprint arXiv:2310.07582 , 2023. · arXiv
  27. Hewitt, J. and Liang, P. Designing and interpreting probes with control tasks. arXiv preprint arXiv:1909.03368 , 2019. · arXiv
  28. Hochreiter, S. Long short-term memory. Neural Computation MIT-Press , 1997.
  29. Iten, R., Metger, T., Wilming, H., del Rio, L., and Renner, R. Discovering Physical Concepts with Neural Networks. Physical Review Letters , 124(1):010508, January 2020. doi: 10.1103/PhysRevLett.124.010508. · DOI
  30. Jablonka, K. M., Schwaller, P., Ortega-Guerrero, A., and Smit, B. Leveraging large language models for predictive chemistry. Nature Machine Intelligence , pp. 1-9, 2024.
  31. Karvonen, A. Emergent world models and latent variable estimation in chess-playing language models. arXiv preprint arXiv:2403.15498 , 2024. · arXiv
  32. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
  33. Kipping, D. M. Parametrizing the exoplanet eccentricity distribution with the beta distribution. Monthly Notices of the Royal Astronomical Society: Letters , 434(1):L51L55, 2013.
  34. Koestler, A. The Sleepwalkers: A History of Man's Changing Vision of the Universe . Hutchinson, London, 1959. First edition.
  35. LeCun, Y. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. OpenReview , 62(1):1-62, 2022.
  36. Lee, C., Rush, A. M., and Vafa, K. Critical thinking: Which kinds of complexity govern optimal reasoning length? In International Joint Conference on Natural Language Processing and Asia-Pacific Chapter of the Association for Computational Linguistics , 2025.
  37. Lemos, P., Jeffrey, N., Cranmer, M., Ho, S., and Battaglia, P. Rediscovering orbital mechanics with machine learning. Machine Learning: Science and Technology , 4(4):045002, 2023.
  38. Li, B. Z., Nye, M., and Andreas, J. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737 , 2021. · arXiv
  39. Li, K., Hopkins, A. K., Bau, D., Vi´ egas, F., Pfister, H., and Wattenberg, M. Emergent world representations: Exploring a sequence model trained on a synthetic task. In International Conference on Learning Representations , 2023.
  40. Jin, C. and Rinard, M. Evidence of meaning in language models trained on programs. arXiv preprint arXiv:2305.11169 , 2023. · arXiv
  41. Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.10749 , 2022. · arXiv
  42. Liu, Z. and Tegmark, M. Ai poincare: Machine learning conservation laws from trajectories. arXiv:2011.04698 , 2021. · arXiv
  43. Marx, C., Calmon, F., and Ustun, B. Predictive multiplicity in classification. In International Conference on Machine Learning , pp. 6765-6774. PMLR, 2020. jylin04, JackS, Karvonen, A., and Rager, C. Othellogpt learned a bag of heuristics, jul 2024. URL https://www.lesswrong. com/posts/gcpNuEZnxAPayaKBY/ othellogpt-learned-a-bag-of-heuristics-1 Posted on LessWrong.
  44. Kansky, K., Silver, T., M´ ely, D. A., Eldawy, M., L´ azaroGredilla, M., Lou, X., Dorfman, N., Sidor, S., Phoenix, S., and George, D. Schema Networks: Zero-shot transfer with a cenerative causal model of intuitive physics. In International Conference on Machine Learning , pp. 18091818. PMLR, July 2017.
  45. McCoy, R. T., Pavlick, E., and Linzen, T. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the Association for Computational Linguistics , 2019.
  46. Merrill, W. and Sabharwal, A. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics , 11:531545, 2023.
  47. Merrill, W., Petty, J., and Sabharwal, A. The illusion of state in state-space models. arXiv preprint arXiv:2404.08819 , 2024. · arXiv
  48. Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 , 2023a. · arXiv
  49. Nanda, N., Lee, A., and Wattenberg, M. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941 , 2023b. · arXiv
  50. Nikankin, Y., Reusch, A., Mueller, A., and Belinkov, Y. Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprint arXiv:2410.21272 , 2024. · arXiv
  51. Patel, R. and Pavlick, E. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations , 2022.
  52. Sch¨ olkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE , 109(5): 612-634, 2021.
  53. Spies, A. F., Edwards, W., Ivanitskiy, M. I., Skapars, A., R¨ auker, T., Inoue, K., Russo, A., and Shanahan, M. Transformers use causal world models in maze-solving tasks. arXiv preprint arXiv:2412.11867 , 2024. · arXiv
  54. Suzgun, M., Belinkov, Y., and Shieber, S. M. On evaluating the generalization of LSTM models in formal languages. arXiv preprint arXiv:1811.01001 , 2018. · arXiv
  55. Toshniwal, S., Wiseman, S., Livescu, K., and Gimpel, K. Chess as a testbed for language model state tracking. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pp. 11385-11393, 2022.
  56. Udrescu, S.-M. and Tegmark, M. AI Feynman: A physicsinspired method for symbolic regression. Science Advances , 6(16):eaay2631, April 2020. doi: 10.1126/sciadv. aay2631. · DOI
  57. Vafa, K., Chen, J. Y., Kleinberg, J., Mullainathan, S., and Rambachan, A. Evaluating the world model implicit in a generative model. Neural Information Processing Systems , 2024.
  58. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Neural Information Processing Systems , 2017.
  59. Wolpert, D. H. The lack of a priori distinctions between learning algorithms. Neural Computation , 8(7):13411390, 10 1996. ISSN 0899-7667. doi: 10.1162/neco. 1996.8.7.1341. URL https://doi.org/10.1162/ neco.1996.8.7.1341 . · DOI
  60. Wu, T. and Tegmark, M. Toward an artificial intelligence physicist for unsupervised learning. Physical Review E , 100(3):033311, 2019.

Cited by

Sorted by citations

Authors
Vafa, K., Chang, P. G., Rambachan, A., & Mullainathan, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2507.06952
arXiv
2507.06952
OpenAlex
W4416283360
Cited by
3
References
0
Citations by year
2026: 2 · 2025: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
foundation models, inductive bias, world models, synthetic data, classical mechanics, sequence prediction, model evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review