PreprintCited by 3Neural Networks and ApplicationsEvolutionary Algorithms and ApplicationsStochastic Gradient Optimization Techniques

Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis

Open access
Kumar, Akarsh, Clune, Jeff, Lehman, Joel, Stanley, Kenneth O.·2025·arXiv

Abstract

Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance. But does better performance necessarily imply better internal representations? While the representational optimist assumes it must, this position paper challenges that view. We compare neural networks evolved through an open-ended search process to networks trained via conventional stochastic gradient descent (SGD) on the simple task of generating a single image. This minimal setup offers a unique advantage: each hidden neuron's full functional behavior can be easily visualized as an image, thus revealing how the network's output behavior is internally constructed neuron by neuron. The result is striking: while both networks produce the same output behavior, their internal representations differ dramatically. The SGD-trained networks exhibit a form of disorganization that we term fractured entangled representation (FER). Interestingly, the evolved networks largely lack FER, even approaching a unified factored representation (UFR). In large models, FER may be degrading core model capacities like generalization, creativity, and (continual) learning. Therefore, understanding and mitigating FER could be critical to the future of representation learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.11581

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Functional agreement and output behavior can persist even when internal representations are misaligned or entangled, suggesting that behavioral compatibility does not imply shared structure.

    “Smith et al. [86] and Kumar et al. [52] show that functional agreement and output behavior can persist even when internal representations are misaligned or entangled, suggesting that behavioral compatibility does not imply shared structure.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 4× · see citing works

    “At the heart of representational optimism is the belief that representation improves with scaling in deep learning (Hoffmann et al., 2022; Huh et al., 2024; Kaplan et al., 2020).”
    Influential

Flagged first, then by citations

Research works printed but not linked: 92

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774 . · arXiv
  2. Bedau, M. A., Snyder, E., and Packard, N. H. (1998). A classification of long-term evolutionary dynamics. In Artificial life vi: Proceedings of the sixth international conference on artificial life , volume 6, page 228. MIT Press.
  3. Bengio, Y., Courville, A., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828.
  4. Bereska, L. and Gavves, E. (2024). Mechanistic interpretability for ai safety-a review. arXiv preprint arXiv:2404.14082 . · arXiv
  5. Boden, M. A. (2004). The creative mind: Myths and mechanisms . Routledge.
  6. Brant, J. C. and Stanley, K. O. (2017). Minimal criterion coevolution: a new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference , GECCO '17, page 67-74, New York, NY, USA. Association for Computing Machinery.
  7. Brant, J. C. and Stanley, K. O. (2020). Diversity preservation in minimal criterion coevolution through resource limitation. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference , GECCO '20, page 58-66, New York, NY, USA. Association for Computing Machinery.
  8. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.
  9. Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. (2018). Understanding disentangling in beta -vae. arXiv preprint arXiv:1804.03599 . · arXiv
  10. Caron, M., Touvron, H., Misra, I., J´ egou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660.
  11. Carter, S., Armstrong, Z., Schubert, L., Johnson, I., and Olah, C. (2019). Activation atlas. Distill , 4(3):e15.
  12. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PmLR.
  13. Clune, J., Stanley, K. O., Pennock, R. T., and Ofria, C. (2011). On the performance of indirect encoding across the continuum of regularity. IEEE Transactions on Evolutionary Computation , 15(3):346-367.
  14. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 . · arXiv
  15. Dawkins, R. (2019). The evolution of evolvability. In Artificial life , pages 201-220. Routledge.
  16. Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. (2023). Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems , 36:70293-70332.
  17. Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. (2021). First return, then explore. Nature , 590(7847):580-586.
  18. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. (2022). Toy models of superposition. arXiv preprint arXiv:2209.10652 . · arXiv
  19. Elsken, T., Metzen, J. H., and Hutter, F. (2019). Neural architecture search: A survey. Journal of Machine Learning Research , 20(55):1-21.
  20. Enquist, M. and Arak, A. (1994). Symmetry, beauty and evolution. Nature , 372(6502):169-172.
  21. Faldor, M., Zhang, J., Cully, A., and Clune, J. (2025). Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code. International Conference on Learning Representations (ICLR) .
  22. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673.
  23. Grill, J.-B., Strub, F., Altch´ e, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. (2020). Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284.
  24. Han, S., Pari, J., Gershman, S. J., and Agrawal, P. (2025). General reasoning requires learning to reason from the get-go. arXiv preprint arXiv:2502.19402 . · arXiv
  25. H¨ ark¨ onen, E., Hertzmann, A., Lehtinen, J., and Paris, S. (2020). Ganspace: Discovering interpretable gan controls. Advances in neural information processing systems , 33:9841-9850.
  26. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738.
  27. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778.
  28. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations .
  29. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 . · arXiv
  30. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems , 43(2):1-55.
  31. Hughes, E., Dennis, M., Parker-Holder, J., Behbahani, F., Mavalankar, A., Shi, Y., Schaul, T., and Rocktaschel, T. (2024). Open-endedness is essential for artificial superhuman intelligence. arXiv preprint arXiv:2406.04268 . · arXiv
  32. Huh, M., Cheung, B., Wang, T., and Isola, P. (2024). The platonic representation hypothesis. arXiv preprint arXiv:2405.07987 . · arXiv
  33. Huizinga, J., Stanley, K. O., and Clune, J. (2018). The emergence of canalization and evolvability in an open-ended, interactive evolutionary system. Artificial life , 24(3):157-181.
  34. Hutter, M. (2000). A theory of universal artificial intelligence based on algorithmic complexity. arXiv preprint cs/0004001 . · arXiv
  35. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural computation , 3(1):79-87.
  36. Jahanian, A., Chai, L., and Isola, P. (2019). On the' steerability' of generative adversarial networks. arXiv preprint arXiv:1907.07171 . · arXiv
  37. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. (2024). Mixtral of experts. arXiv preprint arXiv:2401.04088 . jylin04, JackS, Karvonen, A., and Can (2024). Othellogpt learned a bag of heuristics. AI Alignment Forum. · arXiv
  38. Kambhampati, S. (2024). Can large language models reason and plan? Annals of the New York Academy of Sciences , 1534(1):15-18.
  39. Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410.
  40. Kirschner, M. and Gerhart, J. (1998). Evolvability. Proceedings of the National Academy of Sciences , 95(15):8420-8427.
  41. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541-551.
  42. LeCun, Y., Bottou, L., Orr, G. B., and M¨ uller, K.-R. (2002). Efficient backprop. In Neural networks: Tricks of the trade , pages 9-50. Springer.
  43. Lehman, J., Chen, J., Clune, J., and Stanley, K. O. (2018). Es is more than just a traditional finite-difference approximator. In Proceedings of the genetic and evolutionary computation conference , pages 450-457.
  44. Lehman, J., Meyerson, E., El-Gaaly, T., Stanley, K. O., and Ziyaee, T. (2025). Evolution and the knightian blindspot of machine learning. arXiv preprint arXiv:2501.13075 . · arXiv
  45. Lehman, J. and Stanley, K. O. (2010). Efficiently evolving programs through the search for novelty. In Proceedings of the 12th annual conference on Genetic and evolutionary computation , pages 837-844.
  46. Lehman, J. and Stanley, K. O. (2011). Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation , 19(2):189-223.
  47. Li, K., Hopkins, A. K., Bau, D., Vi´ egas, F., Pfister, H., and Wattenberg, M. (2023). Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR .
  48. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. (2024). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 . · arXiv
  49. Liu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark, M., and Williams, M. (2022a). Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems , 35:34651-34663.
  50. Liu, Z., Michaud, E. J., and Tegmark, M. (2022b). Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117 . · arXiv
  51. Maley, C. C. (1999). Four steps toward open-ended evolution. In Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation-Volume 2 , pages 1336-1343.
  52. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. (2023). Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638 . · arXiv
  53. Miconi, T. (2008). The road to everywhere: Evolution, complexity and progress in natural and artificial systems . PhD thesis, University of Birmingham.
  54. Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. (2024). Gsmsymbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 . · arXiv
  55. Mouret, J.-B. and Clune, J. (2015). Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 . · arXiv
  56. Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 . · arXiv
  57. Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems , 29.
  58. Nguyen, A. M., Yosinski, J., and Clune, J. (2015). Innovation engines: Automated creativity and improved stochastic optimization via deep learning. In Proceedings of the 2015 annual conference on genetic and evolutionary computation , pages 959-966.
  59. Nikankin, Y., Reusch, A., Mueller, A., and Belinkov, Y. (2024). Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprint arXiv:2410.21272 . · arXiv
  60. Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020). Zoom in: An introduction to circuits. Distill , 5(3):e00024-001.
  61. Olah, C., Mordvintsev, A., and Schubert, L. (2017). Feature visualization. Distill , 2(11):e7.
  62. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. (2022). In-context learning and induction heads. arXiv preprint arXiv:2209.11895 . · arXiv
  63. Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 . · arXiv
  64. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PmLR.
  65. Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 . · arXiv
  66. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
  67. Rissanen, J. (1978). Modeling by shortest data description. Automatica , 14(5):465-471.
  68. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695.
  69. Schmidhuber, J. (1997). Low-complexity art. Leonardo , 30(2):97-103.
  70. Secretan, J., Beato, N., D Ambrosio, D. B., Rodriguez, A., Campbell, A., and Stanley, K. O. (2008). Picbreeder: evolving pictures collaboratively online. In Proceedings of the SIGCHI conference on human factors in computing systems , pages 1759-1768.
  71. Secretan, J., Beato, N., D'Ambrosio, D. B., Rodriguez, A., Campbell, A., Folsom-Kovarik, J. T., and Stanley, K. O. (2011). Picbreeder: A case study in collaborative evolutionary exploration of design space. Evolutionary computation , 19(3):373-403.
  72. Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025). Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496 . · arXiv
  73. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 . · arXiv
  74. Solomonoff, R. J. (1964). A formal theory of inductive inference. part i. Information and control , 7(1):1-22.
  75. Soviany, P., Ionescu, R. T., Rota, P., and Sebe, N. (2022). Curriculum learning: A survey. International Journal of Computer Vision , 130(6):1526-1565.
  76. Standish, R. K. (2003). Open-ended artificial evolution. International Journal of Computational Intelligence and Applications , 3(02):167-175.
  77. Stanley, K. O. and Miikkulainen, R. (2002). Evolving neural networks through augmenting topologies. Evolutionary computation , 10(2):99-127.
  78. Stanley, K. O. and Miikkulainen, R. (2003). A taxonomy for artificial embryogeny. Artificial life , 9(2):93-130.
  79. Taylor, T. (2019). Evolutionary innovations and where to find them: Routes to open-ended evolution in natural and artificial systems. Artificial life , 25(2):207-224.
  80. Taylor, T., Bedau, M., Channon, A., Ackley, D., Banzhaf, W., Beslon, G., Dolson, E., Froese, T., Hickinbotham, S., Ikegami, T., McMullin, B., Packard, N., Rasmussen, S., Virgo, N., Agmon, E., Clark, E., McGregor, S., Ofria, C., Ropella, G., Spector, L., Stanley, K. O., Stanton, A., Timperley, C., Vostinar, A., and Wiser, M. (2016). Open-ended evolution: Perspectives from the oee workshop in york. Artificial Life , 22(3):408-423.
  81. Team, A. A., Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023). Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608 . · arXiv
  82. Vafa, K., Chen, J., Rambachan, A., Kleinberg, J., and Mullainathan, S. (2024). Evaluating the world model implicit in a generative model. Advances in Neural Information Processing Systems , 37:26941-26975.
  83. Varma, V., Shah, R., Kenton, Z., Kram´ ar, J., and Kumar, R. (2023). Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390 . · arXiv
  84. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, glyph[suppress] L., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30.
  85. Wagner, G. P. and Altenberg, L. (1996). Perspective: complex adaptations and the evolution of evolvability. Evolution , 50(3):967-976.
  86. Wang, R., Lehman, J., Clune, J., and Stanley, K. O. (2019). Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753 . · arXiv
  87. Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., and Stanley, K. (2020). Enhanced poet: Openended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning , pages 9940-9951. PMLR.
  88. Wolpert, L., Tickle, C., and Arias, A. M. (2015). Principles of development . Oxford University Press, USA.
  89. Woolley, B. G. and Stanley, K. O. (2011). On the deleterious effects of a priori objectives on evolution and representation. In Proceedings of the 13th annual conference on Genetic and evolutionary computation , pages 957-964.
  90. Wu, Z., Qiu, L., Ross, A., Aky¨ urek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. (2024). Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 1819-1862.
  91. Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H. (2015). Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 . · arXiv
  92. Zhou, L., Schellaert, W., Mart´ ınez-Plumed, F., Moros-Daval, Y., Ferri, C., and Hern´ andez-Orallo, J. (2024). Larger and more instructable language models become less reliable. Nature , 634(8032):61-68.

Cited by

Sorted by citations

Authors
Kumar, A., Clune, J., Lehman, J., & Stanley, K. O.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4417299644
Cited by
2
References
1
Citations by year
2026: 1 · 2025: 1
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
neuroevolution, stochastic gradient descent, image generation, factored representation, representation learning, generalization, continual learning, model capacity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review