Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
Open accessAbstract
Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance. But does better performance necessarily imply better internal representations? While the representational optimist assumes it must, this position paper challenges that view. We compare neural networks evolved through an open-ended search process to networks trained via conventional stochastic gradient descent (SGD) on the simple task of generating a single image. This minimal setup offers a unique advantage: each hidden neuron's full functional behavior can be easily visualized as an image, thus revealing how the network's output behavior is internally constructed neuron by neuron. The result is striking: while both networks produce the same output behavior, their internal representations differ dramatically. The SGD-trained networks exhibit a form of disorganization that we term fractured entangled representation (FER). Interestingly, the evolved networks largely lack FER, even approaching a unified factored representation (UFR). In large models, FER may be degrading core model capacities like generalization, creativity, and (continual) learning. Therefore, understanding and mitigating FER could be critical to the future of representation learning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Functional agreement and output behavior can persist even when internal representations are misaligned or entangled, suggesting that behavioral compatibility does not imply shared structure.
“Smith et al. [86] and Kumar et al. [52] show that functional agreement and output behavior can persist even when internal representations are misaligned or entangled, suggesting that behavioral compatibility does not imply shared structure.”
From Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale (2026) · cites this paper as Kumar et al. [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 4× · see citing works
“At the heart of representational optimism is the belief that representation improves with scaling in deep learning (Hoffmann et al., 2022; Huh et al., 2024; Kaplan et al., 2020).”
Influential
Flagged first, then by citations
Research works printed but not linked: 92
- Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774 . · arXiv
- Bedau, M. A., Snyder, E., and Packard, N. H. (1998). A classification of long-term evolutionary dynamics. In Artificial life vi: Proceedings of the sixth international conference on artificial life , volume 6, page 228. MIT Press.
- Bengio, Y., Courville, A., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828.
- Bereska, L. and Gavves, E. (2024). Mechanistic interpretability for ai safety-a review. arXiv preprint arXiv:2404.14082 . · arXiv
- Boden, M. A. (2004). The creative mind: Myths and mechanisms . Routledge.
- Brant, J. C. and Stanley, K. O. (2017). Minimal criterion coevolution: a new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference , GECCO '17, page 67-74, New York, NY, USA. Association for Computing Machinery.
- Brant, J. C. and Stanley, K. O. (2020). Diversity preservation in minimal criterion coevolution through resource limitation. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference , GECCO '20, page 58-66, New York, NY, USA. Association for Computing Machinery.
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.
- Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. (2018). Understanding disentangling in beta -vae. arXiv preprint arXiv:1804.03599 . · arXiv
- Caron, M., Touvron, H., Misra, I., J´ egou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660.
- Carter, S., Armstrong, Z., Schubert, L., Johnson, I., and Olah, C. (2019). Activation atlas. Distill , 4(3):e15.
- Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PmLR.
- Clune, J., Stanley, K. O., Pennock, R. T., and Ofria, C. (2011). On the performance of indirect encoding across the continuum of regularity. IEEE Transactions on Evolutionary Computation , 15(3):346-367.
- Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 . · arXiv
- Dawkins, R. (2019). The evolution of evolvability. In Artificial life , pages 201-220. Routledge.
- Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. (2023). Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems , 36:70293-70332.
- Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. (2021). First return, then explore. Nature , 590(7847):580-586.
- Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. (2022). Toy models of superposition. arXiv preprint arXiv:2209.10652 . · arXiv
- Elsken, T., Metzen, J. H., and Hutter, F. (2019). Neural architecture search: A survey. Journal of Machine Learning Research , 20(55):1-21.
- Enquist, M. and Arak, A. (1994). Symmetry, beauty and evolution. Nature , 372(6502):169-172.
- Faldor, M., Zhang, J., Cully, A., and Clune, J. (2025). Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code. International Conference on Learning Representations (ICLR) .
- Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673.
- Grill, J.-B., Strub, F., Altch´ e, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. (2020). Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284.
- Han, S., Pari, J., Gershman, S. J., and Agrawal, P. (2025). General reasoning requires learning to reason from the get-go. arXiv preprint arXiv:2502.19402 . · arXiv
- H¨ ark¨ onen, E., Hertzmann, A., Lehtinen, J., and Paris, S. (2020). Ganspace: Discovering interpretable gan controls. Advances in neural information processing systems , 33:9841-9850.
- He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738.
- He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778.
- Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations .
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 . · arXiv
- Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems , 43(2):1-55.
- Hughes, E., Dennis, M., Parker-Holder, J., Behbahani, F., Mavalankar, A., Shi, Y., Schaul, T., and Rocktaschel, T. (2024). Open-endedness is essential for artificial superhuman intelligence. arXiv preprint arXiv:2406.04268 . · arXiv
- Huh, M., Cheung, B., Wang, T., and Isola, P. (2024). The platonic representation hypothesis. arXiv preprint arXiv:2405.07987 . · arXiv
- Huizinga, J., Stanley, K. O., and Clune, J. (2018). The emergence of canalization and evolvability in an open-ended, interactive evolutionary system. Artificial life , 24(3):157-181.
- Hutter, M. (2000). A theory of universal artificial intelligence based on algorithmic complexity. arXiv preprint cs/0004001 . · arXiv
- Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural computation , 3(1):79-87.
- Jahanian, A., Chai, L., and Isola, P. (2019). On the' steerability' of generative adversarial networks. arXiv preprint arXiv:1907.07171 . · arXiv
- Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. (2024). Mixtral of experts. arXiv preprint arXiv:2401.04088 . jylin04, JackS, Karvonen, A., and Can (2024). Othellogpt learned a bag of heuristics. AI Alignment Forum. · arXiv
- Kambhampati, S. (2024). Can large language models reason and plan? Annals of the New York Academy of Sciences , 1534(1):15-18.
- Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410.
- Kirschner, M. and Gerhart, J. (1998). Evolvability. Proceedings of the National Academy of Sciences , 95(15):8420-8427.
- LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541-551.
- LeCun, Y., Bottou, L., Orr, G. B., and M¨ uller, K.-R. (2002). Efficient backprop. In Neural networks: Tricks of the trade , pages 9-50. Springer.
- Lehman, J., Chen, J., Clune, J., and Stanley, K. O. (2018). Es is more than just a traditional finite-difference approximator. In Proceedings of the genetic and evolutionary computation conference , pages 450-457.
- Lehman, J., Meyerson, E., El-Gaaly, T., Stanley, K. O., and Ziyaee, T. (2025). Evolution and the knightian blindspot of machine learning. arXiv preprint arXiv:2501.13075 . · arXiv
- Lehman, J. and Stanley, K. O. (2010). Efficiently evolving programs through the search for novelty. In Proceedings of the 12th annual conference on Genetic and evolutionary computation , pages 837-844.
- Lehman, J. and Stanley, K. O. (2011). Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation , 19(2):189-223.
- Li, K., Hopkins, A. K., Bau, D., Vi´ egas, F., Pfister, H., and Wattenberg, M. (2023). Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR .
- Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. (2024). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 . · arXiv
- Liu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark, M., and Williams, M. (2022a). Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems , 35:34651-34663.
- Liu, Z., Michaud, E. J., and Tegmark, M. (2022b). Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117 . · arXiv
- Maley, C. C. (1999). Four steps toward open-ended evolution. In Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation-Volume 2 , pages 1336-1343.
- McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. (2023). Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638 . · arXiv
- Miconi, T. (2008). The road to everywhere: Evolution, complexity and progress in natural and artificial systems . PhD thesis, University of Birmingham.
- Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. (2024). Gsmsymbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 . · arXiv
- Mouret, J.-B. and Clune, J. (2015). Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 . · arXiv
- Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 . · arXiv
- Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems , 29.
- Nguyen, A. M., Yosinski, J., and Clune, J. (2015). Innovation engines: Automated creativity and improved stochastic optimization via deep learning. In Proceedings of the 2015 annual conference on genetic and evolutionary computation , pages 959-966.
- Nikankin, Y., Reusch, A., Mueller, A., and Belinkov, Y. (2024). Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprint arXiv:2410.21272 . · arXiv
- Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020). Zoom in: An introduction to circuits. Distill , 5(3):e00024-001.
- Olah, C., Mordvintsev, A., and Schubert, L. (2017). Feature visualization. Distill , 2(11):e7.
- Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. (2022). In-context learning and induction heads. arXiv preprint arXiv:2209.11895 . · arXiv
- Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 . · arXiv
- Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PmLR.
- Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 . · arXiv
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Rissanen, J. (1978). Modeling by shortest data description. Automatica , 14(5):465-471.
- Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695.
- Schmidhuber, J. (1997). Low-complexity art. Leonardo , 30(2):97-103.
- Secretan, J., Beato, N., D Ambrosio, D. B., Rodriguez, A., Campbell, A., and Stanley, K. O. (2008). Picbreeder: evolving pictures collaboratively online. In Proceedings of the SIGCHI conference on human factors in computing systems , pages 1759-1768.
- Secretan, J., Beato, N., D'Ambrosio, D. B., Rodriguez, A., Campbell, A., Folsom-Kovarik, J. T., and Stanley, K. O. (2011). Picbreeder: A case study in collaborative evolutionary exploration of design space. Evolutionary computation , 19(3):373-403.
- Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025). Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496 . · arXiv
- Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 . · arXiv
- Solomonoff, R. J. (1964). A formal theory of inductive inference. part i. Information and control , 7(1):1-22.
- Soviany, P., Ionescu, R. T., Rota, P., and Sebe, N. (2022). Curriculum learning: A survey. International Journal of Computer Vision , 130(6):1526-1565.
- Standish, R. K. (2003). Open-ended artificial evolution. International Journal of Computational Intelligence and Applications , 3(02):167-175.
- Stanley, K. O. and Miikkulainen, R. (2002). Evolving neural networks through augmenting topologies. Evolutionary computation , 10(2):99-127.
- Stanley, K. O. and Miikkulainen, R. (2003). A taxonomy for artificial embryogeny. Artificial life , 9(2):93-130.
- Taylor, T. (2019). Evolutionary innovations and where to find them: Routes to open-ended evolution in natural and artificial systems. Artificial life , 25(2):207-224.
- Taylor, T., Bedau, M., Channon, A., Ackley, D., Banzhaf, W., Beslon, G., Dolson, E., Froese, T., Hickinbotham, S., Ikegami, T., McMullin, B., Packard, N., Rasmussen, S., Virgo, N., Agmon, E., Clark, E., McGregor, S., Ofria, C., Ropella, G., Spector, L., Stanley, K. O., Stanton, A., Timperley, C., Vostinar, A., and Wiser, M. (2016). Open-ended evolution: Perspectives from the oee workshop in york. Artificial Life , 22(3):408-423.
- Team, A. A., Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023). Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608 . · arXiv
- Vafa, K., Chen, J., Rambachan, A., Kleinberg, J., and Mullainathan, S. (2024). Evaluating the world model implicit in a generative model. Advances in Neural Information Processing Systems , 37:26941-26975.
- Varma, V., Shah, R., Kenton, Z., Kram´ ar, J., and Kumar, R. (2023). Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390 . · arXiv
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, glyph[suppress] L., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30.
- Wagner, G. P. and Altenberg, L. (1996). Perspective: complex adaptations and the evolution of evolvability. Evolution , 50(3):967-976.
- Wang, R., Lehman, J., Clune, J., and Stanley, K. O. (2019). Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753 . · arXiv
- Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., and Stanley, K. (2020). Enhanced poet: Openended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning , pages 9940-9951. PMLR.
- Wolpert, L., Tickle, C., and Arias, A. M. (2015). Principles of development . Oxford University Press, USA.
- Woolley, B. G. and Stanley, K. O. (2011). On the deleterious effects of a priori objectives on evolution and representation. In Proceedings of the 13th annual conference on Genetic and evolutionary computation , pages 957-964.
- Wu, Z., Qiu, L., Ross, A., Aky¨ urek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. (2024). Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 1819-1862.
- Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H. (2015). Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 . · arXiv
- Zhou, L., Schellaert, W., Mart´ ınez-Plumed, F., Moros-Daval, Y., Ferri, C., and Hern´ andez-Orallo, J. (2024). Larger and more instructable language models become less reliable. Nature , 634(8032):61-68.
Cited by
Sorted by citations
Technological folie à deux: feedback loops between AI chatbots and mental health
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00595-8 · Cited by 51 · see citing works
Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2025) · arXiv · doi:10.48550/arxiv.2507.19218 · Cited by 26 · see citing works
“The complete nature of the input-output functional mappings they learn is not accessible to a human user, despite initial appearances (a fact that can be intuitively grasped by recent work on AI representation learning 60 ).”
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Koepke, A. S., Zverev, D., Ginosar, S., & Efros, A. A. (2026) · arXiv · doi:10.48550/arxiv.2604.18572
“Beyond similarity metrics, Smith et al. [86] and Kumar et al. [52] show that functional agreement and output behavior can persist even when internal representations are misaligned or entangled, suggesting that behavioral compatibility does not imply shared structure.”
- Authors
- Kumar, A., Clune, J., Lehman, J., & Stanley, K. O.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2505.11581
- OpenAlex
- W4417299644
- Cited by
- 2
- References
- 1
- Citations by year
- 2026: 1 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- neuroevolution, stochastic gradient descent, image generation, factored representation, representation learning, generalization, continual learning, model capacity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review