The neural architecture of language: Integrative modeling converges on predictive processing
Open accessPublished version of a preprint
Abstract
Significance: Language is a quintessentially human ability. Research has long probed the functional architecture of language in the mind and brain using diverse neuroimaging, behavioral, and computational modeling approaches. However, adequate neurally-mechanistic accounts of how meaning might be extracted from language are sorely lacking. Here, we report a first step toward addressing this gap by connecting recent artificial neural networks from machine learning to human recordings during language processing. We find that the most powerful models predict neural and behavioral responses across different datasets up to noise levels. Models that perform better at predicting the next word in a sequence also better predict brain measurements—providing computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the brain.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies have increasingly explored LLM representations as analogs to neuroimaging data, aligning and comparing the human brain with the LLM brain during natural language processing.
“Recent studies have increasingly explored LLM representations as analogs to neuroimaging data, aligning and comparing the human brain with the LLM”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Schrimpf et al., 2021
The intermediate-to-deep layers (l = 2/3 n_layers) of each LLM merit particular attention due to their demonstrated superior predictive capabilities for human brain activity.
“Additionally, we focused on the intermediate-to-deep layers ( 𝑙 = 2 3 𝑛 𝑙𝑎𝑦𝑒𝑟𝑠 ) of each LLM, as suggested by Caucheteux et al. (2023). These layers merit particular attention due to their demonstrated superior predictive capabilities for human brain activity (Schrimpf et al., 2021; Caucheteux and King, 2022; Caucheteux et al., 2023).”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Schrimpf et al., 2021
The finding that Item 1 corresponds approximately to the 2/3 layer is consistent with previous studies suggesting that the 2/3 layer is optimal for predicting human brain activity.
“This finding is consistent with previous studies, which suggest that the 2 3 layer is optimal for predicting human brain activity (Schrimpf et al., 2021; Caucheteux and King, 2022; Caucheteux et al., 2023).”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Schrimpf et al., 2021
LLM representations are being increasingly studied as analogs of neuroimaging data when comparing and aligning human brain activity with LLM 'brain' activity in natural language processing.
“While LLM representations are being increasingly studied as analogs of neuroimaging data when comparing and aligning human brain activity with LLM”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Schrimpf et al., 2021
The 2/3 layer has been reported to exhibit strong alignment with human brain activity.
“as this layer has been reported to exhibit strong alignment with human brain activity (Schrimpf et al., 2021; Caucheteux and King, 2022; Caucheteux et al., 2023).”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Schrimpf et al., 2021
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Distributed hierarchical processing in the primate cerebral cortex
Felleman, D. J., & Van Essen, D. C. (1991) · Cerebral Cortex · doi:10.1093/cercor/1.1.1 · Cited by 7,783 · see citing works
Whatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924 · see citing works
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · see citing works
Canonical Microcircuits for Predictive Coding
Bastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R., Fries, P., & Friston, K. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.038 · Cited by 2,421 · see citing works
Learning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · see citing works
How to grow a mind: Statistics, structure, and abstraction
Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (2011) · Science · doi:10.1126/science.1192788 · Cited by 1,815 · see citing works
Self-citationThe anatomy of language: a review of 100 fMRI studies published in 2009
Price, C. J. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05444.x · Cited by 1,387 · see citing works
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080 · see citing works
Predictive coding: a fresh view of inhibition in the retina
Srinivasan, M. V., Laughlin, S. B., & Dubs, A. (1982) · Proceedings of the Royal Society of London. Series B. Biological Sciences · doi:10.1098/rspb.1982.0085 · Cited by 1,042 · see citing works
On the computational architecture of the neocortex: I. The role of the thalamo-cortical loop
Mumford, D. (1991) · Biological Cybernetics · doi:10.1007/bf00202389 · Cited by 929 · see citing works
Deep Learning: A Critical Appraisal
Marcus, G. (2018) · arXiv · doi:10.48550/arxiv.1801.00631 · Cited by 853 · see citing works
Predictive Processing: A Canonical Cortical Computation
Keller, G. B., & Mrsic-Flogel, T. D. (2018) · Neuron · doi:10.1016/j.neuron.2018.10.003 · Cited by 836 · see citing works
The cortical language circuit: from auditory perception to sentence comprehension
Friederici, A. D. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.04.001 · Cited by 748 · see citing works
Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016) · Scientific Reports · doi:10.1038/srep27755 · Cited by 716 · see citing works
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544 · see citing works
Rational integration of noisy evidence and prior semantic expectations in sentence interpretation
Gibson, E., Bergen, L., & Piantadosi, S. T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1216438110 · Cited by 407 · see citing works
Recurrence is required to capture the representational dynamics of the human visual system
Kietzmann, T. C., Spoerer, C. J., Sörensen, L. K. A., Cichy, R. M., Hauk, O., & Kriegeskorte, N. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905544116 · Cited by 406 · see citing works
Neurophysiological dynamics of phrase-structure building during sentence processing
Nelson, M. J., El Karoui, I., Giber, K., Yang, X., Cohen, L., Koopman, H., Cash, S. S., Naccache, L., Hale, J. T., Pallier, C., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1701590114 · Cited by 319 · see citing works
Prediction during natural language comprehension
Willems, R. M., Frank, S. L., Nijhof, A. D., Hagoort, P., & van den Bosch, A. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhv075 · Cited by 288 · see citing works
Ponce, C. R., Xiao, W., Schade, P. F., Hartmann, T. S., Kreiman, G., & Livingstone, M. S. (2019) · Cell · doi:10.1016/j.cell.2019.04.005 · Cited by 245 · see citing works
Integrative benchmarking to advance neurally mechanistic models of human intelligence
Schrimpf, M., Kubilius, J., Lee, M. J., Ratan Murty, N. A., Ajemian, R., & DiCarlo, J. J. (2020) · Neuron · doi:10.1016/j.neuron.2020.07.040 · Cited by 233 · see citing works
Self-citationfMRI reveals language-specific predictive coding during naturalistic sentence comprehension
Shain, C., Blank, I. A., van Schijndel, M., Schuler, W., & Fedorenko, E. (2020) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2019.107307 · Cited by 215 · see citing works
Self-citationBlank, I., Kanwisher, N., & Fedorenko, E. (2014) · Journal of Neurophysiology · doi:10.1152/jn.00884.2013 · Cited by 181 · see citing works
Self-citationThought Beyond Language: Neural Dissociation of Algebra and Natural Language
Monti, M. M., Parsons, L. M., & Osherson, D. N. (2012) · Psychological Science · doi:10.1177/0956797612437427 · Cited by 171 · see citing works
Incorporating Context into Language Encoding Models for fMRI
Jain, S., & Huth, A. G. (2018) · doi:10.1101/327601 · Cited by 155 · see citing works
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.158 · Cited by 134 · see citing works
Aligning context-based statistical models of language with brain activity during reading
Wehbe, L., Vaswani, A., Knight, K., & Mitchell, T. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1030 · Cited by 95 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Over-reliance on English hinders cognitive science
Blasi, D. E., Henrich, J., Adamou, E., Kemmerer, D., & Majid, A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.09.015 · Cited by 396 · see citing works
Semantic reconstruction of continuous language from non-invasive brain recordings
Tang, J., LeBel, A., Jain, S., & Huth, A. G. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01304-9 · Cited by 442 · see citing works
A hierarchy of linguistic predictions during natural language comprehension
Heilbron, M., Armeni, K., Schoffelen, J.-M., Hagoort, P., & de Lange, F. P. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2201968119 · Cited by 343 · see citing works
Brains and algorithms partially converge in natural language processing
Caucheteux, C., & King, J.-R. (2022) · Communications Biology · doi:10.1038/s42003-022-03036-1 · Cited by 318 · see citing works
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
Self-citationEvidence of a predictive coding hierarchy in the human brain listening to speech
Caucheteux, C., Gramfort, A., & King, J.-R. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01516-2 · Cited by 308 · see citing works
The neuroconnectionist research programme
Doerig, A., Sommers, R. P., Seeliger, K., Richards, B., Ismael, J., Lindsay, G. W., Kording, K. P., Konkle, T., van Gerven, M. A. J., Kriegeskorte, N., et al. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00705-w · Cited by 233 · see citing works
Can language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142 · see citing works
A foundation model to predict and capture human cognition
Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M. K., Éltető, N., et al. (2025) · Nature · doi:10.1038/s41586-025-09215-4 · Cited by 119 · see citing works
Oh, B.-D., & Schuler, W. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00548 · Cited by 107 · see citing works
Deep language algorithms predict semantic comprehension from brain activity
Caucheteux, C., Gramfort, A., & King, J.-R. (2022) · Scientific Reports · doi:10.1038/s41598-022-20460-9 · Cited by 91 · see citing works
Do Large Language Models Know What Humans Know?
Trott, S., Jones, C., Chang, T., Michaelov, J., & Bergen, B. (2023) · Cognitive Science · doi:10.1111/cogs.13309 · Cited by 99 · see citing works
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322420121 · Cited by 96 · see citing works
Symbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95 · see citing works
Does ChatGPT have semantic understanding? A problem with the statistics-of-occurrence strategy
Titus, L. M. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101174 · Cited by 82 · see citing works
How deep is the brain? The shallow brain hypothesis
Suzuki, M., Pennartz, C. M. A., & Aru, J. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00756-z · Cited by 79 · see citing works
Hu, J., Small, H., Kean, H., Takahashi, A., Zekelman, L., Kleinman, D., Ryan, E., Nieto-Castañón, A., Ferreira, V., & Fedorenko, E. (2023) · Cerebral Cortex · doi:10.1093/cercor/bhac350 · Cited by 73 · see citing works
Self-citationGoldstein, A., Grinstein-Dabush, A., Schain, M., Wang, H., Hong, Z., Aubrey, B., Nastase, S. A., Zada, Z., Ham, E., Feder, A., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-46631-y · Cited by 64 · see citing works
Hosseini, E. A., Schrimpf, M., Zhang, Y., Bowman, S., Zaslavsky, N., & Fedorenko, E. (2024) · Neurobiology of Language · doi:10.1162/nol_a_00137 · Cited by 63 · see citing works
Self-citationMultiple realizability and the spirit of functionalism
Cao, R. (2022) · Synthese · doi:10.1007/s11229-022-03524-1 · Cited by 58 · see citing works
Lampinen, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00525 · Cited by 32 · see citing works
Tikochinski, R., Goldstein, A., Yeshurun, Y., Hasson, U., & Reichart, R. (2023) · Cerebral Cortex · doi:10.1093/cercor/bhad082 · Cited by 14 · see citing works
Universality of representation in biological and artificial neural networks
Hosseini, E., Casto, C., Zaslavsky, N., Conwell, C., Richardson, M., & Fedorenko, E. (2024) · doi:10.1101/2024.12.26.629294 · Cited by 10 · see citing works
Self-citationJudgments of learning distinguish humans from large language models in predicting memory
Huff, M., & Ulakci, E. (2025) · Scientific Reports · doi:10.1038/s41598-025-22290-x · Cited by 9 · see citing works
Do Vision and Language Models Share Concepts? A Vector Space Alignment Study
Li, J., Kementchedjhieva, Y., Fierro, C., & Søgaard, A. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00698 · Cited by 6 · see citing works
State and capacity in neural models of cognition and consciousness
Wang, P., Liu, H., Ilievski, F., Ganushchak, L., Zou, L., Laureys, S., & Paas, F. (2026) · Trends Open · doi:10.1016/j.treopn.2026.04.004 · Cited by 8 · see citing works
Li, J. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100163 · Cited by 3 · see citing works
- Authors
- Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E.
- Affiliations
- McGovern Institute for Brain Research (US)
- University of California, Los Angeles (US)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Fedorenko, Evelina
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2021-11-04
- Volume
- 118
- Issue
- 45
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC8694052
- OpenAlex
- W3210923133
- Cited by
- 517
- References
- 30
- Citations by year
- 2026: 99 · 2025: 120 · 2024: 103 · 2023: 134 · 2022: 55 · 2021: 5 · 2019: 1
- FWCI
- 39.04
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- Takeda Pharmaceuticals U.S.A. (-)
- Massachusetts Institute of Technology (Shoemaker fellowship, Media Lab Consortia, Singleton fellowship, Presidential Graduate fellowship, Friends of the McGovern Institute Institute)
- Semiconductor Research Corporation (C-BRIC)
- National Science Foundation (NSF STC CCF-1231216)
- HHS | National Institutes of Health (R01-DC016607, R01-DC016950)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Keywords
- predictive processing, language comprehension, neural architecture, transformer models, integrative modeling, next-word prediction, functional MRI, electrocorticography, model architecture
- MeSH terms
- Brain, Humans, Language, Models, Neurological, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
