Incorporating Context into Language Encoding Models for fMRI
Open accessAbstract
Language encoding models help explain language processing in the human brain by learning functions that predict brain responses from the language stimuli that elicited them. Current word embedding-based approaches treat each stimulus word independently and thus ignore the influence of context on language understanding. In this work, we instead build encoding models using rich contextual representations derived from an LSTM language model. Our models show a significant improvement in encoding performance relative to state-of-the-art embeddings in nearly every brain area. By varying the amount of context used in the models and providing the models with distorted context, we show that this improvement is due to a combination of better word embeddings learned by the LSTM language model and contextual information. We are also able to use our models to map context sensitivity across the cortex. These results suggest that LSTM language models learn high-level representations that are related to representations in the human brain.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 19 cited sources checked · 0 supported · 0 flagged
- Findingown data§ 5.1 Language encoding performance for context representations · p. 5
Model performance plateaus after 10-15 words of context.
“However, model performance plateaus after 10-15 words, suggesting that the LSTM LM was unable to successfully incorporate contextual information beyond this timescale.”
Quote in the paperSubstantial claim - Findingown data§ 5.3 Model preference across cortex · p. 9
Layer 2 provides the best predictions by a small margin.
“Still, layer 2 provides the best predictions by a small margin, giving many voxels a green tint.”
Quote in the paperSubstantial claim - Findingown data§ 5.2 Distorting context representations · p. 7
The impact of swapped context on encoding performance increases progressively across LSTM layers.
“Additionally, we observe the same patterns of progressively greater impact across layers as in the random context models.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 Language encoding performance for context representations · p. 5
Model performance increases monotonically with context length for all layers.
“Figure 2(A) shows that model performance increases monotonically with context length, for all layers.”
- Also stated in: § 5 Results · p. 5 · plain
“Increasing context length uniformly improves performance in every layer”
Quote in the paperSubstantial claim - Also stated in: § 5 Results · p. 5 · plain
- Findingown data§ 5 Results · p. 5
Swapped-context models perform worse than the original contextual models.
“Swapped-context models also perform worse than the original, showing that linguistically plausible but wrong context also hurts the model.”
- Also stated in: § 5.2 Distorting context representations · p. 7 · plain
“The results in Figure 2(C) show that linguistically plausible but wrong contexts also greatly impact encoding performance.”
Quote in the paperSubstantial claim - Also stated in: § 5.2 Distorting context representations · p. 7 · plain
- Findingown data§ Abstract · p. 1
Contextual encoding models using LSTM language model representations perform significantly better than previously published word embedding models in nearly every brain area.
“Our models show a significant improvement in encoding performance relative to state-of-the-art embeddings in nearly every brain area.”
- Also stated in: § 5.1 Language encoding performance for context representations · p. 5 · plain
“Contextual models significantly beat the embedding in nearly every model variation.”
Quote in the paperSubstantial claim- Rests on an assumption: No existing hand-designed feature space is well-suited for extracting language features that incorporate context.
- Rests on an assumption: The Reddit corpus is more similar to the stimulus stories than most conventional corpora because of its informal and conversational nature.
- Also stated in: § 5.1 Language encoding performance for context representations · p. 5 · plain
- Findingown data§ 5.3 Model preference across cortex · p. 9
Auditory cortex shows a clear preference for layers 1 and 3.
“However, auditory cortex (AC) shows a clear preference for layers 1 and 3, giving it a purple tint.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 Language encoding performance for context representations · p. 6
Layer 1 with zero context is substantially better than the embedding.
“The model for layer 1 with zero context is substantially better than the embedding, suggesting that layer 1 learns to generate a new embedding that is a better match to the brain.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 Language encoding performance for context representations · p. 6
The effect of context length is different for each LSTM layer.
“Further, the effect of context length is different for each layer.”
Quote in the paperNot a substantial claim - Findingown data§ Supplement for Incorporating Context into Language Encoding Models for fMRI · p. 11
The contextualized encoding model can capture known timescale variance across the cortex.
“Additionally, the contextualized encoding model can capture known timescale variance across the cortex.”
Quote in the paperSubstantial claim - Findingown data§ Abstract · p. 1
The improvement of contextual models over embeddings is due to a combination of better word embeddings learned by the LSTM language model and contextual information.
“By varying the amount of context used in the models and providing the models with distorted context, we show that this improvement is due to a combination of better word embeddings learned by the LSTM language model and contextual information.”
- Also stated in: § 5.1 Language encoding performance for context representations · p. 6 · hedged
“The encoding model results in Figure 2(A) also suggest that the improvement of our model can be partially explained by the LSTM LM simply learning a better word embedding than the baseline.”
Quote in the paperSubstantial claim- Rests on an assumption: The Reddit corpus is more similar to the stimulus stories than most conventional corpora because of its informal and conversational nature.
- Also stated in: § 5.1 Language encoding performance for context representations · p. 6 · hedged
- Findingown data§ Abstract · p. 1
Contextual models can be used to map context sensitivity across the cortex.
“We are also able to use our models to map context sensitivity across the cortex.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 Language encoding performance for context representations · p. 5
LSTM language model performance on next-word prediction mirrors encoding model performance.
“This result is mirrored by the performance of the LSTM LM on the language modeling task, shown in Figure 2(C).”
Quote in the paperSubstantial claim - Findingown data§ 5.1 Language encoding performance for context representations · p. 6
The best model for each layer uses the longest context.
“However, the best model for each layer uses the longest context, suggesting that contextual information is important above and beyond the improved embedding.”
Quote in the paperSubstantial claim - Findingown data§ 4 Learning representations of context · p. 4
Fine-tuning the LSTM language model on part of the stimulus stories and evaluating on the rest yielded no benefit.
“Due to the limited amount of stories and their high diversity, we observe no benefit of fine-tuning the LM on a part of S and evaluating on the rest.”
Quote in the paperNot a substantial claim - Interpretationown data§ 5.1 Language encoding performance for context representations · p. 5
Better language models yield better encoding models.
“This suggests that better LMs yield better encoding models.”
Quote in the paperSubstantial claim - Premisecitation§ 1 Introduction · p. 2
Different brain areas have different temporal receptive fields indicating the amount of context they are sensitive to.
“Firstly, different brain areas are known to have different 'temporal receptive fields' that indicate the amount of context they are sensitive to (12).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [12] source
Not checked in the free run.
- Interpretationown data§ 5.1 Language encoding performance for context representations · p. 5
The ability of a language model to generate useful encoding representations is tied to its language modeling ability.
“The similarity between LM and encoding performance suggests that the ability of an LM to generate useful encoding representations is tied to its language modeling ability.”
Quote in the paperSubstantial claim - Premisereasoning§ 1 Introduction · p. 1
Contextual information must be represented in the human cortex.
“Contextual information must thus be represented in the human cortex.”
Quote in the paperSubstantial claim - Premisecitation§ 1 Introduction · p. 2
Different layers from supervised networks predicted different brain areas.
“However, previous fMRI studies have found that different layers from supervised networks predicted different brain areas (2; 7; 5; 11), suggesting that each layer should be tested separately.”
Quote in the paperSubstantial claim0 of 4 sources checked- cites [2] Agrawal, P. , Stansbury, D. , Malik, J. , Gallant, J.L. : Pixels to voxels: Modeling visual representation in the human brain. CoRR abs/1407.5104 (2014)
Not checked in the free run.
- cites [7] source
Not checked in the free run.
- cites [5] source
Not checked in the free run.
- cites [11] source
Not checked in the free run.
- Interpretationown data§ 5.3 Model preference across cortex · p. 8
Auditory cortex, Broca's area, and left temporo-parietal junction selectively represent information about the current word rather than information built up with context.
“These areas thus seem to selectively represent information about the current word rather than information that is built up with context.”
Quote in the paperSubstantial claim - Interpretationown data§ 6 Conclusions · p. 9
The models explain differences in language processing across cortical regions, from low-level to high-level language areas.
“Finally, we show how our models explain the differences in language processing across different cortical regions, from low-level to high-level language areas.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.3 Model preference across cortex · p. 9
The highest-level representations in self-supervised models might emerge at the layer farthest from the input.
“Our results suggest that the highest-level representations in self-supervised models might emerge at the layer which is farthest from the input.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.1 Language encoding performance for context representations · p. 5
The LSTM language model was unable to successfully incorporate contextual information beyond a timescale of about 10-15 words.
“However, model performance plateaus after 10-15 words, suggesting that the LSTM LM was unable to successfully incorporate contextual information beyond this timescale.”
Quote in the paperSubstantial claim - Interpretationown data§ 6 Conclusions · p. 9
The contextual models incorporate context and temporal order, albeit differently across layers.
“Our findings suggest that these models do indeed incorporate context and temporal order, albeit differently across layers.”
Quote in the paperSubstantial claim - Interpretationown data§ Abstract · p. 1
LSTM language models learn high-level representations related to representations in the human brain.
“These results suggest that LSTM language models learn high-level representations that are related to representations in the human brain.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.2 Distorting context representations · p. 7
The context model is picking up information relevant for cortical representations.
“This demonstrates even more effectively than before that the context model is picking up information relevant for cortical representations.”
Quote in the paperSubstantial claim - Premisecitation§ 1 Introduction · p. 2
LSTM language models have been shown to discover high-level semantic and contextual representations useful for many natural language processing tasks.
“LSTM LMs have been shown to discover high-level semantic and contextual representations that are useful for many natural language processing tasks (18; 13; 19).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [18] Peters, M.E. , Ammar, W. , Bhagavatula, C. , Power, R. : Semi-supervised sequence tagging with bidirectional language models. In: ACL (1). pp. 1756–1765. Association for Computational Linguistics (2017) source
Not checked in the free run.
- cites [13] McCann, B. , Bradbury, J. , Xiong, C. , Socher, R. : Learned in translation: Contextualized word vectors. CoRR abs/1708.00107 (2017)
Not checked in the free run.
- cites [19] Peters, M.E. , Neumann, M. , Iyyer, M. , Gardner, M. , Clark, C. , Lee, K. , Zettlemoyer, L. : Deep contextualized word representations. CoRR abs/1802.05365 (2018), http://arxiv.org/abs/1802.05365
Not checked in the free run.
- May combine two claims. 15
- Classified as probably not a substantial claim. 4
May combine two claims.
Findingown data§ 5 Results · p. 5Random-context models perform worse than the original contextual models, and most perform worse than the embedding except in layer 1.
“Random-context models perform worse than the original models, and most perform worse than the embedding, except in layer 1.”
- Also stated in: § 5.2 Distorting context representations · p. 7 · boosted
“It is obvious that the performance greatly suffers from the context distortion.”
Quote in the paperSubstantial claim- Also stated in: § 5.2 Distorting context representations · p. 7 · boosted
May combine two claims.
Findingown data§ Supplement for Incorporating Context into Language Encoding Models for fMRI · p. 11The observed patterns in the supplementary material are consistent across subjects, proving that contextual representations from an LSTM language model are effective for language encoding.
“The observed patterns are consistent across subjects, proving that the contextual representations extracted from an LSTM LM are effective for language encoding.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.1 Language encoding performance for context representations · p. 6Layer 2 predicts roughly the same as baseline with zero context, jumps sharply with one word of context, and is the best model thereafter.
“Layer 2 predicts roughly the same as baseline with zero context, but jumps sharply with even one word of context and is the best model thereafter.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.3 Model preference across cortex · p. 9Overall performance of all LSTM layers is highly similar, rendering most voxels nearly white.
“Overall the performance of all layers is highly similar, rendering most voxels nearly white.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.1 Language encoding performance for context representations · p. 6Layer 3 predicts worse than the embedding with short context, but sees the largest improvement with context length.
“Layer 3 predicts worse than the embedding with short context, but sees the largest improvement with context length.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 5.3 Model preference across cortex · p. 8Most voxels prefer long context, but clusters of short-context-preferring voxels appear in auditory cortex, Broca's area, and left temporo-parietal junction.
“Although most voxels prefer long context, clusters of short-context-preferring voxels appear in auditory cortex (AC), Broca's area, and left temporo-parietal junction (TPJ).”
Quote in the paperSubstantial claimMay combine two claims.
Findingreasoning§ 5.2 Distorting context representations · p. 7The impact of random context distortion on encoding performance increases progressively across LSTM layers, with layer 1 least affected and still beating baseline.
“Further, the impact on performance progressively increases across LSTM layers with layer 1 being affected the least, and still beating the baseline.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.1 Language encoding performance for context representations · p. 6Layer 1 predicts substantially better than baseline even with short context, but improves only modestly with context length.
“Layer 1 predicts substantially better than baseline even with short context, but only improves modestly with context length.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingreasoning§ 5.2 Distorting context representations · p. 7The swapped contexts chosen for each successive context length are sometimes different, leading to increased variance across context lengths.
“Thus, the swapped contexts picked for each successive length are sometimes different, leading to the increased variance across context lengths.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ 5.3 Model preference across cortex · p. 9The finding that the middle LSTM layer learns highest-level representations contrasts with supervised networks where high-level cortical areas were best modeled by higher layers.
“This finding is in stark contrast to experiments that used supervised networks (7; 5; 11).”
Quote in the paperSubstantial claim0 of 3 sources checkedMay combine two claims.
Premisecitation§ Abstract · p. 1Current word embedding-based language encoding approaches treat each stimulus word independently and ignore context.
“Current word embedding-based approaches treat each stimulus word independently and thus ignore the influence of context on language understanding.”
- Also stated in: § 1 Introduction · p. 1 · plain
“Previous language encoding studies have successfully mapped word-level semantic representations using embedding vectors (15; 22; 9; 8), but this approach neglects the effect of context by assuming that the response to each word is independent.”
Quote in the paperSubstantial claim0 of 4 sources checked- cites [15] source
Not checked in the free run.
- cites [22] source
Not checked in the free run.
- cites [9] source
Not checked in the free run.
- cites [8] de Heer, W.A. , Huth, A.G. , Griffiths, T.L. , Gallant, J.L. , Theunissen, F.E. : The hierarchical cortical organization of human speech processing. Journal of Neuroscience (2017), http://www.jneurosci.org/content/early/2017/06/06/JNEUROSCI.3267-16.2017
Not checked in the free run.
- Also stated in: § 1 Introduction · p. 1 · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data§ 5.3 Model preference across cortex · p. 8The context length preference analysis was repeated for other LSTM layers and yielded similar results.
“Layer 2 is used because it has the best performance overall; this analysis was repeated for the other layers and yielded similar results.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ 5.2 Distorting context representations · p. 7The LSTM successfully incorporates context, although there are differences across layers.
“This demonstrates that the LSTM successfully incorporates context, although there are differences across layers.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Premisecitation§ 1 Introduction · p. 2A previous study suggests that representations learned by a language model can be useful for modeling MEG brain responses to reading.
“Further, a previous study suggests that representations learned by an LM can be useful for modeling MEG brain responses to reading (23).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [23] Wehbe, L. , Vaswani, A. , Knight, K. , Mitchell, T.M. : Aligning context-based statistical models of language with brain activity during reading. In: EMNLP (2014) source
Not checked in the free run.
May combine two claims.
Interpretationown data§ 5.3 Model preference across cortex · p. 8The contextual encoding model captures timescale variance across cortex and explicitly models the transformation from stimulus to response.
“Our contextual encoding model captures this timescale variance, but goes further by explicitly modeling the transformation from stimulus to response.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 5.3 Model preference across cortex · p. 9Low-level speech processing in auditory cortex is better modeled by layers 1 and 3, while high-level processing is better modeled by layer 2.
“Thus it seems that low-level speech processing (AC) is better modeled by layers 1 and 3, while high-level processing is better modeled by layer 2.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 5.3 Model preference across cortex · p. 9The middle layer of the LSTM language model learns the highest-level representations, while layers 1 and 3, closer to word embeddings, learn lower-level representations.
“This suggests that the middle layer of the LSTM LM is learning the highest-level representations”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 5.2 Distorting context representations · p. 7Representations beyond layer 1 do not directly use the embedding and increasingly rely on contextual information passed through recurrence.
“Since the representations beyond layer 1 do not directly make use of the embedding and increasingly rely on contextual information passed through recurrence, the progressive impact across layers is justified.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Premisecitation§ 5.3 Model preference across cortex · p. 9Supervised network experiments found that high-level cortical areas were best modeled by higher layers of the network.
“Those experiments found that high-level cortical areas were best modeled by higher layers of the network.”
Quote in the paperSubstantial claim0 of 3 sources checked
The paper relies on this without support.
No existing hand-designed feature space is well-suited for extracting language features that incorporate context.
“However, no existing hand-designed feature space is well-suited for this task.”
The paper relies on this without support.
The Reddit corpus is more similar to the stimulus stories than most conventional corpora because of its informal and conversational nature.
“We chose this particular corpus because the informal and conversational nature of the text is more similar to our stimulus stories than most conventional corpora.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have shown success in predicting neural signals associated with narrative processing.
“Large Language Models (LLMs) have shown success in predicting neural signals associated with narrative processing”
From Incremental accumulation of linguistic context in artificial and biological neural networks (2025) · cites this paper as [8]
Previous studies in the neural encoding literature all used LLMs that were only aware of the short-term context, with context window size as large as several tens of words.
“Previous studies in the neural encoding literature all used LLMs that were only aware of the short-term context, i.e., where the context window size was as large as several tens of words1–10.”
From Incremental accumulation of linguistic context in artificial and biological neural networks (2025) · cites this paper as [8]
In previous neural encoding studies, participants were exposed to language stimuli that encompassed hundreds and even thousands of words.
“This was the case even though the participants in these studies were exposed to language stimuli that encompassed hundreds and even thousands of words.”
From Incremental accumulation of linguistic context in artificial and biological neural networks (2025) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Maximum extracted context is 19 previous words, not 20
MinorThe stated extraction procedure allows context lengths from 0 through M−1. With M = 20, the representations therefore use at most 19 previous words, contrary to the claim of up to 20.
“Our LSTM LM uses information from up to M = 20 previous words to construct L = 3 separate context representations.”
“To obtain context representations for a stimulus story S = [ w 1 , . . . , w n ] with a desired context length m ∈ [0 ..M -1] and layer l ∈ [1 ..L ] , we use the following procedure. First, for each word w i we extract the previous m words to form a length m +1 sequence [ w i -m , . . . , w i ] .”
Language-modeling results are assigned to the wrong Figure 2 panel
MinorThe text directs readers to panel C for language-modeling performance, but the caption identifies panel C as the swapped-context experiment and panel D as language-modeling performance.
“This result is mirrored by the performance of the LSTM LM on the language modeling task, shown in Figure 2(C).”
“( c ) Further, we swap context between different occurrences of each word and re-fit the encoding models. Swapped-context models also perform worse than the original, showing that linguistically plausible but wrong context also hurts the model. ( d ) Performance of the LSTM network on the language modeling task, as a function of context length.”
Cites
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,369 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., & Janvin, C. (2003) · Journal of machine learning research · doi:10.1162/153244303322533223 · Cited by 8,257 · see citing works
An overview of gradient descent optimization algorithms
Ruder, S. (2016) · arXiv · doi:10.48550/arxiv.1609.04747 · Cited by 4,962 · see citing works
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · see citing works
Natural speech reveals the semantic maps that tile human cerebral cortex
Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016) · Nature · doi:10.1038/nature17637 · Cited by 1,504 · see citing works
Self-citationA Selected Core Microbiome Drives the Early Stages of Three Popular Italian Cheese Manufactures
De Filippis, F., La Storia, A., Stellato, G., Gatti, M., & Ercolini, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089680 · Cited by 1,398 · see citing works
Identifying natural images from human brain activity
Kay, K., Naselaris, T., Prenger, R., & Gallant, J. L. (2008) · Nature · doi:10.1038/nature06713 · Cited by 1,272 · see citing works
Predicting Human Brain Activity Associated with the Meanings of Nouns
Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008) · Science · doi:10.1126/science.1152876 · Cited by 1,180 · see citing works
Guclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961 · see citing works
Discovering Event Structure in Continuous Narrative Perception and Memory
Baldassano, C., Chen, J., Zadbood, A., Pillow, J. W., Hasson, U., & Norman, K. A. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.041 · Cited by 952 · see citing works
Naselaris, T., Kay, K., Nishimoto, S., & Gallant, J. L. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.07.073 · Cited by 894 · see citing works
Topographic mapping of a hierarchy of temporal receptive windows using a narrated story
Lerner, Y., Honey, C. J., Silbert, L. J., & Hasson, U. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3684-10.2011 · Cited by 834 · see citing works
Semi-supervised sequence tagging with bidirectional language models
Peters, M., Ammar, W., Bhagavatula, C., & Power, R. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1161 · Cited by 655 · see citing works
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 2× · see citing works
“Some previous natural language processing works use only the last layer (17; 12) while others use a linear combination across layers (18).”
BackgroundKell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544 · see citing works
Seeing it all: Convolutional network layers map the function of the human visual system
Eickenberg, M., Gramfort, A., Varoquaux, G., & Thirion, B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.001 · Cited by 338 · see citing works
COMPLETE FUNCTIONAL CHARACTERIZATION OF SENSORY NEURONS BY SYSTEM IDENTIFICATION
Wu, M. C., David, S. V., & Gallant, J. L. (2006) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.113024 · Cited by 332 · see citing works
Toward a universal decoder of linguistic meaning from brain activation
Pereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, S. J., Kanwisher, N., Botvinick, M., & Fedorenko, E. (2018) · Nature Communications · doi:10.1038/s41467-018-03068-4 · Cited by 324 · see citing works
Wehbe, L., Murphy, B., Talukdar, P. P., Fyshe, A., Ramdas, A., & Mitchell, T. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112575 · Cited by 256 · see citing works
Pycortex: an interactive surface visualizer for fMRI
Gao, J. S., Huth, A. G., Lescroart, M. D., & Gallant, J. L. (2015) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2015.00023 · Cited by 171 · see citing works
Self-citationAligning context-based statistical models of language with brain activity during reading
Wehbe, L., Vaswani, A., Knight, K., & Mitchell, T. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1030 · Cited by 95 · see citing works
Pixels to Voxels: Modeling Visual Representation in the Human BrainPreprint
Agrawal, P., Stansbury, D. E., Malik, J., & Gallant, J. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.5104 · Cited by 91 · see citing works
BrainBench: A Brain-Image Test Suite for Distributional Semantic Models
Xu, H., Murphy, B., & Fyshe, A. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1213 · Cited by 31 · see citing works
Bridging LSTM Architecture and the Neural Dynamics during ReadingPreprint
Qian, P., Qiu, X., & Huang, X. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06635 · Cited by 14 · see citing works
Flagged first, then by citations
5 of the 30 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- de Heer, W.A. , Huth, A.G. , Griffiths, T.L. , Gallant, J.L. , Theunissen, F.E. : The hierarchical cortical organization of human speech processing. Journal of Neuroscience (2017), http://www.jneurosci.org/content/early/2017/06/06/JNEUROSCI.3267-16.2017
- 2013, Advances in Neural Information Processing Systems, 26, 3111
Cited by
Sorted by citations
The neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517 · see citing works
Shared computational principles for language processing in humans and deep language models
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01026-4 · Cited by 447 · see citing works
“These results agree with concurrent studies demonstrating that contextual embeddings model neural responses to words better than static semantic embeddings 15,16,45,46 .”
ResultSemantic reconstruction of continuous language from non-invasive brain recordings
Tang, J., LeBel, A., Jain, S., & Huth, A. G. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01304-9 · Cited by 442 · see citing works
“Previous studies have shown that middle layers of language models extract the best semantic features for predicting brain responses to natural language 8,14,15,17 .”
BackgroundMethodSelf-citationA hierarchy of linguistic predictions during natural language comprehension
Heilbron, M., Armeni, K., Schoffelen, J.-M., Hagoort, P., & de Lange, F. P. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2201968119 · Cited by 343 · see citing works
Evidence of a predictive coding hierarchy in the human brain listening to speech
Caucheteux, C., Gramfort, A., & King, J.-R. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01516-2 · Cited by 308 · see citing works
Language is primarily a tool for communication rather than thought
Fedorenko, E., Piantadosi, S. T., & Gibson, E. A. F. (2024) · Nature · doi:10.1038/s41586-024-07522-w · Cited by 190 · see citing works
“Recent advances in artificial intelligence—the development of neural network models that excel at language 201 —have provided language researchers with a suite of powerful tools to probe the neural codes of linguistic processing 48,202–205 (reviewed in ref.”
BackgroundProbabilistic atlas for the language network based on precision fMRI data from >800 individuals
Lipkin, B., Tuckute, G., Affourtit, J., Small, H., Mineroff, Z., Kean, H., Jouravlev, O., Rakocevic, L. I., Pritchett, B., Siegelman, M., et al. (2022) · Scientific Data · doi:10.1038/s41597-022-01645-3 · Cited by 149 · see citing works
Toneva, M., & Wehbe, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11833 · Cited by 152 · see citing works
“Jain and Huth (2018) aligned layers from a Long Short-Term Memory (LSTM) model to fMRI recordings of subjects listening to stories to differentiate between the amount of context maintained by each brain region.”
BackgroundMethodResultThe neural architecture of language: Integrative modeling converges on predictive processing
(2020) · doi:10.1101/2020.06.26.174482 · Cited by 129 · see citing works
“…et al., 2017; Pallier et al., 2011; Pereira et al., 2018; Rabovsky et al., 2018; Shain et al., 2020; Wehbe et al., 2014; Willems et al., 2016; Gauthier & Ivanova, 2018; Gauthier & Levy, 2019; Hu et al., 2020; Jain & Huth, 2018; S. Wang et al., 2020; Schwartz et al., 2019; Toneva & Wehbe, 2019).”
Constructing and Forgetting Temporal Context in the Human Cerebral Cortex
Chien, H.-Y. S., & Honey, C. J. (2020) · Neuron · doi:10.1016/j.neuron.2020.02.013 · Cited by 86 · see citing works
Deep language algorithms predict semantic comprehension from brain activity
Caucheteux, C., Gramfort, A., & King, J.-R. (2022) · Scientific Reports · doi:10.1038/s41598-022-20460-9 · Cited by 91 · see citing works
“1C), as previously reported 5,7,29 .”
ResultDriving and suppressing the human language network using large language models
Tuckute, G., Sathe, A., Srikant, S., Taliaferro, M., Wang, M., Schrimpf, M., Kay, K., & Fedorenko, E. G. (2024) · Nature Human Behaviour · doi:10.1038/s41562-023-01783-7 · Cited by 85 · see citing works
“…modulates language processing difficulty in both behavioural psycholinguistic work 40–42 and brain imaging investigations 43–47 , we computed sentence-level log probability estimates for each of 2,000 sentences using GPT2-XL (see ‘Sentence properties that modulate brain responses’ in Methods).”
Linking artificial and human neural representations of language
Gauthier, J., & Lévy, R. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1050 · Cited by 72 · see citing works
“Wehbe et al. (2014), Qian et al. (2016), Jain and Huth (2018), and Abnar et al. (2019) next introduced more complex word and sentence meaning representations, demonstrating that neural network language models could better account for brain activation by incorporating repre-Figure sentations of…”
BackgroundAnderson, A. J., Kiela, D., Binder, J. R., Fernandino, L., Humphries, C. J., Conant, L. L., Raizada, R. D. S., Grimm, S., & Lalor, E. C. (2021) · The Journal of Neuroscience · doi:10.1523/jneurosci.1152-20.2021 · Cited by 70 · see citing works
“…Honey, 2020); and also other studies that have begun to use deep network-based models to predict contextualized semantic responses in neural data (Jain and Huth, 2018; Gauthier and Levy, 2019; Sun et al., 2019; Toneva and Wehbe, 2019; Goldstein et al., 2020; Heilbron et al., 2020; Schrimpf et…”
BackgroundGoldstein, A., Grinstein-Dabush, A., Schain, M., Wang, H., Hong, Z., Aubrey, B., Nastase, S. A., Zada, Z., Ham, E., Feder, A., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-46631-y · Cited by 64 · see citing works
“…of (3) using contextual embeddings to understand context-sensitive language processing in the brain: With the recent development of context-sensitive DLMs, we see an explosion of work demonstrating that contextual embeddings better predict brain activity than static embeddings 34 – 41 .”
Missing links: The functional unification of language and memory (L∪M)
Roger, É., Banjac, S., de Schotten, M. T., & Baciu, M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.12.012 · Cited by 61 · see citing works
Shared functional specialization in transformer-based language models and the human brain
Kumar, S., Sumers, T. R., Yamakoshi, T., Goldstein, A., Hasson, U., Norman, K. A., Griffiths, T. L., Hawkins, R. D., & Nastase, S. A. (2024) · Nature Communications · doi:10.1038/s41467-024-49173-5 · Cited by 94 · see citing works
“The success of Transformers has inspired a growing body of neuroscienti fi c work using them to model human brain activity during natural language comprehension ( 43 – 53 ; cf. 52,54 ).”
BackgroundLanguage in Brains, Minds, and Machines
Tuckute, G., Kanwisher, N., & Fedorenko, E. G. (2024) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-120623-101142 · Cited by 69 · see citing works
“Furthermore, most studies that compare LM representations to brains do not test generalization to held-out participants ( Jain & Huth 2018, Toneva & Wehbe 2019, Jain et al. 2020, Schrimpf et al. 2021, Merlin & Toneva 2022, Oota et al. 2022b, Aw & Toneva 2023, Hosseini et al. 2024; cf. Toneva et al.…”
Cortical response to naturalistic stimuli is largely predictable with deep neural networks
Khosla, M., Ngo, G. H., Jamison, K. W., Kuceyeski, A. F., & Sabuncu, M. R. (2021) · Science Advances · doi:10.1126/sciadv.abe7547 · Cited by 57 · see citing works
Hosseini, E. A., Schrimpf, M., Zhang, Y., Bowman, S., Zaslavsky, N., & Fedorenko, E. (2024) · Neurobiology of Language · doi:10.1162/nol_a_00137 · Cited by 63 · see citing works
Predicting speech from a cortical hierarchy of event-based time scales
Schmitt, L., Erb, J., Tune, S., Rysop, A. U., Hartwigsen, G., & Obleser, J. (2021) · Science Advances · doi:10.1126/sciadv.abi6070 · Cited by 56 · see citing works
“Furthermore, LSTM models have been shown to effectively explain neural (46) and behavioral responses to speech (47).”
BackgroundAbnar, S., Beinborn, L., Choenni, R., & Zuidema, W. H. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4820 · Cited by 67 · see citing works
“Jain and Huth (2018) report that the higher layers of the LSTM are better at predicting the activation of brain regions that are known for higher level language functions (a finding seemingly at odds with results from section5).”
Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machinesPreprint
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A. R., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.12.02.403477 · Cited by 51 · see citing works
“These results agree with recent studies demonstrating that contextual embeddings better model neural responses to words than static semantic embeddings 18,19,48 .”
ResultA natural language fMRI dataset for voxelwise encoding models
LeBel, A., Wagner, L., Jain, S., Adhikari-Desai, A., Gupta, B., Morgenthal, A., Tang, J., Xu, L., & Huth, A. G. (2023) · Scientific Data · doi:10.1038/s41597-023-02437-z · Cited by 52 · see citing works
Self-citationSemantic Representations during Language Comprehension Are Affected by Context
Deniz, F., Tseng, C., Wehbe, L., la Tour, T. D., & Gallant, J. L. (2023) · Journal of Neuroscience · doi:10.1523/jneurosci.2459-21.2023 · Cited by 52 · see citing works
Language processing in brains and deep neural networks: computational convergence and its limitsPreprint
Caucheteux, C., & King, J.-R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.03.186288 · Cited by 49 · see citing works
Combining computational controls with natural text reveals aspects of meaning composition
Toneva, M., Mitchell, T. M., & Wehbe, L. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00354-6 · Cited by 60 · see citing works
“In a different line of work, neuroscientists build computational models of meaning through Natural Language Processing (NLP) embeddings of words and sentences (Mitchell et al., 2008; Sudre et al., 2012; Wehbe et al., 2014b,a; Huth et al., 2016; Jain and Huth, 2018; Toneva and Wehbe, 2019; Fyshe et al., 2019).”
BackgroundGPT-2’s activations predict the degree of semantic comprehension in the human brainPreprint
Caucheteux, C., Gramfort, A., & King, J.-R. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.04.20.440622 · Cited by 46 · see citing works
“By extracting GPT-2 activations from multiple processing 50 stages, called "layers" (from layer one to layer twelve), we 51 confirmed that middle layers best aligned with the brain (Fig- 52 ure 1C), as seen in previous studies (2, 4, 9).”
MethodResultSemantic reconstruction of continuous language from non-invasive brain recordings
(2022) · doi:10.1101/2022.09.29.509744 · Cited by 442 · see citing works
Self-citationAntonello, R., & Huth, A. G. (2022) · Neurobiology of Language · doi:10.1162/nol_a_00087 · Cited by 49 · see citing works
“…than earlier methods (Anderson et al., 2021; Antonello et al., 2021; Caucheteux et al., 2021a; Caucheteux & King, 2022; Goldstein et al., 2021; Jain & Huth, 2018; Jat et al., 2019; LeBel, Jain, & Huth, 2021; Li et al., 2021; Schrimpf et al., 2021; Tikochinski et al., 2021; Toneva et al.,…”
BackgroundMethodSelf-citationWhat Limits Our Capacity to Process Nested Long-Range Dependencies in Sentence Comprehension?
Lakretz, Y., Dehaene, S., & King, J.-R. (2020) · Entropy · doi:10.3390/e22040446 · Cited by 31 · see citing works
Desbordes, T., Lakretz, Y., Chanoine, V., Oquab, M., Badier, J., Trébuchon, A., Carron, R., Bénar, C., Dehaene, S., & King, J.-R. (2023) · Journal of Neuroscience · doi:10.1523/jneurosci.1163-22.2023 · Cited by 36 · see citing works
Hubbard, R. J., & Federmeier, K. D. (2021) · Cerebral Cortex · doi:10.1093/cercor/bhab087 · Cited by 31 · see citing works
Disentangling Syntax and Semantics in the Brain with Deep NetworksPreprint
Caucheteux, C., Gramfort, A., & King, J.-R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01620 · Cited by 41 · see citing works
“Interestingly, the activations of language transformers tend to linearly map onto those of the human brain, when presented with the same sentences (Jain & Huth, 2018; Toneva & Wehbe, 2019; Abnar et al., 2019; Schrimpf et al., 2020; Caucheteux & King, 2020; Goldstein et al., 2021).”
BackgroundInterpretable multi-timescale models for predicting fMRI responses to continuous natural speechPreprint
Jain, S., Mahto, S., Turek, J. S., Vo, V. A., LeBel, A., & Huth, A. G. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.10.02.324392 · Cited by 27 · see citing works
“In sum, the CL method can provide evidence for a temporal hierarchy in the cortex, as shown previously [8].”
BackgroundMethodInfluentialSelf-citationDeep learning models to study sentence comprehension in the human brain
Arana, S., Lerousseau, J. P., & Peter Hagoort, A. (2023) · Language Cognition and Neuroscience · doi:10.1080/23273798.2023.2198245 · Cited by 28 · see citing works
“While multiple studies report a spatially varying gradient of contextualisation in the brain (Antonello et al., 2021; Jain & Huth, 2018; Qian et al., 2016; Schmitt et al., 2021; Toneva & Wehbe, 2019), they vary in terms of the exact mapping of this gradient onto brain areas.”
Vu, H., Kim, H., Jung, M., & Lee, J. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117328 · Cited by 28 · see citing works
Yu, S., Gu, C., Huang, K., & Li, P. (2024) · Science Advances · doi:10.1126/sciadv.adn7744 · Cited by 27 · see citing works
“Most findings demonstrated that model-brain alignment peaked or plateaued at a short context length of about 10 words ( 2 , 57 , 58 ).”
BackgroundMethodCan fMRI reveal the representation of syntactic structure in the brain?Preprint
Reddy, A. J., & Wehbe, L. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.16.155499 · Cited by 25 · see citing works
“We use layer 12 because of previous work showing that middle layers of sentence encoders are optimal for predicting brain activity [28, 27].”
MethodIncremental accumulation of linguistic context in artificial and biological neural networks
Tikochinski, R., Goldstein, A., Meiri, Y., Hasson, U., & Reichart, R. (2025) · Nature Communications · doi:10.1038/s41467-025-56162-9 · Cited by 24 · see citing works
“- Recent LLMs like Longformer, a Transformer variant, use a modified attention mechanism for handling longer contexts through a combination of local and global attention.”
Top-down information shapes lexical processing when listening to continuous speech
Gwilliams, L., Marantz, A., Poeppel, D., & King, J.-R. (2023) · Language Cognition and Neuroscience · doi:10.1080/23273798.2023.2171072 · Cited by 28 · see citing works
“One key result of this previous work is that arti fi cial models do better at predicting neural responses when they make use of longer context windows (Caucheteux & King, 2021, 2022; Goldstein et al., 2022; Jain & Huth, 2018).”
Large language models can segment narrative events similarly to humans
Michelmann, S., Kumar, M., Norman, K. A., & Toneva, M. (2025) · Behavior Research Methods · doi:10.3758/s13428-024-02569-z · Cited by 40 · see citing works
“…with human behavior (Caucheteux et al., 2021; Golan et al., 2022; Kumar et al., 2022) and human brain activity during language comprehension (Jain & Huth, 2018; Toneva & Wehbe, 2019; Schrimpf et al., 2021; Caucheteux & King, 2022; Goldstein et al., 2022), finding many significant…”
Driving and suppressing the human language network using large language modelsPreprint
Tuckute, G., Sathe, A., Srikant, S., Taliaferro, M., Wang, M., Schrimpf, M., Kay, K., & Fedorenko, E. G. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.16.537080 · Cited by 20 · see citing works
Gwilliams, L., Flick, G., Marantz, A., Liina Pylkkänen, A., Poeppel, D., & King, J.-R. (2023) · Scientific Data · doi:10.1038/s41597-023-02752-5 · Cited by 22 · see citing works
Behavioral correlates of cortical semantic representations modeled by word vectors
Nishida, S., Blanc, A., Maeda, N., Kado, M., & Nishimoto, S. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009138 · Cited by 20 · see citing works
“The feature representations obtained by these algorithms are also beginning to be used for the modeling of semantic representations (Jain and Huth, 2018; Jat et al., 2019), which show state-of-the-art performance even in brain-response prediction (Schrimpf et al.”
MethodArtificial neural network modelling of the neural population code underlying mathematical operations
Nakai, T., & Nishimoto, S. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.119980 · Cited by 19 · see citing works
Brain-optimized extraction of complex sound features that drive continuous auditory perception
Berezutskaya, J., Freudenburg, Z. V., Güçlü, U., van Gerven, M. A. J., & Ramsey, N. F. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007992 · Cited by 17 · see citing works
“Recent work in computational neuroscience indicates that the multilevel features captured in artificial neural networks (ANNs) relate to the hierarchy of cortical representations in visual perception [10–12], music [13] and language [14] processing.”
BackgroundDesbordes, T., King, J.-R., & Dehaene, S. (2024) · Cell Reports · doi:10.1016/j.celrep.2024.113847 · Cited by 18 · see citing works
Abdou, M., González, A. V., Toneva, M., Hershcovich, D., & Søgaard, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.12608 · Cited by 22 · see citing works
“Previous work has evaluated the plausibility of such candidate models by testing how well representations of text extracted from these models align with brain recordings of humans during language comprehension tasks (Wehbe et al., 2014; Jain and Huth, 2018; Gauthier and Ivanova, 2018; Gauthier and Levy, 2019; Abnar et…”
BackgroundFeature-space selection with banded ridge regression
(2022) · doi:10.1101/2022.05.05.490831 · Cited by 19 · see citing works
Towards decoding individual words from non-invasive brain recordings
d’Ascoli, S., Bel, C., Rapin, J., Banville, H. J., Benchetrit, Y., Pallier, C., & King, J.-R. (2025) · Nature Communications · doi:10.1038/s41467-025-65499-0 · Cited by 20 · see citing works
“Since several studies systematically compared pre-trained language models to the fMRI 20,47 – 50 , intracranial 19,51 and M/EEG responses to spoken 30,52 and written sentences 30 .”
Honari-Jahromi, M., Chouinard, B., Blanco-Elorrieta, E., Liina Pylkkänen, A., & Fyshe, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0242754 · Cited by 16 · see citing works
“Decoding has also been successful using data collected while people listen to language (Huth et al., 2016, Jain & Huth, 2018).”
Language acquisition: do children and language models follow similar learning stages?
Evanson, L., Lakretz, Y., & King, J. R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.773 · Cited by 27 · see citing works
Constructing and Forgetting Temporal Context in the Human Cerebral Cortex
(2019) · doi:10.1101/761593 · Cited by 18 · see citing works
“In future work, we will train HAT variants on linguistic corpora and use these to generate context-aware encoding models of the neural response to complex language (e.g., Jain and Huth, 2018).”
MethodDistributional Semantics: Meaning Through Culture and Interaction
Contreras Kallens, P., & Christiansen, M. H. (2025) · Topics in Cognitive Science · doi:10.1111/tops.12771 · Cited by 17 · see citing works
“For example, LLMs are better at capturing the granularity of the brain’s semantic space (Sun et al., 2020) thanks to the inclusion of context (Goldstein et al., 2022; Jain & Huth, 2018) and the optimization of the space for in-context prediction (Caucheteux & King, 2022; Goldstein et al., 2022).”
Encoding of speech in convolutional layers and the brain stem based on language experience
Beguš, G., Zhou, A., & Zhao, T. C. (2023) · Scientific Reports · doi:10.1038/s41598-023-33384-9 · Cited by 14 · see citing works
“A substantial amount of work exists on paralleling brain imaging with artificial neural networks in the visual domain 7,9,18–24 and relatively fewer works exist in the language or in the speech domain (most works focus on text-based language models 25–27 ).”
The cortical representation of language timescales is shared between reading and listening
Chen, C., la Tour, T. D., Gallant, J. L., Klein, D. B., & Deniz, F. (2024) · Communications Biology · doi:10.1038/s42003-024-05909-z · Cited by 19 · see citing works
“Studies that did differentiate between different levels focused on one modality of language 2,5,6,15 .”
BackgroundMethodla Tour, T. D., Castello, M. V. D. O., & Gallant, J. L. (2025) · Imaging Neuroscience · doi:10.1162/imag_a_00575 · Cited by 18 · see citing works
“…a, *, Matteo Visconti di Oleggio Castello a,b, *, Jack L. Gallant a,b Imaging Neuroscience, Volume 3, 2025 Wehbe et al., 2014), and narrative language ( Caucheteux & King, 2022; de Heer et al., 2017; Deniz et al., 2019; Huth et al., 2016; Jain & Huth, 2018; Jain et al., 2020; Toneva & Wehbe, 2019).”
Djokic, V. G., Maillard, J., Bulat, L., & Shutova, E. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00307 · Cited by 14 · see citing works
“Recently, Jain and Huth (2018) investigated long short-term memory (LSTM) recurrent neural networks and showed that semantic models that incorporate larger-sized context windows outperform those with smaller-sized context windows, as well as the baseline bag-of-words model, in predicting brain…”
InfluentialAnderson, A. J., Davis, C., & Lalor, E. C. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012537 · Cited by 13 · see citing works
Modelling brain representations of abstract concepts
KAISER, D. H., Jacobs, A. M., & Cichy, R. M. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009837 · Cited by 13 · see citing works
“Other studies have probed correspondences between language processing models and the brain through the use of encoding models [2,3,6,48,50].”
BackgroundBeyond linear regression: mapping models in cognitive neuroscience should align with research goalsPreprint
Ivanova, A. A., Schrimpf, M., Anzellotti, S., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.04.02.438248 · Cited by 10 · see citing works
Tang, J., LeBel, A., & Huth, A. G. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.05.19.444701 · Cited by 12 · see citing works
“We then fit voxelwise encoding models that separately predict the fMRI data in each subject from the stimulus words (Huth et al., 2016; Jain and Huth, 2018; Wehbe et al., 2014).”
MethodResultSelf-citationComputational Architecture of Speech Comprehension in the Human Brain
Gwilliams, L., Bhaya-Grossman, I., Zhang, Y., Scott, T. L., Harper, S., & Levy, D. F. (2024) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-111245 · Cited by 12 · see citing works
Privileged representational axes in biological and artificial neural networksPreprint
Khosla, M., Williams, A. H., McDermott, J. H., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.20.599957 · Cited by 11 · see citing works
(2023) · doi:10.1101/2023.05.05.539646 · Cited by 11 · see citing works
“…only be capturing the local processing of the current sentence and would therefore align better with decontextualized sentence representations (see Jain & Huth, 2018; Caucheteux et al., 2021 for evidence that ANN representations with varying amounts of linguistic context lead to differential…”
BackgroundMethodInfluentialLinguistic contextualization in the human hippocampusPreprint
Katlowitz, K. A., Belanger, J. L., Ismail, T., Chavez, A. G., Chericoni, A., Franch, M. C., Mickiewicz, E. A., Mathura, R. K., Paulo, D. L., Bartoli, E., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.23.661103 · Cited by 10 · see citing works
“The fact that word-evoked neural responses depend on preceding words implies that the neural data should be better fit by contextual semantic models than by non-contextual ones 71 .”
Velichkovsky, B. M., Заботкина, В., Nosovets, Z. A., Kotov, A. A., Zaidelman, L., Kartashov, S., Korosteleva, A. N., Malakhov, D. G., Orlov, V. A., Zinina, A., et al. (2020) · Современные технологии в медицине · doi:10.17691/stm2020.12.2.02 · Cited by 11 · see citing works
“New modelling efforts are directed at overcoming this limitation [21].”
BackgroundStepEncog: A Convolutional LSTM Autoencoder for Near-Perfect fMRI Encoding
Oota, S. R., Rowtula, V., Gupta, M., et al. (2019) · doi:10.1109/ijcnn.2019.8852339 · Cited by 9 · see citing works
“We experimented with multiple layers (3 layers) of LSTM, inspired by the experiments from [16].”
MethodInfluentialSchönmann, I., Szewczyk, J., de Lange, F. P., & Heilbron, M. (2025) · eLife · doi:10.7554/elife.106543 · Cited by 9 · see citing works
“…also constitute the highest performing encoding models for predicting brain responses to linguistic stimuli ( Caucheteux and King, 2022 ; Jain and Huth, 2018 ; Schrimpf et al., 2021 )—indicating that LLMs’ internal representations might capture some aspects of human language…”
Semantic representations during language comprehension are affected by context
(2021) · doi:10.1101/2021.12.15.472839 · Cited by 8 · see citing works
“Furthermore, other studies have proposed alternative models that integrate contextual semantic information differently than the model used here (Jain and Huth, 2018; Toneva and Wehbe, 2019), and it is possible that these other models might predict voxel activity better than the semantic model used here.”
BackgroundMethodMapping Brains with Language Models: A Survey
Karamolegkou, A., Abdou, M., & Søgaard, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.618 · Cited by 16 · see citing works
Lee, J., Jung, M., Lustig, N., & Lee, J. (2023) · Human Brain Mapping · doi:10.1002/hbm.26189 · Cited by 5 · see citing works
“A growing body of research has utilized DNN models to understand neural representations within the human brain as measured using non-invasive fMRI (Cross et al., 2020; Jain & Huth, 2018; Kell et al., 2018).”
MethodDeep Recurrent Encoder: an end-to-end network to model magnetoencephalography at scale
Chehab, O., Defossez, A., Loiseau, J.-C., Gramfort, A., & King, J.-R. (2022) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.38668 · Cited by 6 · see citing works
“In particular, deep nets trained on natural images [88, 89, 90], sounds [5], or text [91] are being increasingly used as an encoding modelforpredictingneuralresponsestosensorystimulation[6,92,7,8].”
MethodStacked regressions and structured variance partitioning for interpretable brain maps
Lin, R., Naselaris, T., Kay, K., & Wehbe, L. (2024) · NeuroImage · doi:10.1016/j.neuroimage.2024.120772 · Cited by 8 · see citing works
Brain-inspired computing based on deep learning for human-computer interaction: A review
Yu, B., Zhang, S., Zhou, L., Wei, J., Sun, L., & Bu, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130928 · Cited by 6 · see citing works
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)Preprint
Oota, S. R., Chen, Z., Gupta, M., Bapi, R. S., Jobard, G., Alexandre, F., & Hinaut, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10246 · Cited by 17 · see citing works
“…representation models (Sun et al., 2020; 2019; Toneva & Wehbe, 2019), RNNs (Jain & Huth, 2018; Oota et al., 2019) and Transformer methods (Gauthier & Levy, 2019; Toneva & Wehbe, 2019; Schwartz et al., 2019; Schrimpf et al., 2021; Antonello et al., 2021; Oota et al., 2022b; Aw & Toneva, 2023).”
BackgroundInfluentialA natural language fMRI dataset for voxelwise encoding models
(2022) · doi:10.1101/2022.09.22.509104 · Cited by 6 · see citing works
Self-citationTowards large language models with human-like episodic memory
Dong, C., Lu, Q., Norman, K. A., & Michelmann, S. (2025) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.06.016 · Cited by 10 · see citing works
(2023) · doi:10.1101/2023.02.28.530443 · Cited by 6 · see citing works
“Distributed word representations from NLMs were shown to align with cortical responses to words by means of linear encoding models (Huth et al., 2016; Jain & Huth, 2018; Toneva & Wehbe, 2019; Caucheteux & King, 2020; Caucheteux et al., 2021; Schrimpf et al., 2021; Goldstein et al., 2022).”
MethodRole of Punctuation in Semantic Mapping Between Brain and Transformer Models
Lamprou, Z., Pollick, F. E., & Moshfeghi, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25891-6_35 · Cited by 4 · see citing works
Attention weights accurately predict language representations in the brain
(2022) · doi:10.1101/2022.12.07.519480 · Cited by 5 · see citing works
“Encoding models have previously been used to determine how variables such as lexical and contextual semantics are represented in brain responses (Huth et al., 2016; de Heer et al., 2017; Deniz et al., 2019; Jain and Huth, 2018).”
BackgroundMethodLing, S., Murphy, A., & Fyshe, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00533 · Cited by 6 · see citing works
“Timescales have been extracted from RNNs (Chien and Honey 2020), LSTMs (Jain and Huth 2018; Jain et al. 2020), GPT (Caucheteux, Gramfort, and King 2021; Goldstein et al. 2022b; Heilbron et al. 2022), and BERT (Chen et al. 2024) and compared with brain responses to language.”
InfluentialSame Cause; Different Effects in the Brain
Toneva, M., Williams, J. R., Bollu, A., Dann, C., & Wehbe, L. (2022) · PubMed · doi:10.48550/arxiv.2202.10376 · Cited by 8 · see citing works
“, 2020; Hamilton and Huth, 2020), and to use complex representations from neural networks to build encoding models (Yamins et al., 2014; Wehbe et al., 2014b; Jain and Huth, 2018; Toneva and Wehbe, 2019; Wang et al., 2019; Schrimpf et al., 2020; Caucheteux et al., 2021b; Goldstein et al., 2021; Cross et al., 2021), it…”
BackgroundFunctional brain mapping using whole-head very high-density diffuse optical tomography
Fogarty, M., Rafferty, S. M., Markow, Z. E., O’Sullivan, A. C., Svoboda, C. F., George, T., King, K., Wilhelm, D., Tripathy, K., Mugler, E. M., et al. (2025) · Imaging Neuroscience · doi:10.1162/imag.a.54 · Cited by 5 · see citing works
“For complex functional brain systems, such as the semantic system, naturalistic tasks from visual ( Huth et al., 2012) and auditory ( Huth, De Heer, et al., 2016; Jain & Huth, 2018) content can provide a diverse generalizable sampling of the feature space.”
Robust Evaluation of Language–Brain Encoding Experiments
Beinborn, L., Abnar, S., & Choenni, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-24337-0_4 · Cited by 12 · see citing works
“Jain and Huth [19] calculate a different r 2 value: they multiply the Pearson correlation between the predictions and the observed activations for voxel v i with the absolute correlation ( r 2 ( v i ) = r v i ×| r v i | ).”
InfluentialChen, C. L., Gong, X. L., Tseng, C., Klein, D. L., Gallant, J. L., & Deniz, F. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2503721123 · Cited by 14 · see citing works
A geometric foundation for word meaning in the brainPreprint
Zhu, H., Franch, M. C., Mickiewicz, E. A., Belanger, J. L., Cowan, R. L., Katlowitz, K. A., Chavez, A. G. L., Chericoni, A., Paulo, D. L., Yan, X., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.28.702241 · Cited by 4 · see citing works
Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain
Liu, X., Zhou, M., Shi, G., Du, Y., Zhao, L., Wu, Z., Liu, D., Liu, T., & Hu, X. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.26068 · Cited by 7 · see citing works
Bruera, A., & Poesio, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0291099 · Cited by 4 · see citing works
Attention weights accurately predict language representations in the brain
Lamarre, M., Chen, C., & Deniz, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.330 · Cited by 9 · see citing works
Reilly, J. J., Zuckerman, B. M., Finley, A. M., Litovsky, C. P., & Kenett, Y. N. (2022) · doi:10.31234/osf.io/6fuhv · Cited by 4 · see citing works
Language cognition and language computation — human and machine language understanding
王, 少., 丁, 鼐., 林, 楠., 张, 家., & 宗, 成. (2021) · Scientia Sinica Informationis · doi:10.1360/ssi-2021-0100 · Cited by 4 · see citing works
Bhattasali, S., & Resnik, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.332 · Cited by 4 · see citing works
- Authors
- Jain, S., & Huth, A. G.
- Affiliations
- The University of Texas at Austin (US)
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2018-05-21
- OpenAlex
- W2805003518
- Cited by
- 155
- References
- 30
- Citations by year
- 2026: 8 · 2025: 19 · 2024: 27 · 2023: 36 · 2022: 17 · 2021: 23 · 2020: 17 · 2019: 8
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- Funders
- National Science Foundation (1208203)
- Nvidia
- University of Texas at Austin
- National Institutes of Health
- National Eye Institute (EY019684)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- functional MRI, context sensitivity, LSTM language models, word embeddings, context representation, language processing in the brain
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
