Aligning context-based statistical models of language with brain activity during reading
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Many statistical models for natural language processing exist, including context-based neural networks that (1) model the previously seen context as a latent feature vector, (2) integrate successive words into the context using some learned representation (embedding), and (3) compute output probabilities for incoming words given the context.On the other hand, brain imaging studies have suggested that during reading, the brain (a) continuously builds a context from the successive words and every time it encounters a word it (b) fetches its properties from memory and (c) integrates it with the previous context with a degree of effort that is inversely proportional to how probable the word is.This hints to a parallelism between the neural networks and the brain in modeling context (1 and a), representing the incoming words (2 and b) and integrating it (3 and c).We explore this parallelism to better understand the brain processes and the neural networks representations.We study the alignment between the latent vectors used by neural networks and brain activity observed via Magnetoencephalography (MEG) when subjects read a story.For that purpose we apply the neural network to the same text the subjects are reading, and explore the ability of these three vector representations to predict the observed word-by-word brain activity.Our novel results show that: before a new word i is read, brain activity is well predicted by the neural network latent representation of context and the predictability decreases as the brain integrates the word and changes its own representation of context.Secondly, the neural network embedding of word i can predict the MEG activity when word i is presented to the subject, revealing that it is correlated with the brain's own representation of word i.Moreover, we obtain that the activity is predicted in different regions of the brain with varying delay.The delay is consistent with the placement of each region on the processing pathway that starts in the visual cortex and moves to higher level regions.Finally, we show that the output probability computed by the neural networks agrees with the brain's own assessment of the probability of word i, as it can be used to predict the brain activity after the word i's properties have been fetched from memory and the brain is in the process of integrating it into the context.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further, a previous study suggests that representations learned by an LM can be useful for modeling MEG brain responses to reading (23).”
From Incorporating Context into Language Encoding Models for fMRI (2018) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2010) · International Conference on Machine Learning · Cited by 13,995
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · cited 1×
“We control the false discovery rate using (Benjamini and Yekutieli, 2001) to adjust for the testing at multiple locations and time windows.”
MethodGeneralized Cross-Validation as a Method for Choosing a Good Ridge Parameter
Golub, G. H., Heath, M., & Wahba, G. (1979) · Technometrics · doi:10.1080/00401706.1979.10489751 · Cited by 3,934
Towards a neural basis of auditory sentence processing
Friederici, A. D. (2002) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01839-8 · Cited by 2,231
Information-based functional brain mapping
Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0600244103 · Cited by 2,113 · cited 1×
“Restricting the analysis spatially: a searchlight equivalent We adapt the searchlight method (Kriegeskorte et al., 2006) to MEG.”
MethodSpatiotemporal signal space separation method for rejecting nearby interference in MEG measurements
Taulu, S., & Simola, J. (2006) · Physics in Medicine and Biology · doi:10.1088/0031-9155/51/7/008 · Cited by 1,549 · cited 1×
“For preprocessing, we used Signal Space Separation method (SSS, (Taulu et al., 2004)), followed by its temporal extension (tSSS, (Taulu and Simola, 2006)).”
MethodStatistical Methods for Research Workers
(1926) · Journal of Comparative Pathology and Therapeutics · doi:10.1016/s0368-1742(26)80018-9 · Cited by 1,283
Suppression of Interference and Artifacts by the Signal Space Separation Method
Taulu, S., Kajola, M., & Simola, J. (2004) · Brain Topography · doi:10.1023/b:brat.0000032864.93890.f9 · Cited by 541 · cited 1×
“For preprocessing, we used Signal Space Separation method (SSS, (Taulu et al., 2004)), followed by its temporal extension (tSSS, (Taulu and Simola, 2006)).”
MethodHagoort, P. (2003) · NeuroImage · doi:10.1016/j.neuroimage.2003.09.013 · Cited by 352 · cited 1×
“Less is understood about the cortical dynamics of word integration, as multiple theories exist (Friederici, 2002; Hagoort, 2003).”
Background
Research works printed but not linked: 1
- Tomas Mikolov. 2012. Statistical Language Models Based on Neural Networks. Ph.D. thesis, Brno Uni- versity of Technology.
Cited by
The neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517
Shared computational principles for language processing in humans and deep language models
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01026-4 · Cited by 447
“These results agree with concurrent studies demonstrating that contextual embeddings model neural responses to words better than static semantic embeddings 15,16,45,46 .”
ResultBrains and algorithms partially converge in natural language processing
Caucheteux, C., & King, J.-R. (2022) · Communications Biology · doi:10.1038/s42003-022-03036-1 · Cited by 318
“First, our work complements previous studies 26,27,30 – 34 and con fi rms that the activations of deep language models signi fi cantly map onto the brain responses to written sentences (Fig.”
ResultEvidence of a predictive coding hierarchy in the human brain listening to speech
Caucheteux, C., Gramfort, A., & King, J.-R. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01516-2 · Cited by 308
Incorporating Context into Language Encoding Models for fMRI
Jain, S., & Huth, A. G. (2018) · doi:10.1101/327601 · Cited by 155
“Further, previous studies suggest that representations learned by LMs can be useful for modeling brain responses (23; 19).”
BackgroundFinding syntax in human encephalography with beam search
Hale, J., Dyer, C., Kuncoro, A., & Brennan, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1254 · Cited by 135
Toneva, M., & Wehbe, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11833 · Cited by 127
“The same chapter was shown by Wehbe et al. (2014a) to 3 subjects in MEG with the same rate of 0.5 seconds per word.”
BackgroundMethodInfluentialSelf-citationThe neural architecture of language: Integrative modeling converges on predictive processing
(2020) · doi:10.1101/2020.06.26.174482 · Cited by 129
“For example, several earlier studies used simple embedding models to decode linguistic meaning from fMRI data (e.g., Wehbe et al., 2014; Huth et al., 2016; Anderson et al., 2017; Pereira et al., 2018) and reported reliable decoding not only within the language network, but also across other parts…”
MethodFrank, S. L., & Willems, R. M. (2017) · Language Cognition and Neuroscience · doi:10.1080/23273798.2017.1323109 · Cited by 132
“In neuroimaging studies of language, there is a recent trend towards the use of naturalistic stimuli as opposed to hand-crafted experimental items (e.g. Wehbe et al., 2014; Willems, 2015).”
BackgroundMethodInfluentialLinking artificial and human neural representations of language
Gauthier, J., & Levy, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1050 · Cited by 70
“…representations as proxies for sentence meaning, and demonstrated that human brain activations in response to sentences can match with these meaning representations at well above chance performance (see e.g. Mitchell et al., 2008; Wehbe et al., 2014; Huth et al., 2016; Pereira et al., 2018).”
BackgroundCombining computational controls with natural text reveals aspects of meaning composition
Toneva, M., Mitchell, T. M., & Wehbe, L. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00354-6 · Cited by 97
“Though these NLP systems are not specifically designed to mimic the processing of language in the brain, representations of language extracted from these NLP systems have been shown to predict the brain activity of a person comprehending language better than ever before (Wehbe et al., 2014a; Jain and Huth, 2018; Tonev…”
BackgroundMethodSelf-citationNeurocomputational Models of Language Processing
Hale, J., Campanelli, L., Li, J., Bhattasali, S., Pallier, C., & Brennan, J. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-051421-020803 · Cited by 82
“Interestingly, Dikker & Pylkk¨anen find that noun predictability modulated MEG brain responses not only in fronto-temporal areas more directly associated with lexical access but also in lower level sensory areas like the visual cortex.”
BackgroundMethodInfluentialCogniVal: A Framework for Cognitive Word Embedding Evaluation
Hollenstein, N., de la Torre, A., Langer, N., & Zhang, C. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1050 · Cited by 98
“fM R I HARRY POTTER (Wehbe et al., 2014) text 8 5169 1295 92% ALICE (Brennan et al.”
BackgroundProbabilistic language models in cognitive neuroscience: Promises and pitfalls
Armeni, K., Willems, R. M., & Frank, S. L. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.09.001 · Cited by 75
Abnar, S., Beinborn, L., Choenni, R., & Zuidema, W. H. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4820 · Cited by 107
“…using brain data to evaluate computational models, or using NLP models to build predictive models of the human brain, or both (Murphy et al., 2012; Wehbe et al., 2014a; Ruan et al., 2016; Søgaard, 2016; Xu et al., 2016; Fyshe et al., 2014; Bingel et al., 2016; Bulat et al., 2017; Abnar et al.,…”
Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.12.02.403477 · Cited by 51
“These results agree with recent studies demonstrating that contextual embeddings better model neural responses to words than static semantic embeddings 18,19,48 .”
ResultDecoding EEG Brain Activity for Multi-Modal Natural Language Processing
Hollenstein, N., Renggli, C., Glaus, B., Barrett, M., Tröndle, M., Langer, N., & Zhang, C. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.659410 · Cited by 67
“The word representations used by neural networks and brain activity observed via the process of subjects reading a story can be aligned (Wehbe, Vaswani, Knight, & Mitchell, 2014).”
MethodLanguage processing in brains and deep neural networks: computational convergence and its limits
Caucheteux, C., & King, J.-R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.03.186288 · Cited by 47
Grounding the Vector Space of an Octopus: Word Meaning from Raw Text
Søgaard, A. (2023) · Minds and Machines · doi:10.1007/s11023-023-09622-4 · Cited by 46
“Other research articles that seem to suggest that language model representations are generally isomorphic to brain activity patterns include (Mitchell et al., 2008; Søgaard, 2016; Wehbe et al., 2014; Pereira et al., 2018; Gauthier & Levy, 2019; Caucheteux & King, 2022).”
BackgroundMultilingual Language Models Predict Human Reading Behavior
Hollenstein, N., Pirovano, F., Zhang, C., Jäger, L. A., & Beinborn, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.10 · Cited by 55
Lexical representation explains cortical entrainment during speech comprehension
Frank, S. L., & Yang, J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0197304 · Cited by 55
“holinguistics [11,12] and are not unrelated to cortical representations: They have been shown to allow for the decoding of neural activity during single word com2 prehension [13] or narrative reading [14,15], and distances between vectors are predictive of neural activation during written and spoken language comprehen…”
Computational Language Modeling and the Promise of In Silico Experimentation
Jain, S., Vo, V. A., Wehbe, L., & Huth, A. G. (2023) · Neurobiology of Language · doi:10.1162/nol_a_00101 · Cited by 71
“One recent example of such a high-dimensional feature space that captures semantic similarity (Mikolov et al., 2013; Pennington et al., 2014) is word embeddings , which have been used to characterize semantic language representations across the human brain (de Heer et al., 2017; Huth et al., 2016; Wehbe, Murphy, et al…”
BackgroundMethodInfluentialSelf-citationThe lexical semantics of adjective–noun phrases in the human brain
Fyshe, A., Sudre, G. P., Wehbe, L., Rafidi, N., & Mitchell, T. M. (2019) · Human Brain Mapping · doi:10.1002/hbm.24714 · Cited by 67
“Machine learning techniques have been used in a variety of settings to predict words from brain activity (Mitchell et al., 2008; Murphy, Talukdar, & Mitchell, 2012; Sudre et al., 2012; Wehbe et al., 2014; Wehbe, Vaswani, Knight, & Mitchell, 2014).”
BackgroundSelf-citationLatent representation learning in biology and translational medicine
Kopf, A., & Claassen, M. (2021) · Patterns · doi:10.1016/j.patter.2021.100198 · Cited by 50
“– neural network text word embeddings Wehbe et al.(77)”
BackgroundArmeni, K., Güçlü, U., van Gerven, M. A. J., & Schoffelen, J. (2022) · Scientific Data · doi:10.1038/s41597-022-01382-7 · Cited by 58
Keystroke dynamics as signal for shallow syntactic parsingPreprint
Plank, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03321 · Cited by 34
“Similarly, there are recent studies that predict fMRI activation from reading (Wehbe et al., 2014) or use fMRI data for POS induction (Bingel et al.”
BackgroundPREDICTION AND ERROR-BASED LEARNING IN L2 PROCESSING AND ACQUISITION: A CONCEPTUAL REVIEW
Bovolenta, G., & Marsden, E. J. (2021) · Studies in Second Language Acquisition · doi:10.1017/s0272263121000723 · Cited by 41
Deep learning models to study sentence comprehension in the human brain
Arana, S., Lerousseau, J. P., & Hagoort, P. (2023) · Language Cognition and Neuroscience · doi:10.1080/23273798.2023.2198245 · Cited by 37
“First, this result has been replicated using complete sentences rather than single words (Anderson et al., 2019; Huth et al., 2016; Millet & King, 2021; Millet et al., 2022; Pereira et al., 2018; Wehbe et al., 2014).”
InfluentialAn interpretable measure of semantic similarity for predicting eye movements in reading
Sun, K., Qiu-ying, W., & Lu, X. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02240-8 · Cited by 35
Testing the limits of natural language models for predicting human language judgements
Golan, T., Siegelman, M., Kriegeskorte, N., & Baldassano, C. A. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00718-1 · Cited by 34
“However, there is an ongoing effort to adopt and adapt neural-network language models to serve as computational hypotheses of how humans process language, making use of a variety of different architectures, training corpora and training tasks 11,27–35 .”
BackgroundRelating Simple Sentence Representations in Deep Neural Networks and the Brain
Jat, S., Tang, H., Talukdar, P. P., & Mitchell, T. M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1507 · Cited by 30
“We test correlations between brain activity and deep learning model activations [LeCun et al., 2015] for a given sentence using a classification task, similar to previous works [Mitchell et al., 2008; Wehbe et al., 2014a,b].”
MethodInfluentialSelf-citationComparing Transformers and RNNs on predicting human sentence processing data.Preprint
Merkx, D., & Frank, S. L. (2020) · arXiv (Cornell University) · Cited by 25
Inducing brain-relevant bias in natural language processing modelsPreprint
Schwartz, D., Toneva, M., & Wehbe, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03268 · Cited by 36
“The features we evaluate come from judgments done by Wehbe et al. (2014b) and are available online 4 .”
BackgroundMethodInfluentialSelf-citationAbdou, M., González, A. V., Toneva, M., Hershcovich, D., & Søgaard, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.12608 · Cited by 34
“In the second, more fine-grained dataset, recorded by Wehbe et al. (2014), each brain image corresponds to 4 words.”
BackgroundMethodInfluentialBarrett, M., & Hollenstein, N. (2020) · Language and Linguistics Compass · doi:10.1111/lnc3.12396 · Cited by 23
Neural Network Models of Language Acquisition and Processing
Frank, S. L., Monaghan, P., & Tsoukala, C. (2019) · The MIT Press eBooks · doi:10.7551/mitpress/10841.003.0026 · Cited by 29
Toward Computational Models of Multilingual Sentence Processing
Frank, S. L. (2020) · Language Learning · doi:10.1111/lang.12406 · Cited by 29
Understanding language-elicited EEG data by predicting it from a fine-tuned language model
Schwartz, D., & Mitchell, T. M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1005 · Cited by 27
“The authors in Wehbe et al. (2014) also use a recurrent neural network to predict neural activity directly.”
Self-citationProceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics
(2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4pvp2rzzk9vq · Cited by 17
Cao, L., Huang, D., Zhang, Y., Jiang, X., & Chen, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17493 · Cited by 27
Hollenstein, N., Barrett, M., & Beinborn, L. (2020) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 32
Bridging LSTM Architecture and the Neural Dynamics during ReadingPreprint
Qian, P., Qiu, X., & Huang, X. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06635 · Cited by 18
“In parallel with ⇤ Corresponding author. the fMRI experiment [Wehbe et al. , 2014a], we train a LSTM neural network and use it to generate the sequential representation of the same story.”
BackgroundMethodInfluentialAbnar, S., Beinborn, L., Choenni, R., & Zuidema, W. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01539 · Cited by 14
“…using brain data to evaluate computational models, or using NLP models to build predictive models of the human brain, or both (Murphy et al., 2012; Wehbe et al., 2014a; Ruan et al., 2016; Søgaard, 2016; Xu et al., 2016; Fyshe et al., 2014; Bingel et al., 2016; Bulat et al., 2017; Abnar et al.,…”
Mapping Brains with Language Models: A Survey
Karamolegkou, A., Abdou, M., & Søgaard, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.618 · Cited by 28
An eye-tracking-with-EEG coregistration corpus of narrative sentences
Frank, S. L., & Aumeistere, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09684-x · Cited by 26
“For example, in studies by Wehbe and colleagues, participants read one chapter of a Harry Potter book while their brain activity was recorded using functional magnetic resonance imaging (fMRI; Wehbe et al., 2014) or magnetoencephalography (MEG; Wehbe et al., 2014).”
Probing Multilingual BERT for Genetic and Typological Signals
Rama, T., Beinborn, L., & Eger, S. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.105 · Cited by 32
Studying language in context using the temporal generalization method
Fyshe, A. (2019) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2018.0531 · Cited by 17
“Though the human brain is certainly doing more than just predicting the next word in a sequence, several studies have found that the context vectors correlate well to the brain activity observed before the onset of a word [8,33].”
BackgroundRobust Evaluation of Language–Brain Encoding Experiments
Beinborn, L., Abnar, S., & Choenni, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-24337-0_4 · Cited by 20
Long-range and hierarchical language predictions in brains and algorithmsPreprint
Caucheteux, C., Gramfort, A., & King, J.-R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14232 · Cited by 25
“In line with previous studies (5, 7, 33, 34), the activations of GPT-2 accurately map onto a distributed and bilateral set of brain areas.”
ResultCogniVal: A Framework for Cognitive Word Embedding EvaluationPreprint
Hollenstein, N., de la Torre, A., Langer, N., & Zhang, C. (2019) · arXiv (Cornell University) · doi:10.3929/ethz-b-000394826 · Cited by 9
Can Brain Signals Reveal Inner Alignment with Human Languages?
Qiu, J., Han, W., Zhu, J., Xu, M., Weber, D. J., Li, B., & Zhao, D. X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.120 · Cited by 19
Same Cause; Different Effects in the Brain
Toneva, M., Williams, J. R., Bollu, A., Dann, C., & Wehbe, L. (2022) · PubMed · doi:10.48550/arxiv.2202.10376 · Cited by 19
“…et al., 2020; Hamilton and Huth, 2020), and to use complex representations from neural networks to build encoding models (Yamins et al., 2014; Wehbe et al., 2014b; Jain and Huth, 2018; Toneva and Wehbe, 2019; Wang et al., 2019; Schrimpf et al., 2020; Caucheteux et al., 2021b; Goldstein et…”
BackgroundMethodInfluentialSelf-citationCortical processing of reference in language revealed by computational modelsPreprint
Li, J., Wang, S., Luh, W., Pylkkänen, L., Yang, Y., & Hale, J. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.11.24.396598 · Cited by 7
Le Petit Prince : A multilingual fMRI corpus using ecological stimuliPreprint
Li, J., Bhattasali, S., Zhang, S., Franzluebbers, B., Luh, W., Spreng, R. N., Brennan, J., Yang, Y., Pallier, C., & Hale, J. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.10.02.462875 · Cited by 8
Evidence of syntactic working memory usage in MEG data
van Schijndel, M., Murphy, B., & Schuler, W. (2015) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.3115/v1/w15-1109 · Cited by 7
“This design allows us to examine language processing in a more ecologically realistic manner (Brennan et al., 2012; Wehbe et al., 2014a; Wehbe et al., 2014b), as both the participant experience (reading/listening to a story for enjoyment) and author’s aim are authentic language acts.”
BackgroundEmerging methods for conceptual modelling in neuroimaging
Akama, H., & Murphy, B. (2016) · Behaviormetrika · doi:10.1007/s41237-016-0009-1 · Cited by 7
“Approaches such as these are now starting to be applied to electrophysiological data also (Dmochowski et al 2012; Wehbe et al 2014b) which enables analyses at the temporal scale of natural language (in spontaneous speech we typically produce 3-4 words a second).”
BackgroundMethodLing, S., Murphy, A., & Fyshe, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00533 · Cited by 6
“…brain areas to corresponding timescales during comprehension of words (Mitchell et al. 2008; Huth et al. 2016), phrases (Fyshe et al. 2019), syntax and parts-of-speech (Hale et al. 2018; Murphy et al. 2022), sentence segments (Wehbe et al. 2014a), and complete sentences (Pereira et al. 2018).”
Li, C., Wang, S., Zhang, Y., Zhang, J., & Zong, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1026 · Cited by 19
Brain-inspired computing based on deep learning for human-computer interaction: A review
Yu, B., Zhang, S., Zhou, L., Wei, J., Sun, L., & Bu, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130928 · Cited by 8
The Role of Syntax During Pronoun Resolution: Evidence from fMRI
Li, J., Fabre, M., Luh, W., & Hale, J. (2018) · doi:10.18653/v1/w18-2808 · Cited by 6
“The Precuneus cortex may also be activated with pronouns in general as it has been suggested to track different sorts of story characters (Wehbe et al., 2014).”
BackgroundThe Semantics of Adjective Noun Phrases in the Human BrainPreprint
Fyshe, A., Sudre, G. P., Wehbe, L., Rafidi, N., & Mitchell, T. M. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/089615 · Cited by 6
“Such techniques have been used in a variety of settings to predict words from brain activity [15, 16, 17, 18, 19].”
BackgroundSelf-citationAttention-aware semantic relevance predicting Chinese sentence reading
Sun, K., & Liu, H. (2024) · Cognition · doi:10.1016/j.cognition.2024.105991 · Cited by 12
Single-Trial MEG Data Can Be Denoised Through Cross-Subject Predictive Modeling
Ravishankar, S., Toneva, M., & Wehbe, L. (2021) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2021.737324 · Cited by 6
“INTRODUCTION Naturalistic stimuli are becoming increasingly more common in cognitive neuroscience (Nishimoto et al., 2011; Wehbe et al., 2014a; Huth et al., 2016; Sonkusare et al., 2019; Hamilton and Huth, 2020; Nastase et al., 2020).”
BackgroundMethodSelf-citationBruera, A., & Poesio, M. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.796793 · Cited by 8
“MEG data, which grants higher machine learning performances given the superior signal quality, was instead used for decoding to word vectors from brain processing of pictures referring to concrete concepts (Sudre et al., 2012) and visually presented stories (Wehbe et al., 2014b).”
BackgroundMethodHashemzadeh, M., Kaufeld, G., White, M., Martin, A. E., & Fyshe, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.57 · Cited by 5
Branco, A., Rodrigues, J. A., Salawa, M., Branco, R., & Saedi, C. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.354 · Cited by 8
Bhattasali, S., & Resnik, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.332 · Cited by 10
“…studies demonstrating the role of those regions in broader contextual prediction during language comprehension: studies on narrative shifts (Whitney et al., 2009), contrasting sentences and narratives (Xu et al., 2005), contrasting coherent and incoherent narratives (Maguire et al., 1999),…”
BackgroundLanguage cognition and language computation — human and machine language understanding
王, 少., 丁, 鼐., 林, 楠., 张, 家., & 宗, 成. (2021) · Scientia Sinica Informationis · doi:10.1360/ssi-2021-0100 · Cited by 12
Role of Punctuation in Semantic Mapping Between Brain and Transformer Models
Lamprou, Z., Pollick, F. E., & Moshfeghi, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25891-6_35 · Cited by 8
Understanding language-elicited EEG data by predicting it from a fine-tuned language modelPreprint
Schwartz, D., & Mitchell, T. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01548 · Cited by 4
“The authors in Wehbe et al. (2014) also use a recurrent neural network to predict neural activity directly.”
Self-citationCheng, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03529 · Cited by 4
A unifying computational account of temporal context effects in language across the human cortexPreprint
Vo, V. A., Jain, S., Beckage, N., Chien, H.-Y. S., Obinwa, C., & Huth, A. G. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.08.03.551886 · Cited by 6
Tu, Z., Dai, L., Zhang, B., Chen, S., Yang, Y., Meng, D., Gong, Y., & Sun, J. (2025) · Brain and Language · doi:10.1016/j.bandl.2025.105569 · Cited by 2
When Computational Representation Meets Neuroscience: A Survey on Brain Encoding and Decoding
Cao, L., Huang, D., & Zhang, Y. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/594 · Cited by 5
“Based on this theory, subsequent studies extend the prediction tasks from concrete nouns to abstract nouns [Wang et al. , 2013], text fragments [Wehbe et al. , 2014a; Huth et al. , 2016], and sentences [Sun et al. , 2021].”
ResultBrain-Informed Fine-Tuning for Improved Multilingual Understanding in Language Models
Negi, A., Oota, S. R., Núñez-Elizalde, A. O., Gupta, M., & Deniz, F. (2025) · neural information processing systems · doi:10.52202/085713-5084 · Cited by 7
Modeling Brain Activity Associated with Pronoun Resolution in English and Chinese
Li, J., Fabre, M., Luh, W., & Hale, J. (2018) · Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC) · doi:10.18653/v1/w18-0710 · Cited by 4
Linking artificial and human neural representations of languagePreprint
Gauthier, J., & Lévy, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01244 · Cited by 13
“…representations as proxies for sentence meaning, and demonstrated that human brain activations in response to sentences can match with these meaning representations at well above chance performance (see e.g. Mitchell et al., 2008; Wehbe et al., 2014; Huth et al., 2016; Pereira et al., 2018).”
BackgroundAn interpretable semantic similarity for predicting eye movements in reading: Approach comparisonPreprint
Sun, K. (2022) · doi:10.31234/osf.io/jcp64 · Cited by 2
Divergences between Language Models and Human BrainsPreprint
Zhou, Y., Liu, E., Neubig, G., Tarr, M. J., & Wehbe, L. (2023) · PubMed Central · doi:10.48550/arxiv.2311.09308 · Cited by 5
“Building upon established research that demonstrates the capability of LM embeddings to linearly predict MEG responses [Wehbe et al., 2014a, Jain and Huth, 2018, Caucheteux and King, 2022a], we utilized a linear ridge regression model as the encoding model.”
InfluentialSelf-citationLinguistics and human brain: a perspective of computational neuroscience
Zhang, F., Chai, B., Wu, Y., Siok, W. T., & Wang, N. (2026) · Cognitive Neurodynamics · doi:10.1007/s11571-026-10496-2 · Cited by 7
“It enables comparison across models and examination of how model architecture, training scale, or input representations affect prediction performance (Wehbe et al. 2014; Toneva and Wehbe 2019).”
Quantifying Context With and Without Statistical Language Models
Jacobs, C. L. (2022) · Springer eBooks · doi:10.1007/978-3-031-03945-4_17 · Cited by 1
Machine Reading with Background KnowledgePreprint
Nakashole, N., & Mitchell, T. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.05348 · Cited by 1
Self-citationThe spatio-temporal representation of natural reading
Wehbe, L. (2015) · International Conference on Artificial Intelligence
Self-citationBrainprints: identifying individuals from magnetoencephalogramsPreprint
Wu, S., Ramdas, A., & Wehbe, L. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.18.159913
Self-citationJat, S., Laing, E. J. C., Talukdar, P. P., & Mitchell, T. M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.22.163808
“In addition, Deep Learning models trained on a large corpus of data are also used to provide rich stimulus features for ML models [14, 38, 29].”
MethodSelf-citationCogniFNN: A Fuzzy Neural Network Framework for Cognitive Word Embedding EvaluationPreprint
Liu, X., Cao, Z., & Tran, S. N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11485
“• fMRI - 4 data sources: HARRY POTTER (Wehbe et al., 2014), ALICE (Brennan et al., 2016), PEREIRA (Pereira et al., 2018), and NOUNS (Mitchell et al., 2008) were selected as fMRI datasets with 1000 voxels, and one scan of fMRI covers multiple words with continuous stimuli such as natural reading or storey listening.”
BackgroundMethodInfluentialQuantifying Context With and Without Statistical Language Models
Jacobs, C. L. (2021) · Springer eBooks · doi:10.1007/978-3-030-44982-7_17-1
The meaning emerging from combining words can be detected in space but not time
(2022) · Nature Computational Science · doi:10.1038/s43588-022-00361-7
Beinborn, L., & Hollenstein, N. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-43260-6_5
An interpretable semantic similarity for predicting eye movements in reading: Approach comparisonPreprint
Sun, K. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/jcp64_v1
Low-Rank Tensor Encoding Models Decompose Natural Speech Comprehension ProcessesPreprint
Lewis, L., Pitkow, X., & Wehbe, L. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.02.657514 · Cited by 1
“For modeling language responses, a popular and performant choice of features is to use a pretrained neural network language model [2, 5, 6, 7, 3], which host a battery of rich features useful for predicting a range of brain measurements including MEG [2, 6, 3].”
Self-citationTransforming AI models into useful model organisms
Toneva, M. (2026) · The Transmitter · doi:10.53053/vvef2936
Metadata
- Authors
- Wehbe, L., Vaswani, A., Knight, K., & Mitchell, T.
- Affiliations
- Carnegie Mellon University (US)
- University of Southern California (US)
- Venue
- Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Published
- 2014
- Pages
- 233-243
- OpenAlex
- W2170167574
- Cited by
- 95
- References
- 21
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 2 · 2023: 15 · 2022: 10 · 2021: 16 · 2020: 14 · 2019: 15 · 2018: 7 · 2017: 2 · 2016: 5 · 2015: 2
- FWCI
- 5.57
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- reading, context modeling, latent representations, word embeddings, magnetoencephalography, word predictability, word integration, memory retrieval, natural language processing models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.