The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality

“Resulting p-values were corrected for multiple comparisons within each participant using the false discovery rate (FDR) procedure (Benjamini and Hochberg, 1995).”

“One participant was left handed, all other participants were right handed or ambidextrous according to the Edinburgh handedness inventory (Oldfield, 1971) (laterality quotient of 100: entirely left- handed, 100: entirely right-handed).”

“Semantic model construction To account for response variance caused by the semantic content of the story stimuli a 985-parameter semantic feature space based on word co-occurrence statistics in a large corpus of text (Deerwester et al., 1990; Lund and Burgess, 1996; Mitchell et al., 2008; Huth et al., 2016) was used.”

“Each functional run was motion-corrected using the FMRIB Linear Image Registration Tool (FLIRT) from FSL 5.0 (Jenkinson and Smith, 2001; Jenkinson et al., 2002).”

“Cortical surface reconstruction and visualization Cortical surface meshes were generated from the T1-weighted anatomical scans using Freesurfer software (Dale et al., 1999).”

“Banded ridge regression is a special case of the well-established statistical approach called Tikhonov regression (Tikhonov and Arsenin, 1977).”

“The contrast between faces and objects was used to define the fusiform face area (Kanwisher et al., 1997) and occipital face area (Halgren et al.”

“Each functional run was motion-corrected using the FMRIB Linear Image Registration Tool (FLIRT) from FSL 5.0 (Jenkinson and Smith, 2001; Jenkinson et al., 2002).”

“This is supported by a recent neuroimaging study [17].”

“Encoding model weights can reveal the latent directions of largest variance in the tuning to a stimulus-representation (Huth et al., 2012, 2016; Deniz et al., 2019) and even how tuning in the same zone is affected by a specific task (i.e., attending to humans) (Çukur et al., 2013).”

“The statistical significance of the prediction performance r for each voxel is determined by a permutation test with 10000 iterations (Deniz et al., 2019; Lescroart and Gallant, 2019; Jain et al., 2020; Ra-makrishnan and Deniz, 2021).”

“Naturalistic imaging tasks can include listening ( LeBel et al., 2023), movie viewing ( Finn & Bandettini, 2021; Vanderwal et al., 2019), reading ( Deniz et al., 2019; Kell et al., 2017; Woolnough et al., 2021), or even exploring a virtual reality environment ( Bulgarelli et al., 2023; Roberts et…”

“Previous studies using word embeddings or ANN-based features often assume that regression weights in these high-dimensional spaces are directly interpretable [22, 50, 54, 13].”

“As in previous works (Chen et al., 2024; Deniz et al., 2019), blocks of 10 temporally contiguous samples (corresponding to 20 s of data) were randomly permuted to account for the existing temporal autocorrelation in the data.”

“Recently [1], we proposed a well–grounded conjecture that a sentence—read or pronounced as the two activities are similarly processed by the brain [2]—is elaborated by the short–term memory (STM), with two independent processing units in series that have similar buffer size.”

“The here presented work wants to address the problem of mentally and physically exhausting silent speech BCI training procedures by introducing a transfer learning approach which makes use of the similarity between the concepts of silently reading and silently speaking words [7], [8].”

“Hence, language is a MS phenomenon involving high-level feature representations independent of the sensory modality (Fatma et al., 2019).”

“The voxelwise encoding modelling (VEM) framework is a powerful approach that has been validated in many previous experiments 79–82 .”

“Thesedurationswere chosentoapproximatenaturalreadingspeed 57 .”

“This suggests that oscillatory behavior is not a necessary consequence of shared architectural components (RoPE, GQA, or FlashAttention2), but rather may emerge from interactions between architecture and specific input statistics, as confirmed by the absence of oscillations when considering shorter contexts or on the…”

“However, these analyses of how semantics evolve through model layers did not test how these models align with the brain’s high-level late language regions, which are thought to be the endpoint of linguistic and speech processing in the brain [11].”

“Tools such as BrainBrowser [19], BrainGL [20], and BrainViewer [21] allow interactive exploration of structural and functional data, while Brain3X [22] enables real-time interaction with brain network dynamics.”

“Deniz et al. (2019): Modality-invariant semantic maps (their Figure 3) confirm that the category-region associations in Huth et al. generalize across listening and reading modalities, with concreteness in posterior temporal, affect in anterior temporal, and spatial relations in angular gyrus.”

“They found that “semantic representations do not differ between listening and reading” (Deniz et al. 2019, 7733).”

“In terms of language research, researchers have gradually begun to use computational modeling methods to study the language understanding process of the human brain under experimental stimulation conditions of natural texts [19, 20, 21, 22, 23, 24, 25, 26].”

“…Transformer-based language models exhibit a high degree of alignment with brain activity when participants engage in reading or listening naturalistic stories (Toneva and Wehbe, 2019; Deniz et al., 2019; Schrimpf et al., 2021; Millet et al., 2022; Vaidya et al., 2022; Caucheteux and King, 2022).”

“This observation aligns with previous research indicating that most language experiments with naturalistic stimuli reveal bilateral language representations [Wehbe et al., 2014b, Huth et al., 2016, Deniz et al., 2019, Toneva et al., 2022].”

“…in each region ( de Heer et al., 2017; Eickenberg et al., 2017; G üç l ü & van Gerven, 2015; LeBel et al., 2021; Wehbe et al., 2014) and identify regions that perform similar functions across different tasks ( C. Chen et al., 2026; Deniz et al., 2019; Popham et al., 2021; Tang, Du, et al., 2023).”

“Brain Data We used brain responses from prior work (Tuckute et al., 2025), in which eight proficient English speakers underwent ultra-high-field 7 Tesla (7T) fMRI while listening to 200 linguistically diverse sentences, each repeated three times in pseudorandomized order.”

“…are active during both production and comprehension, process written, spoken, and signed language, and respond to typologically diverse languages ( Deniz et al., 2019; Fedorenko et al., 2010; Malik-Moraleda, Ayyash et al., 2022; Menenti et al., 2011; Neville et al., 1998; Regev et al., 2013;…”

“While earlier findings supported the concept that hearing and reading language activate different brain networks (Buchweitz et al ., 2009), a more recent study showed that both spoken and written language activate the same regions overall (Deniz et al ., 2019).”

“Deniz et al [13] found that the representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modalities.”

“Specifically, variance partitioning separates the brain response variance that can be attributed to two models based on their unique and overlapping contributions (Vaidya et al., 2022; Deniz et al., 2019).”

“This observation aligns with previous research indicating that most language experiments with naturalistic stimuli reveal bilateral language representations (Wehbe et al., 2014a; Huth et al., 2016; Deniz et al., 2019; Toneva et al., 2022).”

“For example, using fMRI, Huth et al. (2016) reported differences in semantic tuning (preferences for linguistic input that express particular meanings) across the brain, including in frontal and temporal language areas (see also Deniz et al. 2019; Zhang et al. 2020).”

“Our findings are consistent with Deniz et al. (2019).”

“Voxelwise modeling (Naselaris et al., 2011; Nishimoto et al., 2011; Huth et al., 2012, 2016; Deniz et al., 2019; Lescroart and Gallant, 2019; Nunez-Elizalde et al., 2019) was used to quantify semantic tuning across the brain in each of the two experiments.”

“That work has shown that each voxel is preferentially selective for certain concepts, and that the cortical organization of concept selectivity is largely consistent across individuals (Huth et al. 2016; Deniz et al. 2019; Popham et al. 2021; Chen, Gong, et al. 2024).”

“The choice of these specific permutation test configurations is based on established methodologies in previous research (Deniz et al., 2019; Reddy & Wehbe, 2021; Oota et al., 2024).”

“Early studies established that static lexical embeddings can predict brain responses during both listening and reading (Huth et al. 2016; Deniz et al. 2019).”

“The fMRI loss is defined as: Where α is set to 0.1 during training, which balances absolute error minimization (via MSE) with representational alignment (via cosine similarity).”

“A recent study used pTx to improve image quality in ventral anterior temporal regions for echo-planar imaging (EPI) 7T fMRI (Ding et al. 2022). pTx improved tSNR across the brain compared to a standard sequence, particularly in the temporal lobes.”

“Moreover, based on previous fMRI studies targeting modality-independent semantic processing of words and narratives (Booth et al., 2002; Deniz et al., 2019; Wilson et al., 2018), we expected that this putative neural tracking of words in the auditory and visual modality involves overlapping regions…”

“Two recent neuroimaging studies [1,2] have decoded the structure and semantic content of static visual images from human brain activity.”

“160 seconds) of predicted fMRI data, similar to the permutation test used by Deniz et al.[19] to account for autocorrelation in voxel responses.”

“The second dataset, provided by Deniz et al. 2019, consists of brain recordings from six participants who read and listened to 11 stories from The Moth Radio Hour.”

“In neurologically healthy participants, concepts are encoded by distributed patterns of brain activity that are largely consistent across individuals 13 , stimulus modalities 20,21,29 , and languages 30 .”

“…system is likely optimised through lifelong experience for extracting semantic and discourse-level information under default conditions (e.g., Deniz et al., 2019; Swinney, 1979), and thus participants would have naturally read for meaning even without explicit instructions (i.e., defaulting…”

“…DeLong et al., 2005; Federmeier et al., 2007; Henderson et al., 2016; Shain et al., 2020; Heilbron et al., 2022) and to higher-level aspects of linguistic representation, including the construction of linguistic meaning (e.g., Huth et al., 2016; Deniz et al., 2019; Tuckute et al., 2024b, 2025).”

“Voxelwise modeling (Naselaris et al., 2011; Nishimoto et al., 2011; Huth et al., 2012, 2016; Deniz et al., 2019; Lescroart and Gallant, 2019; Nunez-Elizalde et al., 2019) was used to quantify visual-semantic tuning across the brain in each of the two experiments.”

“This fits with evidence that natural meaning is distributed across large cortical maps and that semantic representations can be partially shared across input formats while still retaining modality-and region-dependent structure 26,32,33 .”

“For instance, neural responses to semantic content (Deniz et al., 2019; Huth et al., 2016; Zhang et al., 2020) and syntactic complexity (Blank et al., 2016) are widely distributed across frontal and temporal language areas, but not confined to specific regions.”

“In one example, Honey and colleagues 5 found that in English-Russian bilinguals who listened to the same story in each language, neural responses were language-specific in early auditory areas, but activations in anterior temporal, parietal, and frontal cortices were shared across languages.”

“A number of studies have used text-based language models have been shown to predict both text-and speech-evoked brain responses with high fidelity (Jain & Huth, 2018; Deniz et al., 2019; Toneva & Wehbe, 2019; Caucheteux & King, 2022; Antonello et al., 2021; Oota et al., 2022a, 2024b).”

“Encoding models (EMs) have become a central tool for relating complex, naturalistic stimuli to brain activity, particularly in fMRI studies of language comprehension (Deniz et al., 2019; Huth et al., 2016; Wehbe et al., 2014).”

“Studenternas upplevelser av projektet har varit övervägande positiva och bekräftar de goda effekterna av högläsning och annan uppläst text som uppnåtts med barn och unga av Deniz et al. (2019), Mol och Bus (2011) samt Wedin (2009 och 2011).”

“…et al., 2011; Seeliger et al., 2019; Visconti di Oleggio Castello et al., 2020), and reading or listening to stories (de Heer et al., 2017; Deniz et al., 2019; Huth et al., 2016; Jain et al., 2020; Millet et al., 2022; Nastase et al., 2021; Schrimpf et al., 2021; Vaidya et al., 2022;…”

“Using fMRI recordings collected while participants listened to naturalistic stories from the Moth Radio Hour dataset (Deniz et al., 2019), we evaluate language models from three model families (LLaMA-3.”

“A 2019 fMRI study found no difference in how the brain processes written and aural texts (Deniz et al. 2019: 7733) and a large study of ninety-one college-educated adults found “comparable comprehension and retention of text regardless of the modality of presentation” (Rogowsky, Cal-houn, and…”

“The FIR model created time-shifted versions of each feature space at delays of 2, 4, 6, and 8 seconds to capture the hemodynamic response of individual voxels, following prior work (Huth et al. 2016; Deniz et al. 2019; Nakai et al. 2021).”

“So we apply a FIR model 1 on R , following previous work [11], [12].”

“This sample size is based on earlier studies using encoding models to model human language processing 19,25,73 that also used an intensive sampling strategy, extensively recording individual participant data in order to ensure sufficient training and testing data for within-participant encoding…”

“Prior fMRI work shows that visual and auditory semantics share a common neural representation [10, 15, 74, 75].”

“Many recent studies have applied them during passive language comprehension (Caucheteux & King, 2022; de Heer et al., 2017; Deniz et al., 2019; Goldstein et al., 2022; Heilbron et al., 2022; Huth et al., 2016; Kumar et al., 2024; Schrimpf et al., 2021).”

“Deniz et al. found that there is a shared neural basis for language meaning across modalities (Deniz et al. 2019).”

“…instance, increased access to audiobooks in particular may be useful for the development of literacy as it appears that “the semantic representations evoked by listening versus reading are almost identical” [18]; while musical perception skills supports phonological awareness and reading skills[7].”

“…that shared semantic information can be extracted from brain activity, consistent with previous studies reporting shared brain networks for modality-independent semantics processing (Regev et al. 2013; Deniz et al. 2019; Nakai, Yamaguchi, and Nishimoto 2021; Popham et al. 2021; Tang and Huth 2025).”

“Our goal was to fit encoding models that map semantic information to observed and predicted fMRI data (Naselaris et al. 2011, Deniz et al. 2019, Caucheteux et al. 2023).”

“Many models of language processing assume that input modality primarily a ff ects unimodal processing stages, while semantic and syntactic processes are heteromodal (Friederici, 2011; Price, 2012; Carreiras et al., 2014; Deniz et al., 2019).”

“…used to investigate cortical distributed representations of semantic information associated with thousands of words for data-driven approaches (Deniz et al., 2019; Güçlü and van Gerven, 2015; Huth et al., 2012, 2016; Matsumoto et al., 2022; Nishida et al., 2020a, 2021; Popham et al., 2021;…”

“- Phoneme rate and presence of phonemes [10, 14, 18].”

“Previous research has shown that the left posterior fusiform gyrus is much involved in visual word form processing after the basic visual processes (McCandliss, Cohen, and Dehaene, 2003; Dehaene & Cohen, 2011) and that both listening and reading comprehension is supported by widely distributed brain regions in the tem…”

“A 2019 study therefore represented a major breakthrough as the researchers looked at the brain’s reaction to whole stories rather than words or sentences (Deniz et al , 2019).”