The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality

“Resulting p-values were corrected for multiple comparisons within each participant using the false discovery rate (FDR) procedure (Benjamini and Hochberg, 1995).”

“One participant was left handed, all other participants were right handed or ambidextrous according to the Edinburgh handedness inventory (Oldfield, 1971) (laterality quotient of 100: entirely left- handed, 100: entirely right-handed).”

“Semantic model construction To account for response variance caused by the semantic content of the story stimuli a 985-parameter semantic feature space based on word co-occurrence statistics in a large corpus of text (Deerwester et al., 1990; Lund and Burgess, 1996; Mitchell et al., 2008; Huth et al., 2016) was used.”

“Each functional run was motion-corrected using the FMRIB Linear Image Registration Tool (FLIRT) from FSL 5.0 (Jenkinson and Smith, 2001; Jenkinson et al., 2002).”

“Cortical surface reconstruction and visualization Cortical surface meshes were generated from the T1-weighted anatomical scans using Freesurfer software (Dale et al., 1999).”

“Banded ridge regression is a special case of the well-established statistical approach called Tikhonov regression (Tikhonov and Arsenin, 1977).”

“The contrast between faces and objects was used to define the fusiform face area (Kanwisher et al., 1997) and occipital face area (Halgren et al.”

“Each functional run was motion-corrected using the FMRIB Linear Image Registration Tool (FLIRT) from FSL 5.0 (Jenkinson and Smith, 2001; Jenkinson et al., 2002).”

“Although previous studies have demonstrated that most parts of cortex are active during language processing 5,22–24 , it is unclear which regions represent language at the granularity of words and phrases 25 , which regions are consistently engaged in language processing 26 and whether different…”

“Below we briefly summarize a body of evidence from cognitive science and neuroscience that supports this dissociation.”

“Second, the language areas are modality-independent: they respond to diverse kinds of input, including spoken, written and signed language 19,64–67 (Fig.”

“…Heusser et al., 2018), comparing natural language across modalities and contexts (Stephens et al., 2010; Regev et al., 2013; Yeshurun et al., 2017; Deniz et al., 2019), and using data-driven modeling to capture the complexity of naturalistic neural responses (Haxby et al., 2011, 2020; Baldassano…”

“1) that responds robustly and selectively to linguistic input (23, 24), whether auditory or visual (25, 26).”

“a memory probe task), and modality of presentation (visual and auditory: Fedorenko et al., 2010; Braze et al., 2011; Vagharchakian et al., 2012; Deniz et al., 2019).”

“25 combinatorial processing during comprehension, regardless of the input modality (e.g., Fedorenko et al., 2010, 2016; Vagharchakian et al., 2012; Regev et al., 2013; Scott et al., 2017; Deniz et al., 2019).”

“As additional effects, we modeled other variables that could modulate the BOLD response but that were not our critical variables of interest, including item (sentence), run order within a session (1-10), and sentence order within a run (1-50):”

“These findings are consistent with previous studies that mapped how different concepts in language are represented across cortex [30, 43].”

“These regions respond robustly to linguistic input, both visual and auditory (Deniz et al., 2019; Fedorenko et al., 2010; Nakai et al., 2020; Regev et al., 2013; Scott et al., 2017).”

“For example, the comprehension of narratives activates large areas of the cerebral cortex, and not simply those canonically viewed as ‘language’ areas (Deniz et al., 2019; Huth et al., 2016).”

“For example, according to the hub-and-spoke theory (Lambon Ralph et al., 2017; Patterson et al., 2007), the activated network includes both common areas (i.e., the ATL) and meaning-specific areas that are different for each word (see, e.g., Deniz et al., 2019; Huth et al., 2016).”

“The word-level semantic model uses a 985-dimensional conceptual feature space that is based on word co-occurrence statistics across a large corpus of written English(de Heer et al., 2017; Deniz et al., 2019; Huth et al., 2016).”

“To reap the benefits of both interpretable controlled experiments and generalizable naturalistic stimuli, some studies have deployed a hybrid experimental design (Chien & Honey, 2020; Deniz et al., 2019; Lerner et al., 2011; Overath et al., 2015; Yeshurun et al., 2017).”

“Note that this is similar to methods that directly compare the encoding model weights estimated for two voxels (Çukur et al., 2013; Huth et al., 2016; Deniz et al., 2019), but spatial generations looks not only at how much the weights are different, but also at how much this difference affects the ability to predict h…”

“These regions respond robustly to linguistic input (visual or auditory; Bemis & Pylkkänen, 2013; Deniz et al., 2019; Fedorenko et al., 2010) but show little or no response to tasks in non-linguistic domains, such as executive functions, math, logic, music, action observation, or non-linguistic…”

“Remarkable, it turned out that the representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality [4].”

“Given that most large public brain recordings datasets are of speech-evoked language (LeBel et al., 2023; Nastase et al., 2021; Deniz et al., 2019; Momenian et al., 2024), having access to speech models with improved brain-relevant semantics is important and will provide better model organisms for…”

“have already been used to expand the understanding of the neural basis for language processing, particularly for natural language comprehension [139, 140].”

“We also use the articulation and power spectrum (PowSpec) feature vectors provided in Deniz et al. (2019).”

“…in vision (Naselaris et al., 2011; Huth et al., 2012; Yamins et al., 2014; Eickenberg et al., 2017; Wen et al., 2018; Kubilius et al., 2019; Conwell et al., 2023; Wang et al., 2022), language (Huth et al., 2016; Deniz et al., 2019), and at the interface of vision and language (Popham et al., 2021).”

“Following others (Huth et al., 2016; Deniz et al., 2019; Shain et al., 2020), we align the word-times features X , of length M , to the dynamics of the fMRI signals applying a finite impulse response (FIR) model g (cf.”

“At each permutation iteration, the timecourse of the held-out test dataset was permuted by blockwise shuffling (shuffling was performed in blocks of 10 TRs in order to account for autocorrelations in voxel responses (Deniz et al., 2019; Jain et al., 2020)).”

“…assumed nature of comprehension is supported by findings from brain imaging research comparing listening and reading stories indicating that although sensory processes clearly differ by modality, the representations of the meaning in the verbal information are similar (Deniz et al., 2019).”

“At the same time, the representation of the language semantics in the brain is independent of whether meaning is extracted from spoken or written words [91].”

“…resemblance has been demonstrated between two sets of semantic representations recalled by spoken and written stories during language comprehension (Deniz et al., 2019), and the left anterodorsal pars triangularis (Brodmann area 45, BA 45) is significantly correlated with cross-modal semantic…”

“We choose a common value of 10TRs [25].”

“…effect binomial (semantic versus phonemic) or multinomial (single phonemes vs diphone vs triphone) statistical models in the main text and, just as for d 0 , this calculation is repeated with the “ equal weight" per subject and shown in the supplementary tables for Supplementary Table 2 and 5.”

“The DMN is known to be involved in narrative comprehension and interpretation (Deniz et al., 2019; Nguyen et al., 2019; Simony et al., 2016; Yeshurun et al., 2021).”

“Some previous studies using narratives and fMRI reported that supramodal activation was not restricted to the left hemisphere (Jobard et al., 2007; Regev et al., 2013; Deniz et al., 2019).”

“The advantage of decoding semantic information, as opposed to sensory based information such as visual details, is that semantic representation is partially invariant across modalities 9–13 .”

“, sentences and speech [14,21,26]; however, it was not applicable when the response is evoked by nonlinguistic stimuli, e.”

“…in the human brain has also been highlighted through datadriven approaches, by assessing brain responses to naturally spoken narrative stories (Deniz et al., 2019; Huth et al., 2016), further mapping single word-related semantic selectivity on the whole brain surface (see also Pereira et al.,…”

“These findings, which have benefited from more precise functional mapping approaches, link with prior work suggesting broad semantic coding within PCC/DMN [72,73].”

“Only when the connection of various representations reached a particular strength could mutual activation and diffusion be realized, and the output of orthography could be realized efficiently (Deniz et al., 2019).”

“This ‘language network’ supports both comprehension (spoken, written, and signed; e.g., Deniz et al., 2019, Fedorenko et al., 2010, MacSweeney et al., 2002, Regev et al., 2013, Scott et al., 2017) and production Menenti et al., 2011, Hu et al., 2022b.”

“…a, *, Matteo Visconti di Oleggio Castello a,b, *, Jack L. Gallant a,b Imaging Neuroscience, Volume 3, 2025 Wehbe et al., 2014), and narrative language ( Caucheteux & King, 2022; de Heer et al., 2017; Deniz et al., 2019; Huth et al., 2016; Jain & Huth, 2018; Jain et al., 2020; Toneva & Wehbe, 2019).”

“…language areas, responds to both written and auditory linguistic inputs (Fedorenko et al., 2010; Vagharchakian et al., 2012; Regev et al., 2013; Deniz et al., 2019), its computations plausibly have to do with the content of the linguistic signals rather than their surface (visual or auditory)…”

“In each permutation, the test responses were shuffled in blocks of 10 TRs (19, 44, 71, 81–85).”

“This method is similar to a variance-partitioning analysis which is often employed in encoding models, e.g. Deniz et al. [2019].”

“On the other hand, many parts of temporal, parietal, and prefrontal cortex process both written and spoken language (e.g., Booth et al., 2002; Buchweitz et al., 2009; Liuzzi et al., 2017; Regev et al., 2013; Deniz et al., 2019; Nakai et al., 2021).”

“The choice of these specific permutation test configurations is based on established methodologies in previous research (Deniz et al., 2019; Reddy & Wehbe, 2021; Oota et al., 2024a).”

“However, it is possible to ensure that text and audio are presented for a precisely equal time period to ensure faithful comparison (Deniz et al., 2019).”

“Future studies could employ tasks, such as category verification or story listening/reading (Huth et al., 2016; Deniz et al., 2019; Popham et al., 2021) that encourage deep processing of words and their context in modality-independent rather than modality-focused details.”

“Researchers have adopted this approach to comprehensively examine semantic representations (de Heer et al. 2017; Deniz et al. 2019; Huth et al. 2016), visual object category representations (Çukur et al. 2013a; Çukur et al. 2016; Çelik et al. 2019; Huth et al. 2012), and how attention modulates…”

“Other studies have probed correspondences between language processing models and the brain through the use of encoding models [2,3,6,48,50].”

“2012) that represent distinct features of stimuli (Huth et al. 2012; Deniz et al. 2019; Morton et al. 2021).”

“[26] Fatma Deniz, Anwar O Nunez-Elizalde, Alexander G Huth, and Jack L Gallant.”

“Existing studieshaveexploredneuralactivationpatternsbetweenspeechandread-ing,emphasizingdistinctionsbetweenunimodalversusmultimodal regions,andmodality-speci fi c effects in language comprehension 61 – 65 .”

“This is underpinned for example by a semantic model forming accurate predictions irrespective of modality (Deniz et al., 2019) and an MEG study showing word-speci fi c neural activity shared between modalities and subjects (Arana et al., 2020).”

“This region underpins both the comprehension (Deniz et al., 2019; Scott et al., 2017; Regev et al., 2013; Fedorenko et al., 2010) and production (Menenti et al., 2011; Hu et al., 2021) of language across spoken, written, and signed modalities.”

“Significance was evaluated for each network in each participant by comparing the N -1 group’s network-predicted activity to a null distribution 40,42 .”

“, 2016), regions within a distributed, bilateral semantic network display functional specialization for conceptual categories presented in narratives and this semantic selectivity appears regardless of sensory input modality (Deniz et al., 2019; Zhang et al., 2020).”

“Despite its old age, the EHI is still extensively used in the field of psychology, especially cognitive neuroscience, as a valid measure of handedness (Voss et al., 2010; Erickson et al., 2011; Stoodley et al., 2012; Barch et al., 2013; Frolov et al., 2017; Deniz et al., 2019).”

“…models can predict brain responses to written and spoken stimuli with high accuracy (Wehbe et al., 2014a; Jain & Huth, 2018; Toneva & Wehbe, 2019; Deniz et al., 2019; Abdou et al., 2021; Toneva et al., 2022; Antonello et al., 2021; Oota et al., 2022; Aw & Toneva, 2023; Oota et al., 2024b;…”

“Brain regions supporting language comprehension are engaged by both listening and reading (Regev et al., 2013; Deniz et al., 2019), as well as speaking and listening (Stephens et al., 2014; Zada et al., 2024, 2025).”

“Many recent studies have applied them during passive language comprehension (Caucheteux & King, 2022; de Heer et al., 2017; Deniz et al., 2019; Goldstein et al., 2022; Heilbron et al., 2022; Huth et al., 2016; Kumar et al., 2024; Schrimpf et al., 2021).”

“Indeed, neuroimaging studies show that hearing or reading a given description evokes identical semantic representations in the human brain (Arana et al., 2020; Deniz et al., 2019).”

“RSA is powerful for integrating different level/scale/ modality (e.g., neural, behavioral, physical, theoretical) activities to identify cognitive manipulation (FischerBaum et al., 2017; Wang et al., 2018; Deniz et al., 2019).”

“…sentences, the development and validation of the voxelwise encoding model framework for language processing (Huth et al., 2016; de Heer et al., 2017; Deniz et al., 2019; Popham et al., 2021) has made it possible to rigorously test hypotheses about semantic processing with natural language stimuli.”

“LLMs provide computational platforms for examining modern linguistic phenomena such as semantic integration, long-distance dependencies, and predictive processing.”

“…semantics to sentence meaning, ultimately culminating in a coherent narrative (Leaner et al., 2011; Regev et al., 2013; de Heer et al., 2017; Deniz et al., 2019; Popham et al., 2021, Deniz et al., 2023; Caucheteux et al., 2023; Huth et al., 2016; Schrimpf et al., 2021; Goldstein et al.,…”

“Specifically, variance partitioning separates the brain response variance that can be attributed to two models based on their unique and overlapping contributions (Vaidya et al., 2022; Deniz et al., 2019).”

“Stimulus-driven decompositions for interpretability have also been explored, in domains such as speech and audio [14, 42, 57] and visual stimuli [7, 17, 28].”

“These results are consistent with prior naturalistic investigations of semantic processing (Thye et al., 2023; Wu et al., 2022; Deniz et al., 2019; Huth et al., 2012, 2016) and with the language comprehension network identified across diverse languages, especially in the left hemisphere…”

“…knowledge relies on speci fi c hubs, but explain how semantic information integrates between brain areas and progresses from a modality-based towards a modality-independent conceptual representation (Deniz et al., 2019; Handjaras et al., 2016; Mahon & Caramazza, 2011; Popham et al., 2021).”

“However, in a study that compared brain activation between audio-only and text-based narratives, researchers found that the same parts of the brain were activated with both audio-only and reading books [46].”

“…(meaningless) and information with “value” has been discussed for a long time under the name “semantic” information [(Bar-Hillel and Carnap, 1953; Deniz et al., 2019; Lundgren, 2019; Miłkowski, 2021), see also emphasis on the requirement for recursive modeling to account for information in the…”

“…(as shown in Figure 2): (1) these vectors are resampled at times corresponding to the fMRI acquisitions using a three-lobe Lanczos filter [13, 24]; (2) vectors are reduced to d = 90 dimensions using PCA for computational efficiency; (3) finally, based on the linearized finite impulse…”

“This language network supports both comprehension (spoken, written, and signed) (Deniz et al., 2019; Terri L. Scott and Fedorenko, 2017) and production (Menenti et al., 2011; Hu et al., 2022).”

“We found that the regions in the semantic network spanned most of the bilateral semantic system, consistent with prior findings that bilateral activation was observed in response to complex stimuli, such as narratives (Deniz et al., 2019).”