Modality-Independent Decoding of Semantic Information from the Human Brain
Open accessAbstract
An ability to decode semantic information from fMRI spatial patterns has been demonstrated in previous studies mostly for 1 specific input modality. In this study, we aimed to decode semantic category independent of the modality in which an object was presented. Using a searchlight method, we were able to predict the stimulus category from the data while participants performed a semantic categorization task with 4 stimulus modalities (spoken and written names, photographs, and natural sounds). Significant classification performance was achieved in all 4 modalities. Modality-independent decoding was implemented by training and testing the searchlight method across modalities. This allowed the localization of those brain regions, which correctly discriminated between the categories, independent of stimulus modality. The analysis revealed large clusters of voxels in the left inferior temporal cortex and in frontal regions. These voxels also allowed category discrimination in a free recall session where subjects recalled the objects in the absence of external stimuli. The results show that semantic information can be decoded from the fMRI signal independently of the input modality and have clear implications for understanding the functional mechanisms of semantic memory.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When able speakers imagine speaking or hearing a word, neural activity in the STG can be used to decode those words with above-chance accuracy.
“Along these lines, when able speakers are instructed to imagine speaking or hearing a word, neural activity in the STG can be used to decode those words with above-chance accuracy59–63.”
From The speech neuroprosthesis (2024) · cites this paper as [63]
The Searchlight Results (p. 5) identify STG voxels that distinguish semantic categories when participants were presented with spoken or written words. But the Experimental Design (p. 3) describes covert recall of category exemplars, not instructions to imagine speaking or hearing a word; the paper reports decoding animal-versus-tool categories, not individual imagined words. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Nature of Statistical Learning Theory
Vapnik, V. (2000) · Statistics for Engineering and Information Science · doi:10.1007/978-1-4757-3264-1 · Cited by 46,387 · see citing works
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138 · see citing works
Oostenveld, R., Fries, P., Maris, E., & Schoffelen, J.-M. (2011) · Computational Intelligence and Neuroscience · doi:10.1155/2011/156869 · Cited by 10,323 · see citing works
Self-citationBrain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927 · see citing works
The cortical organization of speech processing
Hickok, G., & Poeppel, D. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2113 · Cited by 5,495 · see citing works
(2005) · Choice Reviews Online · doi:10.5860/choice.43-0626 · Cited by 4,726 · see citing works
Jackendoff, R. S. (2002) · doi:10.1093/acprof:oso/9780198270126.001.0001 · Cited by 4,472 · see citing works
Binder, J. R., Desai, R. H., Graves, W. W., & Conant, L. L. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhp055 · Cited by 4,060 · see citing works
A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data
Eickhoff, S. B., Stephan, K. E., Mohlberg, H., Grefkes, C., Fink, G. R., Amunts, K., & Zilles, K. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.034 · Cited by 4,033 · see citing works
Distributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940 · see citing works
Where do you know what you know? The representation of semantic knowledge in the human brain
Patterson, K., Nestor, P. J., & Rogers, T. T. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2277 · Cited by 2,706 · see citing works
The neurobiology of semantic memory
Binder, J. R., & Desai, R. H. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.10.001 · Cited by 2,114 · see citing works
Information-based functional brain mapping
Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0600244103 · Cited by 2,113 · see citing works
Decoding the visual and subjective contents of the human brain
Kamitani, Y., & Tong, F. (2005) · Nature Neuroscience · doi:10.1038/nn1444 · Cited by 1,829 · see citing works
Decoding mental states from brain activity in humans
Haynes, J., & Rees, G. E. (2006) · Nature reviews. Neuroscience · doi:10.1038/nrn1931 · Cited by 1,804 · see citing works
Frackowiak, R. S. J. (1997) · Cited by 1,713 · see citing works
Machine learning classifiers and fMRI: A tutorial overview
Pereira, F., Mitchell, T., & Botvinick, M. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.11.007 · Cited by 1,624 · see citing works
The Representation of Object Concepts in the Brain
Martin, A. D. (2006) · Annual Review of Psychology · doi:10.1146/annurev.psych.57.102904.190143 · Cited by 1,576 · see citing works
The anatomy of language: a review of 100 fMRI studies published in 2009
Price, C. J. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05444.x · Cited by 1,387 · see citing works
Semantic memory and the brain: structure and processes
Martin, A., & Chao, L. L. (2001) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(00)00196-3 · Cited by 1,339 · see citing works
Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects
Chao, L. L., Haxby, J. V., & Martin, A. D. (1999) · Nature Neuroscience · doi:10.1038/13217 · Cited by 1,246 · see citing works
Predicting Human Brain Activity Associated with the Meanings of Nouns
Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008) · Science · doi:10.1126/science.1152876 · Cited by 1,180 · see citing works
Functional anatomy of a common semantic system for words and pictures
Vandenberghe, R. R., Price, C. J., Wise, R., Josephs, O., & Frackowiak, R. S. J. (1996) · Nature · doi:10.1038/383254a0 · Cited by 1,172 · see citing works
Cox, D., & Savoy, R. L. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00049-1 · Cited by 1,057 · see citing works
Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus
Beauchamp, M. S., Lee, K. E., Argall, B., & Martin, A. D. (2004) · Neuron · doi:10.1016/s0896-6273(04)00070-4 · Cited by 765 · see citing works
Spatiotemporal dynamics of modality-specific and supramodal word processing
Marinkovic, K., Dhond, R. P., Dale, A. M., Glessner, M., Carr, V., & Halgren, E. (2003) · Neuron · doi:10.1016/s0896-6273(03)00197-1 · Cited by 520 · see citing works
Concepts and Categories: A Cognitive Neuropsychological Perspective
Mahon, B. Z., & Caramazza, A. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163532 · Cited by 512 · see citing works
Poser, B. A., Versluis, M. J., Hoogduin, J. M., & Norris, D. G. (2006) · Magnetic Resonance in Medicine · doi:10.1002/mrm.20900 · Cited by 440 · see citing works
Behavioral Neurology and Neuropsychology
WRIGHT, R. A., & Petersen, R. C. (1997) · Mayo Clinic Proceedings · doi:10.1016/s0025-6196(11)63558-1 · Cited by 387 · see citing works
Behavioral Neurology and Neuropsychology
Farah, M. J., & Feinberg, T. E. (1996) · Medical Entomology and Zoology · Cited by 377 · see citing works
Birn, R. M., Kenworthy, L., Case, L., Caravella, R., Jones, T. B., Bandettini, P. A., & Martin, A. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.07.036 · Cited by 377 · see citing works
Buckner, R. L. (2000) · Brain · doi:10.1093/brain/123.3.620 · Cited by 343 · see citing works
Distinct Cortical Pathways for Processing Tool versus Animal Sounds
Lewis, J. W., Brefczynski‐Lewis, J. A., Phinney, R. E., Janik, J. J., & DeYoe, E. A. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.0419-05.2005 · Cited by 331 · see citing works
The brain–computer interface cycle
van Gerven, M. A. J., Farquhar, J. D. R., Schaefer, R. S., Vlek, R. J., Geuze, J., Nijholt, A., Ramsey, N. F., Haselager, P., Vuurpijl, L. G., Gielen, S. C. A. M., et al. (2009) · Journal of Neural Engineering · doi:10.1088/1741-2560/6/4/041001 · Cited by 328 · see citing works
Moore, C. J., & Price, C. J. (1999) · Brain · doi:10.1093/brain/122.5.943 · Cited by 288 · see citing works
Interpreting fMRI data: maps, modules and dimensions
de Beeck, H. O., Haushofer, J., & Kanwisher, N. (2008) · Nature reviews. Neuroscience · doi:10.1038/nrn2314 · Cited by 279 · see citing works
Single-trial classification of vowel speech imagery using common spatial patterns
DaSalla, C. S., Kambara, H., Sato, M., & Koike, Y. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.05.008 · Cited by 274 · see citing works
Brain–computer interfaces for speech communication
Brumberg, J. S., Nieto-Castañón, A., Kennedy, P. R., & Guenther, F. H. (2010) · Speech Communication · doi:10.1016/j.specom.2010.01.001 · Cited by 267 · see citing works
Pobric, G., Jefferies, E., & Ralph, M. A. L. (2010) · Current Biology · doi:10.1016/j.cub.2010.03.070 · Cited by 262 · see citing works
Snijders, T. M., Vosse, T., Kempen, G., Berkum, J. J. A. V., Petersson, K. M., & Peter Hagoort, A. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhn187 · Cited by 249 · see citing works
Anatomic basis of transcortical motor aphasia
Freedman, M., ALEXANDER, M. P., & Naeser, M. A. (1984) · Neurology · doi:10.1212/wnl.34.4.409 · Cited by 238 · see citing works
Semantic systems or system? Neuropsychological evidence re-examined
Riddoch, M. J., Humphreys, G. W., Coltheart, M., & Funnell, E. (1988) · Cognitive Neuropsychology · doi:10.1080/02643298808252925 · Cited by 233 · see citing works
Semantics and the multisensory brain: How meaning modulates processes of audio-visual integration
Doehrmann, O., & Naumer, M. J. (2008) · Brain Research · doi:10.1016/j.brainres.2008.03.071 · Cited by 226 · see citing works
Dynamic aphasia: an inability to select between competing verbal responses?
Robinson, G., Blair, J., & Cipolotti, L. (1998) · Brain : a journal of neurology · doi:10.1093/brain/121.1.77 · Cited by 221 · see citing works
Experience-dependent Modulation of Category-related Cortical Activity
Chao, L. L. (2002) · Cerebral Cortex · doi:10.1093/cercor/12.5.545 · Cited by 219 · see citing works
A Neurosemantic Theory of Concrete Noun Representation Based on the Underlying Brain Codes
Just, M. A., Cherkassky, V. L., Aryal, S., & Mitchell, T. M. (2010) · PLoS ONE · doi:10.1371/journal.pone.0008622 · Cited by 211 · see citing works
Automatic Priming of Semantically Related Words Reduces Activity in the Fusiform Gyrus
Wheatley, T. P., Weisberg, J., Beauchamp, M. S., & Martin, A. D. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/089892905775008689 · Cited by 206 · see citing works
Functional Magnetic Resonance Imaging (fMRI)
Savoy, R. L. (2002) · Elsevier eBooks · doi:10.1016/b0-12-227210-2/00146-1 · Cited by 200 · see citing works
Two Distinct Neural Mechanisms for Category-selective Responses
Noppeney, U., Price, C. J., Penny, W. D., & Friston, K. J. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhi123 · Cited by 196 · see citing works
Unitary vs multiple semantics: PET studies of word and picture processing
Bright, P., Moss, H. E., & TYLER, L. K. (2004) · Brain and Language · doi:10.1016/j.bandl.2004.01.010 · Cited by 188 · see citing works
Wei, T., Liang, X., Yong He, A., Zang, Y., Han, Z., Caramazza, A., & Bi, Y. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.1953-11.2012 · Cited by 182 · see citing works
Shinkareva, S. V., Mason, R. A., Malave, V. L., Wang, W., Mitchell, T. M., & Just, M. A. (2008) · PLoS ONE · doi:10.1371/journal.pone.0001394 · Cited by 179 · see citing works
Commonality of neural representations of words and pictures
Shinkareva, S. V., Malave, V. L., Mason, R. A., Mitchell, T. M., & Just, M. A. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.10.042 · Cited by 159 · see citing works
Formisano, E., Martino, F. D., & Valente, G. (2008) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2008.01.052 · Cited by 138 · see citing works
Multisensory Identification of Natural Objects in a Two-Way Crossmodal Priming Paradigm
Schneider, T. R., Engel, A. K., & Debener, S. (2007) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1027/1618-3169.55.2.121 · Cited by 134 · see citing works
Cortical localisation of the visual and auditory word form areas: A reconsideration of the evidence
Price, C. J., Winterburn, D., Giraud, A., Moore, C. J., & Noppeney, U. (2003) · Brain and Language · doi:10.1016/s0093-934x(02)00544-8 · Cited by 122 · see citing works
Reddy, L., & Kanwisher, N. (2007) · Current Biology · doi:10.1016/j.cub.2007.10.043 · Cited by 108 · see citing works
Hauk, O., Davis, M. H., Kherif, F., & Friedemann Pulvermüller, A. (2008) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2008.06143.x · Cited by 103 · see citing works
A Comparison of Neural Circuits Underlying Auditory and Visual Object Categorization
Adams, R. B., & Janata, P. (2002) · NeuroImage · doi:10.1006/nimg.2002.1088 · Cited by 101 · see citing works
Hagoort, P., Baggio, G., & Willems, R. M. (2009) · The Cognitive Neurosciences · doi:10.7551/mitpress/8029.003.0072 · Cited by 99 · see citing works
Self-citationMandler, J. M. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.09.004 · Cited by 97 · see citing works
Cortical Plasticity of Audio-Visual Object Representations
Naumer, M. J., Doehrmann, O., Müller, N., Muckli, L. F., Kaiser, J., & Hein, G. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhn200 · Cited by 80 · see citing works
Tranel, D., Grabowski, T. J., Lyon, J., et al. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929055002508 · Cited by 76 · see citing works
Word Reading and Posterior Temporal Dysfunction in Amnestic Mild Cognitive Impairment
Vandenbulcke, M., Peeters, R. R., Dupont, P. J., Hecke, P. V., & Vandenberghe, R. R. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhj179 · Cited by 68 · see citing works
Category-related activation for written words in the posterior fusiform is task specific
Devlin, J. T., Rushworth, M. F. S., & Matthews, P. M. (2004) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2004.06.013 · Cited by 66 · see citing works
Probing category selectivity for environmental sounds in the human auditory brain
Doehrmann, O., Naumer, M. J., Volz, S., Kaiser, J., & Altmann, C. F. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2008.05.011 · Cited by 64 · see citing works
Neural correlates of naming animals from their characteristic sounds
Tranel, D., Damásio, H., Eichhorn, G. R., Grabowski, T. J., Ponto, L. L. B., & Hichwa, R. D. (2003) · Neuropsychologia · doi:10.1016/s0028-3932(02)00223-3 · Cited by 63 · see citing works
Kircher, T. T. J., Saß, K., Sachs, O., & Krach, S. (2009) · Human Brain Mapping · doi:10.1002/hbm.20833 · Cited by 51 · see citing works
The amodal system for conscious word and picture identification in the absence of a semantic task
Doren, L. V., Dupont, P. J., Grauwe, S. D., Peeters, R. R., & Vandenberghe, R. R. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.12.005 · Cited by 39 · see citing works
An Overview of Speech Comprehension and Production
(2004) · Elsevier eBooks · doi:10.1016/b978-012264841-0/50028-7 · Cited by 14 · see citing works
Mental encoding and neural decoding of abstract cognitive categories: A commentary and simulation
Vindiola, M. M., & Wolmetz, M. E. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.091 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Silva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103 · see citing works
“Along these lines, when able speakers are instructed to imagine speaking or hearing a word, neural activity in the STG can be used to decode those words with above-chance accuracy 59–63 .”
Kumar, M., Federmeier, K. D., Fei-Fei, L., & Beck, D. M. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.03.037 · Cited by 20 · see citing works
- Authors
- Simanova, I., Hagoort, P., Oostenveld, R., & van Gerven, M. A. J.
- Affiliations
- Radboud University Nijmegen (NL)
- Corresponding author
- Simanova, Irina
- ORCID
- Oostenveld, Robert
- Venue
- Cerebral Cortex
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-10-12
- Volume
- 24
- Issue
- 2
- Pages
- 426-434
- ISSN
- 1047-3211 (ISSN-L)
- 1460-2199
- OpenAlex
- W2159295653
- Cited by
- 138
- References
- 71
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 14 · 2023: 5 · 2022: 8 · 2021: 9 · 2020: 9 · 2019: 31 · 2018: 8 · 2017: 14 · 2016: 8 · 2015: 11 · 2014: 6 · 2013: 3
- FWCI
- 3.28
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Cerebral Cortex (publishedVersion)
- PubMed (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Keywords
- semantic decoding, semantic memory, semantic classification, object recognition, photographs, natural sounds, inferior temporal cortex, frontal cortex, free recall, brain-computer interface
- MeSH terms
- Adolescent, Adult, Auditory Perception, Brain, Brain Mapping, Female, Humans, Imagination, Judgment, Magnetic Resonance Imaging, Male, Names, Reading, Mental Recall, Semantics, Signal Processing, Computer-Assisted, Speech Perception, Visual Perception, Young Adult, Support Vector Machine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
