Who “Is Saying What”? Brain-based decoding of human voice and speech
Open accessAbstract
Can we decipher speech content ("what" is being said) and speaker identity ("who" is saying it) from observations of brain activity of a listener? Here, we combine functional magnetic resonance imaging with a data-mining algorithm and retrieve what and whom a person is listening to from the neural fingerprints that speech and voice signals elicit in the listener's auditory cortex. These cortical fingerprints are spatially distributed and insensitive to acoustic variations of the input so as to permit the brain-based recognition of learned speech from unknown speakers and of learned voices from previously unheard utterances. Our findings unravel the detailed cortical layout and computational properties of the neural populations at the basis of human speech recognition and speaker identification.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
The cortical organization of speech processing
Hickok, G., & Poeppel, D. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2113 · Cited by 5,495 · see citing works
Distributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940 · see citing works
The TRACE model of speech perception
McClelland, J. L., & Elman, J. L. (1986) · Cognitive Psychology · doi:10.1016/0010-0285(86)90015-0 · Cited by 2,940 · see citing works
Information-based functional brain mapping
Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0600244103 · Cited by 2,113 · see citing works
Self-citationDecoding the visual and subjective contents of the human brain
Kamitani, Y., & Tong, F. (2005) · Nature Neuroscience · doi:10.1038/nn1444 · Cited by 1,829 · see citing works
Voice-selective areas in human auditory cortex
Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P. A., & Pike, G. B. (2000) · Nature · doi:10.1038/35002078 · Cited by 1,789 · see citing works
Human temporal lobe activation by speech and nonspeech sounds
Binder, J. R. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.5.512 · Cited by 1,345 · see citing works
Identifying natural images from human brain activity
Kay, K., Naselaris, T., Prenger, R., & Gallant, J. L. (2008) · Nature · doi:10.1038/nature06713 · Cited by 1,272 · see citing works
Language-specific phoneme representations revealed by electric and magnetic brain responses
Näätänen, R., Lehtokoski, A., Lennes, M., Cheour, M., Huotilainen, M., Iivonen, A., Vainio, M., Alku, P., Ilmoniemi, R. J., Luuk, A., et al. (1997) · Nature · doi:10.1038/385432a0 · Cited by 1,260 · see citing works
Thinking the voice: neural correlates of voice perception
Belin, P., Fecteau, S., & Bédard, C. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.01.008 · Cited by 908 · see citing works
Predicting the orientation of invisible stimuli from activity in human primary visual cortex
Haynes, J., & Rees, G. E. (2005) · Nature Neuroscience · doi:10.1038/nn1445 · Cited by 866 · see citing works
Hierarchical Processing in Spoken Language Comprehension
Davis, M. H., & Johnsrude, I. S. (2003) · Journal of Neuroscience · doi:10.1523/jneurosci.23-08-03423.2003 · Cited by 763 · see citing works
The neuroanatomical and functional organization of speech perception
Scott, S. K., & Johnsrude, I. S. (2003) · Trends in Neurosciences · doi:10.1016/s0166-2236(02)00037-1 · Cited by 704 · see citing works
Shortlist B: A Bayesian model of continuous speech recognition.
Norris, D. G., & McQueen, J. M. (2008) · Psychological Review · doi:10.1037/0033-295x.115.2.357 · Cited by 687 · see citing works
Martino, F. D., Valente, G., Staeren, N., Ashburner, J., Goebel, R., & Formisano, E. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.06.037 · Cited by 501 · see citing works
Self-citationNeural Substrates of Phonemic Perception
Liebenthal, E., Binder, J. R., Spitzer, S. M., Possing, E. T., & Medler, D. A. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhi040 · Cited by 448 · see citing works
Adaptation to speaker’s voice in right anterior temporal lobe
Belin, P., & Zatorre, R. J. (2003) · Neuroreport · doi:10.1097/00001756-200311140-00019 · Cited by 410 · see citing works
Sustained firing in auditory cortex evoked by preferred stimuli
Wang, X., Lu, T., Snider, R. K., & Li, L. (2005) · Nature · doi:10.1038/nature03565 · Cited by 370 · see citing works
Processing of complex stimuli and natural scenes in the auditory cortex
Nelken, I. (2004) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2004.06.005 · Cited by 223 · see citing works
Cortical activity patterns predict speech discrimination ability
Engineer, C. T., Perez, C., Chen, Y. H., Carraway, R. S., Reed, A., Shetake, J. A., Jakkamsetti, V., Chang, K. Q., & Kilgard, M. P. (2008) · Nature Neuroscience · doi:10.1038/nn.2109 · Cited by 201 · see citing works
Phoneme representation and classification in primary auditory cortex
Mesgarani, N., David, S. V., Fritz, J. B., & Shamma, S. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2816572 · Cited by 198 · see citing works
Vowel sound extraction in anterior superior temporal cortex
Obleser, J., Boecker, H., Drzezga, A., Haslinger, B., Hennenlotter, A., Roettinger, M., Eulitz, C., & Rauschecker, J. P. (2006) · Human Brain Mapping · doi:10.1002/hbm.20201 · Cited by 168 · see citing works
Multiple Stages of Auditory Speech Perception Reflected in Event-Related fMRI
Obleser, J., Zimmermann, J. B., van Meter, J., & Rauschecker, J. P. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhl133 · Cited by 167 · see citing works
Formisano, E., Martino, F. D., & Valente, G. (2008) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2008.01.052 · Cited by 138 · see citing works
Self-citationMagnetic Brain Response Mirrors Extraction of Phonological Features from Spoken Vowels
Obleser, J., Lahiri, A., & Eulitz, C. (2004) · Journal of Cognitive Neuroscience · doi:10.1162/089892904322755539 · Cited by 129 · see citing works
Left Posterior Temporal Regions are Sensitive to Auditory Categorization
Desai, R. H., Liebenthal, E., Waldron, E. J., & Binder, J. R. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2008.20081 · Cited by 120 · see citing works
Human brain mechanisms for the early analysis of voices
Warren, J. D., Scott, S. K., Price, C. J., & Griffiths, T. D. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.034 · Cited by 102 · see citing works
Orderly cortical representation of vowels based on formant interaction
Ohl, F. W., & Scheich, H. (1997) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.94.17.9440 · Cited by 88 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Prediction of individual brain maturity using fMRI
Dosenbach, N. U. F., Nardos, B., Cohen, A. L., Fair, D. A., Power, J. D., Church, J. A., Nelson, S. M., Wig, G. S., Vogel, A. C., Lessov-Schlaggar, C. N., et al. (2010) · Science · doi:10.1126/science.1194144 · Cited by 2,183 · see citing works
Phonetic Feature Encoding in Human Superior Temporal Gyrus
Mesgarani, N., Cheung, C., Johnson, K., & Chang, E. F. (2014) · Science · doi:10.1126/science.1245994 · Cited by 993 · see citing works
Decoding Neural Representational Spaces Using Multivariate Pattern Analysis
Haxby, J. V., Connolly, A. C., & Guntupalli, J. S. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062012-170325 · Cited by 717 · see citing works
Reconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604 · see citing works
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544 · see citing works
The free-energy self: a predictive coding account of self-recognition
Apps, M. A. J., & Tsakiris, M. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.01.029 · Cited by 514 · see citing works
Decoding Patterns of Human Brain Activity
Tong, F., & Pratte, M. S. (2012) · Annual Review of Psychology · doi:10.1146/annurev-psych-120710-100412 · Cited by 375 · see citing works
Brain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297 · see citing works
Nonstimulated early visual areas carry information about surrounding context
Smith, F. W., & Muckli, L. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1000233107 · Cited by 238 · see citing works
Beauchamp, M. S., Nath, A. R., & Pasalar, S. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4865-09.2010 · Cited by 240 · see citing works
Pei, X., Barbour, D. L., Leuthardt, E. C., & Schalk, G. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/4/046028 · Cited by 237 · see citing works
Blank, H., & Davis, M. H. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002577 · Cited by 181 · see citing works
Generative embedding for model-based classification of fMRI data
Brodersen, K. H., Schofield, T. M., Leff, A. P., Ong, C. S., Lomakina, E. I., Buhmann, J. M., & Stephan, K. E. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002079 · Cited by 165 · see citing works
Evans, S., & Davis, M. H. (2015) · Cerebral Cortex · doi:10.1093/cercor/bhv136 · Cited by 130 · see citing works
Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids
Ramsey, N. F., Salari, E., Aarnoutse, E. J., Vansteensel, M. J., Bleichner, M. G., & Freudenburg, Z. V. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.10.011 · Cited by 124 · see citing works
Distributed neural representations of phonological features during speech perception
Arsenault, J. S., & Buchsbaum, B. R. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.2454-14.2015 · Cited by 111 · see citing works
Expertise with artificial nonspeech sounds recruits speech-sensitive cortical regions
Leech, R., Holt, L. L., Devlin, J. T., & Dick, F. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.5758-08.2009 · Cited by 83 · see citing works
McGettigan, C., Evans, S., Rosen, S., Agnew, Z. K., Shah, P., & Scott, S. K. (2012) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00161 · Cited by 56 · see citing works
Hierarchy of speech-driven spectrotemporal receptive fields in human auditory cortex
Venezia, J. H., Thurman, S. M., Richards, V. M., & Hickok, G. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2018.11.049 · Cited by 47 · see citing works
Schelinski, S., Riedel, P., & von Kriegstein, K. (2014) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2014.09.031 · Cited by 37 · see citing works
Zevin, J. D., Yang, J., Skipper, J. I., & McCandliss, B. D. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4599-09.2010 · Cited by 36 · see citing works
Model-based feature construction for multivariate decoding
Brodersen, K. H., Haiss, F., Ong, C. S., Jung, F., Tittgemeyer, M., Buhmann, J. M., Weber, B., & Stephan, K. E. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.04.036 · Cited by 36 · see citing works
Mackey, S. C. (2013) · The Journal of Pain · doi:10.1016/j.jpain.2013.01.001 · Cited by 24 · see citing works
- Authors
- Formisano, E., De Martino, F., Bonte, M., & Goebel, R.
- Affiliations
- Maastricht University (NL)
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, University of Maastricht, 6200 MD Maastricht, Netherlands.
- Corresponding author
- Formisano, Elia
- Venue
- Science
- Published
- 2008-11-07
- Volume
- 322
- Issue
- 5903
- Pages
- 970-973
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1985969995
- Cited by
- 599
- References
- 28
- Citations by year
- 2026: 4 · 2025: 24 · 2024: 13 · 2023: 23 · 2022: 56 · 2021: 28 · 2020: 21 · 2019: 54 · 2018: 41 · 2017: 42 · 2016: 35 · 2015: 38 · 2014: 41 · 2013: 36 · 2012: 44
- FWCI
- 17.11
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neuroscience and Music Perception
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Keywords
- speech recognition, speaker identification, voice recognition, auditory cortex, neural decoding, neural fingerprints, functional MRI, data mining
- MeSH terms
- Adult, Algorithms, Auditory Cortex, Brain Mapping, Female, Humans, Magnetic Resonance Imaging, Male, Phonetics, Speech, Speech Perception, Voice, Pattern Recognition, Physiological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review