Reconstructing Speech from Human Auditory Cortex
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





![Figure 6 Schematic of nonlinear modulation model. (A) The input spectrogram (top left) is transformed by a linear modulation filter bank (right) followed by a nonlinear magnitude operation (not shown). This nonlinear operation extracts the modulation energy of the incoming spectrogram and generates phase invariance to local fluctuations in the spectrogram envelope. The input representation is the two-dimensional spectrogram S(f,t) across frequency f and time t. The output (bottom left) is the four-dimensional modulation energy representation M(s,r,f,t) across spectral modulation scale s, temporal modulation rate r, frequency f, and time t. In the full modulation representation [18], negative rates by convention correspond to upward frequency sweeps, while positive rates correspond to downward frequency sweeps. Accuracy for positive and negative rates was averaged unless otherwise shown. See Materials and Methods. (B) Schematic of linear (spectrogram envelope) and nonlinear (modulation energy) temporal coding. Left: acoustic waveform (black curve) and spectrogram of a temporally modulated tone. The linear spectrogram model (top) assumes that neural responses are a linear function of the spectrogram envelope (plotted for the tone center frequency channel, top right). In this case, the instantaneous output may be high or low and does not directly indicate the modulation rate of the envelope. The nonlinear modulation model (bottom) assumes that neural responses are a linear function of modulation energy. This is an amplitude-based coding scheme (plotted for the peak modulation channel, bottom right). The nonlinear modulation model explicitly estimates the modulation rate by taking on a constant value for a constant rate [32].](https://figures.cochat.ai/paper-figures/d8911f10b353486dba673852c364a006/reconstructing-speech-from-human-auditory-cortex-figure-6.jpg)
![Figure 7 Example of nonlinear modulation coding and reconstruction. (A) Top: the spectrogram of an isolated word (“waldo”) presented aurally to one participant. Blue curve plots the spectrogram envelope, summed over all frequencies. Left panels: induced high gamma responses (black curves, trial averaged) at four different STG sites. Temporal modulation energy of the stimulus (dashed red curves) is overlaid (computed from 2, 4, 8, and 16 Hz modulation filters and normalized to maximum value). Dashed black lines indicate baseline response level. Right panels: nonlinear modulation rate tuning curves for each site (estimated from nonlinear STRFs). Shaded regions and error bars indicate SEM. (B) Original spectrogram (top), modulation-based reconstruction (middle), and spectrogram-based reconstruction (bottom), linearly decoded from a fixed set of STG electrodes. The modulation reconstruction is projected into the spectrogram domain using an iterative projection algorithm and an overcomplete set of modulation filters [18]. The displayed spectrogram is averaged over 100 random initializations of the algorithm.](https://figures.cochat.ai/paper-figures/2d2eaf31d85c437bb041f38dd7527a14/reconstructing-speech-from-human-auditory-cortex-figure-7.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Neural activity in the STG encodes acoustic features — for example, recurring speech sound patterns in the mel-spectrogram — during speech perception53,54 and may be used to reconstruct perceived speech55,56.”
From The speech neuroprosthesis (2024) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex
Hubel, D. H., & Wiesel, T. N. (1962) · The Journal of Physiology · doi:10.1113/jphysiol.1962.sp006837 · Cited by 13,680 · cited 1×
“The modulation model therefore represents an auditory analog to the classical energy model of complex cells in the visual system [32–36], which are invariant to the spatial phase of visual stimuli.”
MethodFundamentals of speech recognition
Rabiner, L. R., & Juang, B.-H. F. (1993) · Cited by 7,680
The cortical organization of speech processing
Hickok, G., & Poeppel, D. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2113 · Cited by 5,495 · cited 2×
“For speech comprehension, it is hypothesized that intermediate and higher-order auditory areas extract or construct information-rich features of speech, while discarding nonessential low-level acoustic information [4,5,9, 10,43].”
BackgroundSpatiotemporal energy models for the perception of motion
Adelson, E. H., & Bergen, J. R. (1985) · Journal of the Optical Society of America A · doi:10.1364/josaa.2.000284 · Cited by 3,442 · cited 2×
“Such amplitude-based coding schemes are broadly referred to as ‘‘energy models’’ [32,33].”
BackgroundDISCONNEXION SYNDROMES IN ANIMALS AND MAN
GESCHWIND, N. (1965) · Brain · doi:10.1093/brain/88.2.237 · Cited by 3,403 · cited 5×
“To recover the acoustic waveform from the spectrogram, we used an iterative projection algorithm to estimate the phase [8].”
BackgroundMethodInfluentialTIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,508
Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing
Rauschecker, J. P., & Scott, S. K. (2009) · Nature Neuroscience · doi:10.1038/nn.2331 · Cited by 1,766 · cited 5×
“Beyond primary auditory areas, further processing in intermediate and higher-order auditory cortex likely results in additional stimulus transformations [5].”
BackgroundResultInfluentialLogothetis, N. K., & Wandell, B. A. (2004) · Annual Review of Physiology · doi:10.1146/annurev.physiol.66.082602.092845 · Cited by 1,568 · cited 1×
“This is consistent with robust speech-induced high gamma responses reported in previous intracranial studies [24–29] and with observed correlations between high gamma power and local spike rate [30].”
ResultIdentifying natural images from human brain activity
Kay, K. N., Naselaris, T., Prenger, R. J., & Gallant, J. L. (2008) · Nature · doi:10.1038/nature06713 · Cited by 1,286 · cited 2×
“An important application of this approach has also been demonstrated in the study of visual object recognition in the primate visual system [36,49,50].”
BackgroundMethod
Cited by
Items 201–300 of 694
Identifying the attended speaker using electrocorticographic (ECoG) signals
Dijkstra, K., Brunner, P., Gunduz, A., Coon, W. G., Ritaccio, A. L., Farquhar, J. D. R., & Schalk, G. (2015) · Brain-Computer Interfaces · doi:10.1080/2326263x.2015.1063363 · Cited by 35
Including Measures of High Gamma Power Can Improve the Decoding of Natural Speech From EEG
Synigal, S. R., Teoh, E. S., & Lalor, E. C. (2020) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.00130 · Cited by 34
Nourski, K. V., Steinschneider, M., Oya, H., Kawasaki, H., & Howard, M. A. (2014) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2014.03.006 · Cited by 34
“Quantitative analysis of the ERBP focused on the high gamma ECoG frequency band, which has been shown to be a sensitive and specific indicator of auditory cortical activation (Crone et al., 2001; Edwards et al., 2009; Steinschneider et al., 2008, 2011; Mesgarani and Chang, 2012; Pasley et al., 2012; Nourski et al., 20…”
Neural Tuning to Low-Level Features of Speech throughout the Perisylvian Cortex
Berezutskaya, J., Freudenburg, Z. V., Güçlü, U., van Gerven, M. A. J., & Ramsey, N. F. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.0238-17.2017 · Cited by 34
“Multiple previous studies have shown a relationship between the neural responses in posterior STG and the spectrotemporal features of sound (Scho¨nwiesner and Zatorre, 2009; Pasley et al., 2012; Martin et al., 2014; Santoro et al., 2014; Hullett et al., 2016).”
Incorporating behavioral and sensory context into spectro-temporal models of auditory encoding
David, S. V. (2017) · Hearing Research · doi:10.1016/j.heares.2017.12.021 · Cited by 34
Self-citationPrefrontal High Gamma in ECoG Tags Periodicity of Musical Rhythms in Perception and Imagination
Herff, S. A., Herff, C., Milne, A. J., Johnson, G. D., Shih, J. J., & Krusienski, D. J. (2020) · eNeuro · doi:10.1523/eneuro.0413-19.2020 · Cited by 34
“The present finding supports previous ECoG studies that highlight the importance of high-gamma activity in auditory processing (Leuthardt et al., 2011; Pei et al., 2011; Schalk and Leuthardt, 2011; Pasley et al., 2012; Potes et al., 2012; Sturm et al., 2014; Herff et al., 2015; see Cervenka et al., 2011).”
BackgroundResultInfluentialAhmadi, S. (2017) · Radboud Repository (Radboud University) · Cited by 34
Laurent, R., Barnaud, M.-L., Schwartz, J., Bessìère, P., & Diard, J. (2017) · Psychological Review · doi:10.1037/rev0000069 · Cited by 33
Decoding Speech for Understanding and Treating Aphasia
Pasley, B. N., & Knight, R. T. (2013) · Progress in brain research · doi:10.1016/b978-0-444-63327-9.00018-7 · Cited by 33
“(Reproduced from Pasley et al., 2012.)”
InfluentialSelf-citationA brain-to-text framework for decoding natural tonal sentences
Zhang, D., Wang, Z., Qian, Y., Zhao, Z., Liu, Y., Hao, X., Li, W., Lu, S., Zhu, H., Chen, L., et al. (2024) · Cell Reports · doi:10.1016/j.celrep.2024.114924 · Cited by 32
Spoken and Inner Speech-related EEG Connectivity in Different Spatial Direction
Kiroy, V. N., Бахтин, О. М., Krivko, E., Lazurenko, D. M., Aslanyan, E. V., Shaposhnikov, D. G., & Щербань, И. В. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103224 · Cited by 33
Derix, J., Iljina, O., Weiske, J., Schulze‐Bonhage, A., Aertsen, A., & Ball, T. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00383 · Cited by 32
“ORIGINAL RESEARCH ARTICLE published: 13 June 2014 doi: 10.3389/fnhum.2014.00383 From speech to thought: the neuronal basis of cognitive units in non-experimental, real-life communication investigated using ECoG”
BackgroundMethodInfluentialRepresentation of Instantaneous and Short-Term Loudness in the Human Cortex
Thwaites, A., Glasberg, B. R., Nimmo‐Smith, I., Marslen‐Wilson, W. D., & Moore, B. C. J. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00183 · Cited by 32
“While imaging studies suggest that loudness is represented by activation in the cortex (Hall et al., 2001; Langers et al., 2007; Röhl and Uppenkamp, 2012; Giordano et al., 2013), inferring when and where short-term and long-term loudness are constructed is challenging.”
BackgroundContribution of spiking activity in the primary auditory cortex to detection in noise
Christison-Lagay, K. L., Bennur, S., & Cohen, Y. E. (2017) · Journal of Neurophysiology · doi:10.1152/jn.00521.2017 · Cited by 32
“Similarly, population codes have been shown to be important in the encoding of vocalizations and speech sounds (Chang et al. 2010; Engineer et al. 2008; Fishman et al. 2016; Ince et al. 2013; Mizrahi et al. 2014; Pasley et al. 2012; Russ et al. 2008a; Schneider and Woolley 2013).”
BackgroundAngrick, M., Herff, C., Johnson, G. D., Shih, J. J., Krusienski, D. J., & Schultz, T. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.10.080 · Cited by 32
Das, N., Vanthornhout, J., Francart, T., & Bertrand, A. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116211 · Cited by 32
EEG-Based Decoding of Auditory Attention to a Target Instrument in Polyphonic Music
Cantisani, G., Essid, S., & Richard, G. (2019) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa.2019.8937219 · Cited by 33
“In the case of magnitude and Mel spectrograms where the features correspond to frequency bins, the underlying assumption is that a spectrogram is a time-varying representation of the amplitude envelope at each frequency bin [5].”
BackgroundJoint Spatial-Spectral Feature Space Clustering for Speech Activity Detection from ECoG Signals
Kanas, V. G., Mporas, I., Benz, H. L., Sgarbas, K., Bezerianos, A., & Crone, N. E. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2298897 · Cited by 32
Self-citationAttention to natural auditory signals
Bluvas, E. C., & Gentner, T. Q. (2013) · Hearing Research · doi:10.1016/j.heares.2013.08.007 · Cited by 31
“In line with these results, attending to speech, as opposed to non-linguistic stimuli, activates regions such as the superior temporal sulcus and gyri that are known to be involved in language processing (Alho et al., 2006; Belin et al., 2000; Hari et al., 1989; Pasley et al., 2012; Rämä et al., 2012), often more prom…”
EEG-based detection of the locus of auditory attention with convolutional neural networks
Vandecappelle, S., Deckers, L., Das, N., Ansari, A. H., Bertrand, A., & Francart, T. (2019) · Lirias (KU Leuven) · Cited by 30
Spoken sentences decoding based on intracranial high gamma response using dynamic time warping
Zhang, D., Gong, E., Wu, W., Lin, J., Zhou, W., & Hong, B. (2012) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2012.6346668 · Cited by 30
Decoding the memorization of individual stimuli with direct human brain recordings
van Gerven, M. A. J., Maris, E., Sperling, M. R., Sharan, A. D., Litt, B., Anderson, C. T., Baltuch, G. H., & Jacobs, J. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2012.12.059 · Cited by 30
“, 2010), and perceived words (Kellis et al., 2010; Pasley et al., 2012).”
Decoding the perception of endogenous pain from resting-state MEG
Kuo, P., Chen, Y.-T., Chen, Y.-S., & Chen, L.-F. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.09.040 · Cited by 30
“Many recent studies have investigated the use of non-invasive techniques for the decoding of neuroimaging data related to visual or auditory perception (Formisano et al., 2008; Kuo et al., 2014; Nishimoto et al., 2011; Pasley et al., 2012).”
Haider, C. L., Suess, N., Hauswald, A., Park, H., & Weisz, N. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119044 · Cited by 30
“On the other hand, it also offers the possibility of assigning low weights to channels that are not particularly important in the reconstruction process, while still including channels that can capture additional variance (Pasley et al., 2012).”
BackgroundSmith, E. H., Duede, S., Hanrahan, S. J., Davis, T. S., House, P. A., & Greger, B. E. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073148 · Cited by 30
“To examine this idea, we used a conditional information theoretic analysis [23,24] to determine the transfer of information about visual or auditory stimulus identity among the three STG electrodes (Fig.”
BackgroundInfluentialCorrespondence of categorical and feature‐based representations of music in the human brain
Nakai, T., Koide‐Majima, N., & Nishimoto, S. (2020) · Brain and Behavior · doi:10.1002/brb3.1936 · Cited by 30
“…auditory pathways (de Heer et al., 2017; Saenz & Langers, 2014), but modulation-selective responses have been detected in the primary auditory cortex in ferrets (Depireux et al., 2001) and humans (Hullett et al., 2016; Langers et al., 2003; Pasley et al., 2012; Schonwiesner & Zatorre, 2009).”
BackgroundThe time‐course of speech perception revealed by temporally‐sensitive neural measures
Getz, L. M., & Toscano, J. C. (2020) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1541 · Cited by 29
Decoding visual object categories from temporal correlations of ECoG signals
Majima, K., Matsuo, T., Kawasaki, K., Kawai, K., Saito, N., Hasegawa, I., & Kamitani, Y. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.12.020 · Cited by 29
“The combination of ECoG recordings and neural decoding techniques has become a powerful approach to the investigation of neural representations in the last decade (Chao et al., 2010; Hammer et al., 2013; Liu et al., 2009; Pasley et al., 2012; Tsuchiya et al., 2008; Yanagisawa et al., 2009).”
BackgroundLakretz, Y., Ossmy, O., Friedmann, N., Mukamel, R., & Fried, I. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117499 · Cited by 28
Invasive Computational Psychiatry
Sáez, I., & Gu, X. (2022) · Biological Psychiatry · doi:10.1016/j.biopsych.2022.09.032 · Cited by 29
Classification of speech-evoked brainstem responses to English vowels
Sadeghian, A., Dajani, H. R., & Chan, A. D. C. (2015) · Speech Communication · doi:10.1016/j.specom.2015.01.003 · Cited by 28
Low-Frequency Oscillations Code Speech during Verbal Working Memory
Gehrig, J., Michalareas, G., Forster, M.-T., Lei, J., Hok, P., Laufs, H., Senft, C., Seifert, V., Schoffelen, J., Hanslmayr, S., et al. (2019) · Journal of Neuroscience · doi:10.1523/jneurosci.0018-19.2019 · Cited by 28
“…the encoding to occur in the phase variations, rather than in the power of low-frequency oscillations (Luo and Poeppel, 2007; Mesgarani and Chang, 2012; Golumbic et al., 2013), the phase TPSim analysis was performed on phase variation similarities between trials rather than power…”
BackgroundMethodInfluentialStrong and weak phonological patterns: How the transmission of meaning shapes phonologyPreprint
Wedel, A. B., Hall, K. C., Jaeger, T. F., & Hume, E. V. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/sbyqk · Cited by 28
Cortical Tracking of Sung Speech in Adults vs Infants: A Developmental Analysis
Attaheri, A., Panayiotou, D., Phillips, A., Choisdealbha, Á. N., Liberto, G. M. D., Rocha, S., Brusini, P., Mead, N., Flanagan, S. A., Olawole‐Scott, H., et al. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.842447 · Cited by 27
“Similarly, in adult studies theta band cortical tracking has been related to syllable parsing and speech intelligibility (Luo and Poeppel, 2007; Peelle et al., 2013; Millman et al., 2015; Baltzell et al., 2017).”
BackgroundAuditory attention decoding from EEG-based Mandarin speech envelope reconstruction
Xu, Z., Bai, Y., Zhao, R., Zheng, Q., Ni, G., & Ming, D. (2022) · Hearing Research · doi:10.1016/j.heares.2022.108552 · Cited by 27
“The regions mainly cover the superior temporal gyrus, prefrontal, and parietal lobes, consistent with the previous study ( Hamilton et al., 2021 ; Mesgarani and Chang, 2012 ; Upadhyay et al., 2008 ).”
InfluentialCausal evidence for a coordinated temporal interplay within the language network
Schroën, J. A. M., Gunter, T. C., Numssen, O., Kroczek, L. O. H., Hartwigsen, G., & Friederici, A. D. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2306279120 · Cited by 27
Kim, H., & Bao, S. (2013) · Journal of Neurophysiology · doi:10.1152/jn.00230.2013 · Cited by 27
Repeated stimuli elicit diminished high-gamma electrocorticographic responses
Merzagora, A. R., Coffey, T. J., Sperling, M. R., Sharan, A. D., Litt, B., Baltuch, G. H., & Jacobs, J. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.07.006 · Cited by 27
General principles of machine learning for brain-computer interfacing
Iturrate, I., Chavarriaga, R., & Millán, J. D. R. (2020) · Handbook of clinical neurology · doi:10.1016/b978-0-444-63934-9.00023-8 · Cited by 26
Decoding Multiple Sound-Categories in the Auditory Cortex by Neural Networks: An fNIRS Study
Yoo, S.-H., Santosa, H., Kim, C., & Hong, K. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.636191 · Cited by 26
“, 2015), eletrocorticogram (ECoG; Pasley et al., 2012; Herff et al., 2015), functional magnetic resonance imaging (fMRI; Wong et al.”
BackgroundMai, G., & Wang, W. Y. (2023) · Human Brain Mapping · doi:10.1002/hbm.26503 · Cited by 27
“This is because previous neurophysiological and fMRI studies have shown that cortical responses can be well-modelled using spectrotemporal modulations (Daube et al., 2019; Pasley et al., 2012; Santoro et al., 2014; Sohoglu & Davis, 2020).”
BackgroundResultEEG-assisted modulation of sound sources in the auditory scene
Haghighi, M., Moghadamfalahi, M., Akçakaya, M., & Deniz Erdogmus, B. (2017) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2017.08.008 · Cited by 26
“In the identification of the attended sound source in a cocktail party problem, stimulus reconstruction to estimate the envelope of the input speech stream from high density EEG measurements is the state-of-the-art practice [25], [22].”
BackgroundBlack, J. M., Xia, Z., & Hoeft, F. (2017) · Language and Linguistics Compass · doi:10.1111/lnc3.12252 · Cited by 26
Learning Speech Production and Perception through Sensorimotor Interactions
Shamma, S., Patel, P., Mukherjee, S., Marion, G., Khalighinejad, B., Han, C., Herrero, J. L., Bickel, S., Mehta, A. D., & Mesgarani, N. (2020) · Cerebral Cortex Communications · doi:10.1093/texcom/tgaa091 · Cited by 25
“Stimulus reconstruction decoders were calculated (Mesgarani et al. 2009; Pasley et al. 2012) using custom code to implement”
MethodSelf-citationDifferential Processing of Consonance and Dissonance within the Human Superior Temporal Gyrus
Foo, F., King‐Stephens, D., Weber, P. B., Laxer, K. D., Parvizi, J., & Knight, R. T. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00154 · Cited by 25
“Additionally, key features in speech sounds have been accurately reconstructed using γhigh activations in lateral STG (Pasley et al., 2012).”
BackgroundSelf-citationEffect of number and placement of EEG electrodes on measurement of neural tracking of speechPreprint
Montoya-Martínez, J., Vanthornhout, J., Bertrand, A., & Francart, T. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/800979 · Cited by 25
Venezia, J. H., Martin, A.-G., Hickok, G. S., & Richards, V. M. (2019) · Journal of Speech Language and Hearing Research · doi:10.1044/2018_jslhr-h-18-0045 · Cited by 24
“From a neurobiological perspective, the response of cortical neurons and neuronal ensembles to speech and other complex sounds is well described in terms of STM tuning (Hullett, Hamilton, Mesgarani, Schreiner, & Chang, 2016; Pasley et al., 2012; Shamma, 2001).”
BackgroundVisual objects in the auditory system in sensory substitution: How much information do we need?
Brown, D. J., Simpson, A. J. R., & Proulx, M. J. (2014) · Multisensory Research · doi:10.1163/22134808-00002462 · Cited by 23
Anderson, A. J., Davis, C., & Lalor, E. C. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012537 · Cited by 23
Similarity of Cortical Activity Patterns Predicts generalization Behavior
Engineer, C. T., Perez, C., Carraway, R. S., Chang, K. Q., Roland, J. L., Sloan, A. M., & Kilgard, M. P. (2013) · PLoS ONE · doi:10.1371/journal.pone.0078607 · Cited by 23
Real-time voice activity detection for ECoG-based speech brain machine interfaces
Kanas, V. G., Mporas, I., Benz, H. L., Sgarbas, K. N., Bezerianos, A., & Crone, N. E. (2014) · 2014 19th International Conference on Digital Signal Processing · doi:10.1109/icdsp.2014.6900790 · Cited by 20
“Several studies have made advances toward the development of effective motor [1]-[4] and speech prostheses [5]-[8] based on biological signals.”
Self-citationReconstruction of audio waveforms from spike trains of artificial cochlea models
Zai, A. T., Bhargava, S., Mesgarani, N., & Liu, S. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00347 · Cited by 23
Self-citationInvoluntary polymodal imagery involving olfaction, audition, touch, taste, and vision
Dou, W., Li, Y., Geisler, M. W., & Morsella, E. (2018) · Consciousness and Cognition · doi:10.1016/j.concog.2018.04.007 · Cited by 23
Heelan, C., Nurmikko, A. V., & Truccolo, W. (2018) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2018.8512415 · Cited by 23
“We envision the application of this system to decoding speech from auditory cortex [9], [10], for example.”
BackgroundMethodDetection and identification of speech sounds using cortical activity patterns
Centanni, T. M., Sloan, A. M., Reed, A., Engineer, C. T., Rennaker, R. L., & Kilgard, M. P. (2013) · Neuroscience · doi:10.1016/j.neuroscience.2013.11.030 · Cited by 22
“These patterns of activity can be used to identify the evoking auditory stimulus in both human (Steinschneider et al., 2005; Chang et al., 2010; Pasley et al., 2012) and animal auditory cortex (Engineer et al.”
Neural substrates of subphonemic variation and lexical competition in spoken word recognition
Luthra, S., Guediche, S., Blumstein, S. E., & Myers, E. B. (2018) · Language Cognition and Neuroscience · doi:10.1080/23273798.2018.1531140 · Cited by 22
“…even when individuals only listen to syllables, regions in the STG respond in a graded fashion to degradations in the input (Evans et al., 2014; Pasley et al., 2012), leading to the hypothesis that the STG should show preferential tuning (i.e. increased activation) for “good” versions of the…”
BackgroundDas, P., Brodbeck, C. J., Simon, J. Z., & Babadi, B. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116528 · Cited by 22
Tracking cortical entrainment in neural activity: auditory processes in human temporal cortex
Thwaites, A., Nimmo‐Smith, I., Fonteneau, E., Patterson, R. D., Buttery, P. J., & Marslen‐Wilson, W. D. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00005 · Cited by 22
“Lalor, E. C., Power, A. J., Reilly, R. B., and Foxe, J. J. (2009). Resolving precise temporal processing properties of the auditory system using continuous stimuli.”
BackgroundFujiwara, Y., Matsumoto, R., Nakae, T., Usami, K., Matsuhashi, M., Kikuchi, T., Yoshida, K., Kunieda, T., Miyamoto, S., Mima, T., et al. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.11.058 · Cited by 22
“…resonance imaging [fMRI]) are weighted and combined to decode perceptions and motor behaviors (e.g., movement direction or trajectory) (Taylor et al., 2002; Leuthardt et al., 2004; Kamitani and Tong, 2005; Yanagisawa et al., 2009; Bradberry et al., 2010; Pasley et al., 2012; Haar et al., 2015).”
Neural representation of vowel formants in tonotopic auditory cortex
Fisher, J. M., Dick, F., Levy, D. F., & Wilson, S. M. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.05.072 · Cited by 21
“, 2016) and intracranial electrocorticography (Chang et al., 2010; Pasley et al., 2012; Chan et al., 2014; Mesgarani et al., 2014; Leonard et al., 2016; Moses et al., 2016) have shown that phonemes can be reconstructed and discriminated by machine learning algorithms based on the activity of multiple voxels or electro…”
Cortical Auditory Attention Decoding During Music and Speech Listening
Simon, A., Loquet, G., Østergaard, J., & Bech, S. (2023) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2023.3291239 · Cited by 21
Constantino, F. C., & Simon, J. Z. (2018) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2018.00056 · Cited by 21
Sharon, R., Narayanan, S. S., Sur, M., & Murthy, H. A. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683572 · Cited by 21
“Previous studies of speech envelope reconstruction from EEG indicate the existence of speech signatures in EEG [12–14].”
Workshops of the eighth international brain–computer interface meeting: BCIs: the next frontier
Huggins, J. E., Krusienski, D. J., Vansteensel, M. J., Valeriani, D., Thelen, A., Stavisky, S. D., Norton, J. J. S., Nijholt, A., Müller-Putz, G., Kosmyna, N., et al. (2022) · Brain-Computer Interfaces · doi:10.1080/2326263x.2021.2009654 · Cited by 21
Comprehending auditory speech: previous and potential contributions of functional MRI
Evans, S., & McGettigan, C. (2017) · Language Cognition and Neuroscience · doi:10.1080/23273798.2016.1272703 · Cited by 21
“…of speech perception at a level of temporal and spatial precision that far exceeds fMRI (see Nourski et al., 2013, 2014 for auditory applications, and for speech see Chang et al., 2010; Leonard & Chang, 2014; Mesgarani & Chang, 2012; Mesgarani, Cheung, Johnson, & Chang, 2014; Pasley et al., 2012).”
BackgroundDecoding speech perception from single cell activity in humans
Ossmy, O., Fried, I., & Mukamel, R. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.05.001 · Cited by 21
“, 2010) and that slow and intermediate temporal fluctuations corresponding to syllable rate can be reconstructed based on power in high-gamma frequency band (Pasley et al., 2012).”
BackgroundStimulus Speech Decoding from Human Cortex with Generative Adversarial Network Transfer Learning
Wang, R., Chen, X., Khalilian-Gourtani, A., Chen, Z., Yu, L., Flinker, A., & Wang, Y. (2020) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi45749.2020.9098589 · Cited by 20
“We also compared with the linear regression model decoding accuracy reported by Pasley et al [4].”
MethodResultSelf-citationInvoluntary Entry Into Consciousness From the Activation of Sets: Object Counting and Color Naming
Bhangal, S., Merrick, C. M., Cho, H., & Morsella, E. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01017 · Cited by 20
A Neurophysiological Perspective on Speech Processing in “The Neurobiology of Language”
Arnal, L. H., Poeppel, D., & Giraud, A. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-407794-2.00038-9 · Cited by 20
Sleep deprivation detected by voice analysis
Thoret, E., Andrillon, T., Gauriau, C., Léger, D., & Pressnitzer, D. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011849 · Cited by 19
Impaired Cortical Tracking of Speech in Children with Developmental Language Disorder
Nora, A., Rinkinen, O., Renvall, H., Service, E., Arkkila, E. P., Smolander, S., Laasonen, M. R., & Salmelin, R. H. (2024) · Journal of Neuroscience · doi:10.1523/jneurosci.2048-23.2024 · Cited by 19
“In addition to displaying an organization by frequency, the primary and secondary auditory cortices respond to di ff erent rates of temporal modulations at di ff erent spectral scales (Pasley et al., 2012; Santoro et al., 2014, 2017).”
BackgroundCongruent audiovisual speech enhances auditory attention decoding with EEG
Fu, Z., Wu, X. H., & Chen, J. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab4340 · Cited by 19
Sufficient sampling for kriging prediction of cortical potential in rat, monkey, and human µECoG
Trumpis, M., Chiang, C., Orsborn, A. L., Bent, B., Li, J., Rogers, J. A., Pesaran, B., Cogan, G. B., & Viventi, J. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abd460 · Cited by 19
“The high signal amplitude and spatial precision resulting from direct cortical contact has provided neurophysiologists with an important tool for studying speech and skeletomotor systems [2, 3, 4, 5, 6, 7, 8, 9].”
Moon, J., Orlandi, S., & Chau, T. T. (2022) · Brain Research · doi:10.1016/j.brainres.2022.147778 · Cited by 19
Cooney, C., Folli, R., & Coyle, D. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104783 · Cited by 19
FMRI speech tracking in primary and non-primary auditory cortex while listening to noisy scenes
Hausfeld, L. G., Hamers, I. M. H., & Formisano, E. (2024) · Communications Biology · doi:10.1038/s42003-024-06913-z · Cited by 18
“…Previous research highlighted that spatial activation patterns across the (auditory) cortical surface represent auditory objects including speech streams and that these spatiotemporal representations are signi fi cantly affected by selective attention in multi-talker scenes 6,11,18 – 21 .”
A Joint Auditory Attention Decoding and Adaptive Binaural Beamforming Algorithm for Hearing Devices
Pu, W., Xiao, J., Zhang, T., & Luo, Z.-Q. T. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682514 · Cited by 18
Cortical neuroprosthetics from a clinical perspective
Tsu, A., Burish, M. J., Godlove, J. M., & Ganguly, K. (2015) · Neurobiology of Disease · doi:10.1016/j.nbd.2015.07.015 · Cited by 18
“…2014; Mugler et al., 2014; Pasley et al., 2012; Pei et al., 2011); more specifically, features of speech heard by a subject could be decoded from sensory cortical areas (Pasley et al., 2012) and it was possible to classify the identity of 1 of 7 phonemes above chance levels (Cogan et al., 2014).”
A system identification approach to determining listening attention from EEG signals
Alickovic, E., Lunner, T., & Gustafsson, F. (2016) · European Signal Processing Conference (EUSIPCO) · doi:10.1109/eusipco.2016.7760204 · Cited by 18
“Regression methods and stimulus reconstruction approaches have already been successfully applied to intracranial EEG or electroencephalographic (ECoG) data [4], [5], [6], MEG data [3], [7] and EEG data [8], [2].”
MethodSmith, E. H., Kellis, S., House, P. A., & Greger, B. E. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/1/016010 · Cited by 18
Model-based Sparse Component Analysis for Multiparty Distant Speech Recognition
Asaei, A. (2013) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5075/epfl-thesis-5723 · Cited by 18
Li, C., Liu, Y., Li, J., Miao, Y., Liu, J., & Song, L. (2024) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2023.3348981 · Cited by 17
“Numerous studies [4], [5], [6], [7], [8], [9] have successfully decoded Electrocor-ticogram (ECoG) signals of silent reading words, phrases, and sentences.”
BackgroundCarney, L. H. (2024) · Hearing Research · doi:10.1016/j.heares.2024.108966 · Cited by 17
Brain-optimized extraction of complex sound features that drive continuous auditory perception
Berezutskaya, J., Freudenburg, Z. V., Güçlü, U., van Gerven, M. A. J., & Ramsey, N. F. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007992 · Cited by 17
“At each time point within 500 ms from the audio onset a vector of 1768 STMFs was added to the model following the setup from the previous studies [4,25].”
MethodVocalMind: A Stereotactic EEG Dataset for Vocalized, Mimed, and Imagined Speech in Tonal Language
He, T., Wei, M., Wang, R., Wang, R., Du, S., Cai, S., Tao, W., & Li, H. (2025) · Scientific Data · doi:10.1038/s41597-025-04741-2 · Cited by 17
Human brain function and brain-computer interfaces
Ramsey, N. F. (2020) · Handbook of clinical neurology · doi:10.1016/b978-0-444-63934-9.00001-9 · Cited by 16
A modular high-density μ ECoG system on macaque vlPFC for auditory cognitive decoding
Chiang, C., Lee, J., Wang, C., Williams, A. J., Lucas, T. H., Cohen, Y. E., & Viventi, J. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab9986 · Cited by 16
Dynamic time-locking mechanism in the cortical representation of spoken words
Nora, A., Faisal, A., Seol, J., Renvall, H., Formisano, E., & Salmelin, R. H. (2020) · eNeuro · doi:10.1523/eneuro.0475-19.2020 · Cited by 16
“, 2014, 2017); computationally demanding time-varying MPS features are decodable from spatially limited intracranial recordings (Pasley et al., 2012).”
BackgroundMethodResultInfluentialJohannesen, P. T., Leclère, T., Wijetillake, A., Segovia-Martínez, M., & Lopez‐Poveda, E. A. (2022) · Hearing Research · doi:10.1016/j.heares.2022.108621 · Cited by 16
Synigal, S. R., Anderson, A. J., & Lalor, E. C. (2026) · eNeuro · doi:10.1523/eneuro.0069-26.2026 · Cited by 16
FULL WAVE MODELING OF BRAIN WAVES AS ELECTROMAGNETIC WAVES (Invited Paper)
Jain, S., Mittra, R., & Wiart, J. (2015) · Electromagnetic waves · doi:10.2528/pier15011404 · Cited by 16
“Researchers have succeeded in converting the brain wave patterns recorded by using electroencephalogram (EEG) into alphabets and words [1, 2].”
MethodA frequency-band coupling model of EEG signals can capture features from an input audio stimulus
Moinnereau, M.-A., Rouat, J., Whittingstall, K., & Plourde, É. (2020) · Hearing Research · doi:10.1016/j.heares.2020.107994 · Cited by 16
How learning to abstract shapes neural sound representations
Ley, A., Vroomen, J., & Formisano, E. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00132 · Cited by 16
“E. V., Tryfon, A., and Hyde, K. L. (2012). Auditorymusical processing in autism spectrum disorders: a review of behavioral and”
BackgroundCarvalho, J. P., Carvalho, V. R., Mendes, E. M. A. M., Fallah, A., B, T. J. S. A., Lainscsek, C. S. M., & Comstock, L. B. (2025) · Scientific Data · doi:10.1038/s41597-025-05187-2 · Cited by 16
“Neural signals from electrocorticography (ECoG) provide a measure of task-relevant brain activity with a minimal processing latency and a superior signal-to-noise ratio 2,3 .”
Neural2speech: A Transfer Learning Framework for Neural-Driven Speech Reconstruction
Li, J., Guo, C., Fu, L., Fan, L., Chang, E. F., & Li, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446614 · Cited by 15
“Neural-driven speech reconstruction represents an innovative inverse mapping technique designed to approximate natural speech patterns by decoding the neural activity elicited in the brain [1].”
BackgroundSelf-citationReconstructing Covert Consciousness: Neural Decoding as a Novel Consciousness Assessment
Fischer, D. B., Edlow, B. L., Freeman, H. J., Alaiev, D., Wu, Q., Ware, J. B., Detre, J. A., & Aguirre, G. K. (2025) · Neurology · doi:10.1212/wnl.0000000000210208 · Cited by 15
Park, J.-Y., Tsukamoto, M., Tanaka, M., & Kamitani, Y. (2025) · PLoS Biology · doi:10.1371/journal.pbio.3003293 · Cited by 15
“Invasive recordings, such as intracranial electrodes, can capture both high temporal and spatial resolutions to decode overt and imagined speech [14,15] and reconstruct speech from the nonprimary auditory cortex (AC) [16].”
Progress, challenges and future of linguistic neural decoding with deep learning
Wang, Y., Liu, H., Wang, Y., Xuan, C., Hou, Y., Feng, S., Liu, H., Liao, Y., & Wang, Y. (2025) · Communications Biology · doi:10.1038/s42003-025-08511-z · Cited by 15
“67, the researchers opened up the possibility of speech restoration with evoked brain recordings.”
Improving hearing aid fitting using the speech-evoked auditory brainstem response
Dajani, H. R., Heffernan, B. P., & Giguère, C. (2013) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2013.6610125 · Cited by 15
Current Advances in Neural Decoding
van Gerven, M. A. J., Seeliger, K., Güçlü, U., & Güçlütürk, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_21 · Cited by 15
Metadata
- Authors
- Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F.
- Affiliations
- University of California, Berkeley (US)
- University of Maryland, College Park (US)
- University of California, San Francisco (US)
- Johns Hopkins University (US)
- Corresponding author
- Pasley, Brian N.
- ORCID
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2012-01-31
- Volume
- 10
- Issue
- 1
- Pages
- e1001251
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC3269422
- OpenAlex
- W2005066437
- Cited by
- 604
- References
- 61
- Citations by year
- 2026: 21 · 2025: 30 · 2024: 28 · 2023: 35 · 2022: 36 · 2021: 32 · 2020: 51 · 2019: 66 · 2018: 42 · 2017: 53 · 2016: 43 · 2015: 51 · 2014: 42 · 2013: 50 · 2012: 24
- FWCI
- 27.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- National Institutes of Health (2r01ns021135-33, 5r00ns065120-04)
- U.S. Public Health Service (P04813)
- National Institute on Deafness and Other Communication Disorders (K99DC010439, K99DC012804, R01DC012379)
- National Institute of Neurological Disorders and Stroke (NS-40596, NS-21135, K99NS065120, R56NS021135, R37NS021135, R01NS040596, R00-NS065120, R01 NS-021135)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neuroscience and Music Perception
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Keywords
- speech perception, auditory cortex, superior temporal gyrus, neural decoding, speech reconstruction, acoustic features, syllable rate, word recognition, nonlinear decoding, speech intelligibility
- MeSH terms
- Algorithms, Auditory Cortex, Brain Mapping, Computer Simulation, Electrodes, Implanted, Electroencephalography, Female, Humans, Male, Models, Biological, Speech Acoustics, Linear Models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.