Reconstructing Speech from Human Auditory Cortex
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





![Figure 6 Schematic of nonlinear modulation model. (A) The input spectrogram (top left) is transformed by a linear modulation filter bank (right) followed by a nonlinear magnitude operation (not shown). This nonlinear operation extracts the modulation energy of the incoming spectrogram and generates phase invariance to local fluctuations in the spectrogram envelope. The input representation is the two-dimensional spectrogram S(f,t) across frequency f and time t. The output (bottom left) is the four-dimensional modulation energy representation M(s,r,f,t) across spectral modulation scale s, temporal modulation rate r, frequency f, and time t. In the full modulation representation [18], negative rates by convention correspond to upward frequency sweeps, while positive rates correspond to downward frequency sweeps. Accuracy for positive and negative rates was averaged unless otherwise shown. See Materials and Methods. (B) Schematic of linear (spectrogram envelope) and nonlinear (modulation energy) temporal coding. Left: acoustic waveform (black curve) and spectrogram of a temporally modulated tone. The linear spectrogram model (top) assumes that neural responses are a linear function of the spectrogram envelope (plotted for the tone center frequency channel, top right). In this case, the instantaneous output may be high or low and does not directly indicate the modulation rate of the envelope. The nonlinear modulation model (bottom) assumes that neural responses are a linear function of modulation energy. This is an amplitude-based coding scheme (plotted for the peak modulation channel, bottom right). The nonlinear modulation model explicitly estimates the modulation rate by taking on a constant value for a constant rate [32].](https://figures.cochat.ai/paper-figures/d8911f10b353486dba673852c364a006/reconstructing-speech-from-human-auditory-cortex-figure-6.jpg)
![Figure 7 Example of nonlinear modulation coding and reconstruction. (A) Top: the spectrogram of an isolated word (“waldo”) presented aurally to one participant. Blue curve plots the spectrogram envelope, summed over all frequencies. Left panels: induced high gamma responses (black curves, trial averaged) at four different STG sites. Temporal modulation energy of the stimulus (dashed red curves) is overlaid (computed from 2, 4, 8, and 16 Hz modulation filters and normalized to maximum value). Dashed black lines indicate baseline response level. Right panels: nonlinear modulation rate tuning curves for each site (estimated from nonlinear STRFs). Shaded regions and error bars indicate SEM. (B) Original spectrogram (top), modulation-based reconstruction (middle), and spectrogram-based reconstruction (bottom), linearly decoded from a fixed set of STG electrodes. The modulation reconstruction is projected into the spectrogram domain using an iterative projection algorithm and an overcomplete set of modulation filters [18]. The displayed spectrogram is averaged over 100 random initializations of the algorithm.](https://figures.cochat.ai/paper-figures/2d2eaf31d85c437bb041f38dd7527a14/reconstructing-speech-from-human-auditory-cortex-figure-7.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Neural activity in the STG encodes acoustic features — for example, recurring speech sound patterns in the mel-spectrogram — during speech perception53,54 and may be used to reconstruct perceived speech55,56.”
From The speech neuroprosthesis (2024) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex
Hubel, D. H., & Wiesel, T. N. (1962) · The Journal of Physiology · doi:10.1113/jphysiol.1962.sp006837 · Cited by 13,680 · cited 1×
“The modulation model therefore represents an auditory analog to the classical energy model of complex cells in the visual system [32–36], which are invariant to the spatial phase of visual stimuli.”
MethodFundamentals of speech recognition
Rabiner, L. R., & Juang, B.-H. F. (1993) · Cited by 7,680
The cortical organization of speech processing
Hickok, G., & Poeppel, D. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2113 · Cited by 5,495 · cited 2×
“For speech comprehension, it is hypothesized that intermediate and higher-order auditory areas extract or construct information-rich features of speech, while discarding nonessential low-level acoustic information [4,5,9, 10,43].”
BackgroundSpatiotemporal energy models for the perception of motion
Adelson, E. H., & Bergen, J. R. (1985) · Journal of the Optical Society of America A · doi:10.1364/josaa.2.000284 · Cited by 3,442 · cited 2×
“Such amplitude-based coding schemes are broadly referred to as ‘‘energy models’’ [32,33].”
BackgroundDISCONNEXION SYNDROMES IN ANIMALS AND MAN
GESCHWIND, N. (1965) · Brain · doi:10.1093/brain/88.2.237 · Cited by 3,403 · cited 5×
“To recover the acoustic waveform from the spectrogram, we used an iterative projection algorithm to estimate the phase [8].”
BackgroundMethodInfluentialTIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,508
Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing
Rauschecker, J. P., & Scott, S. K. (2009) · Nature Neuroscience · doi:10.1038/nn.2331 · Cited by 1,766 · cited 5×
“Beyond primary auditory areas, further processing in intermediate and higher-order auditory cortex likely results in additional stimulus transformations [5].”
BackgroundResultInfluentialLogothetis, N. K., & Wandell, B. A. (2004) · Annual Review of Physiology · doi:10.1146/annurev.physiol.66.082602.092845 · Cited by 1,568 · cited 1×
“This is consistent with robust speech-induced high gamma responses reported in previous intracranial studies [24–29] and with observed correlations between high gamma power and local spike rate [30].”
ResultIdentifying natural images from human brain activity
Kay, K. N., Naselaris, T., Prenger, R. J., & Gallant, J. L. (2008) · Nature · doi:10.1038/nature06713 · Cited by 1,286 · cited 2×
“An important application of this approach has also been demonstrated in the study of visual object recognition in the primate visual system [36,49,50].”
BackgroundMethod
Cited by
Items 101–200 of 694
Silva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103
“Neural activity in the STG encodes acoustic features — for example, recurring speech sound patterns in the mel-spectrogram — during speech perception 53,54 and may be used to reconstruct perceived speech 55,56 .”
Self-citationHearing in Time: Evoked Potential Studies of Temporal Processing
Picton, T. W. (2013) · Ear and Hearing · doi:10.1097/aud.0b013e31827ada02 · Cited by 102
“Pasley et al. (2012) were able to reconstruct the acoustic speech signal from neuronal activity recorded from the posterior temporal lobe, using an algorithm that largely depended on spectral transitions.”
MethodCarney, L. H., Li, T., & McDonough, J. M. (2015) · eNeuro · doi:10.1523/eneuro.0004-15.2015 · Cited by 100
“Recent results in the cortex suggest that phonetic features are encoded in the responses of the superior temporal gyrus (Mesgarani et al., 2008; Pasley et al., 2012), but the problem of how neural maps at cortical levels are created from the acoustic stimulus remains.”
BackgroundCortical Activation Patterns Correlate with Speech Understanding After Cochlear Implantation
Olds, C. E., Pollonini, L., Abaya, H., Larky, J. B., Loy, M., Bortfeld, H., Beauchamp, M. S., & Oghalai, J. S. (2015) · Ear and Hearing · doi:10.1097/aud.0000000000000258 · Cited by 97
“…and beyond the auditory cortex, and thus neuroimaging could provide a way to supplement behavioral measures in assessing the ability of the speech information provided by the CI to correctly reach and stimulate the language centers of the brain (Pasley et al. 2012; Steinschneider et al. 2014).”
BackgroundSpeaker-normalized sound representations in the human auditory cortex
Sjerps, M. J., Fox, N. P., Johnson, K. A., & Chang, E. F. (2019) · Nature Communications · doi:10.1038/s41467-019-10365-z · Cited by 96
Self-citationElectrocorticographic representations of segmental features in continuous speech
Lotte, F., Brumberg, J. S., Brunner, P., Gunduz, A., Ritaccio, A. L., Guan, C., & Schalk, G. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00097 · Cited by 86
Music can be reconstructed from human auditory cortex activity using nonlinear decoding models
Bellier, L., Llorens, A., Marciano, D., Schalk, G., Brunner, P., Knight, R. T., & Pasley, B. N. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002176 · Cited by 93
“This shows that previous methodological findings in speech decoding [34,35] also apply to music decoding.”
BackgroundMethodResultInfluentialSelf-citationManning, J. R., Sperling, M. R., Sharan, A., Rosenberg, E. A., & Kahana, M. J. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.5321-11.2012 · Cited by 87
“In exciting extensions of this work, similar approaches were used to decode movie clips (Nishimoto et al., 2011) and spoken words (Pasley et al., 2012).”
MethodSung, C., Jeon, W., Nam, K. S., Kim, Y., Butt, H., & Park, S. (2020) · Journal of Materials Chemistry B · doi:10.1039/d0tb00872a · Cited by 89
Millman, R. E., Johnson, S. R., & Prendergast, G. (2014) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00719 · Cited by 86
“Moreover, a restricted focus based on the assumption of a linear relationship between the speech temporal envelope and neural responses ignores the importance of nonlinear sound representations, for example, within high gamma (>70 Hz) power (Zion Golumbic et al., 2013; Pasley et al., 2012; Nourski et al., 2009).”
BackgroundKey considerations in designing a speech brain-computer interface
Bocquelet, F., Hueber, T., Girin, L., Chabardès, S., & Yvert, B. (2016) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2017.07.002 · Cited by 73
“Such reconstruction was all the more accurate that the number of electrode sites was high, and the most informative sites were found to be in temporal auditory areas (Pasley et al., 2012).”
InfluentialDynamic Encoding of Speech Sequence Probability in Human Temporal Cortex
Leonard, M. K., Bouchard, K. E., Tang, C., & Chang, E. F. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.4100-14.2015 · Cited by 77
“…anterior STG population activity has been shown to encode a relatively fine-scale representation of the spectrotemporal profile of speech sounds (Pasley et al., 2012; Steinschneider, 2013; Mesgarani et al., 2014), while at the same time showing sensitivity to contextual processes, such as…”
BackgroundSelf-citationRecent advances in understanding the auditory cortex
King, A. J., Teki, S., & Willmore, B. D. B. (2018) · F1000Research · doi:10.12688/f1000research.15580.1 · Cited by 86
“This type of approach is part of a growing trend to investigate the decoding or reconstruction of sound features from the measured responses of populations of neurons (or other multidimensional measures of brain activity, such as fMRI voxel responses 82 or electroencephalography signals).”
BackgroundBrain-computer interfaces for communication
Vansteensel, M. J., & Jarosiewicz, B. (2020) · Handbook of clinical neurology · doi:10.1016/b978-0-444-63934-9.00007-x · Cited by 84
“Several studies have decoded perceived speech from the auditory cortex (Pasley et al., 2012; Akbari et al., 2019).”
BackgroundAuditory Object Formation and Selection
Shinn‐Cunningham, B. G., Best, V., & Lee, A. K. C. (2017) · Springer handbook of auditory research · doi:10.1007/978-3-319-51662-2_2 · Cited by 82
“One ECoG study analyzed the high gamma (75–150 Hz) local field potentials recorded directly from human posterior superior temporal gyrus (Mesgarani and Chang 2012), which provided an opportunity to estimate the speech spectrogram represented by the population neural response using a stimulus reconstruction method (Pas…”
MethodEffect of number and placement of EEG electrodes on measurement of neural tracking of speech
Montoya-Martínez, J., Vanthornhout, J., Bertrand, A., & Francart, T. (2021) · PLoS ONE · doi:10.1371/journal.pone.0246769 · Cited by 82
Brumberg, J. S., Krusienski, D. J., Chakrabarti, S., Gunduz, A., Brunner, P., Ritaccio, A. L., & Schalk, G. (2016) · PLOS ONE · doi:10.1371/journal.pone.0166872 · Cited by 71
Structured neuronal encoding and decoding of human speech features
Tankus, A., Fried, I., & Shoham, S. (2012) · Nature Communications · doi:10.1038/ncomms1995 · Cited by 69
“Overall, 8% of the analyzed units (49) were speech-related, of which, more than a half (25) were vowel-tuned, showing significantly different activation for the 5 vowels (see Supplementary Figure S1).”
BackgroundPotes, C., Brunner, P., Gunduz, A., Knight, R. T., & Schalk, G. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.04.045 · Cited by 78
“Electrocorticographic (ECoG) recordings from the surface of the brain have recently been used to study the neural dynamics during processing of complex sounds, in particular speech (Edwards et al., 2009; Lachaux et al., 2007; Pasley et al., 2012; Sinai et al., 2009).”
Self-citationBrain2Char: a deep architecture for decoding text from brain recordings
Sun, P., Anumanchipalli, G. K., & Chang, E. F. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc742 · Cited by 77
“For dilated CNN, the filter size is fixed as 11, and the dilated ratios for five sub-layer are [1, 2, 4, 8, 16].”
MethodSelf-citationProgress in speech decoding from the electrocorticogram
Chakrabarti, S., Sandberg, H. M., Brumberg, J. S., & Krusienski, D. J. (2015) · Biomedical Engineering Letters · doi:10.1007/s13534-015-0175-1 · Cited by 75
“(2012), in a study that examined ECoG recordings from the STG to reconstruct the speech spectrogram of aurally presented words and sentences [57].”
Dmochowski, J., Ki, J. J., DeGuzman, P., Sajda, P., & Parra, L. C. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.05.037 · Cited by 75
“…et al., 2006; Thirion et al., 2006; Miyawaki et al., 2008; Kay et al., 2008; Nishimoto et al., 2011; Horikawa et al., 2013) and auditory stimuli (Pasley et al., 2012; Luo and Poeppel, 2007; Mesgarani et al., 2014; Mesgarani and * Corresponding…”
Michelmann, S., Price, A. R., Aubrey, B., Strauss, C. K., Doyle, W., Friedman, D. J., Dugan, P. C., Devinsky, O., Devore, S., Flinker, A., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-25376-y · Cited by 74
Self-citationAdult-like processing of naturalistic sounds in auditory cortex by 3- and 9-month old infants
Wild, C. J., Linke, A. C., Zubiaurre‐Elorza, L., Herzmann, C., Duffy, H., Han, V. K. M., Lee, D. S. C., & Cusack, R. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.038 · Cited by 70
“…explanation is that it could indicate the processing of more complex acoustic features, such as spectro-temporal modulations (Hall et al., 2002b; Hullett et al., 2016), which are critical for speech recognition (Pasley et al., 2012), or even phonemes (Chang et al., 2010; Mesgarani et al., 2014).”
EEG-Based Auditory Attention Detection and Its Possible Future Applications for Passive BCI
Belo, J., Clerc, M., & Schön, D. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.661178 · Cited by 70
“This approach, called EEG-based auditory attention detection (AAD), is based on fundamental research findings demonstrating that, in a multi speech scenario, cortical tracking of the envelope of the attended speech is enhanced compared to the unattended speech (Mesgarani et al., 2009; Ding and Simon, 2012; Mesgarani a…”
BackgroundDirect-Sense Brain–Computer Interfaces and Wearable Computers
Lin, C., & Do, T. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.3041382 · Cited by 70
“For example, Pasley et al. [103] demonstrated STG neural activity recorded by invasive sensor ECoG can be directly decoded into acoustics.”
BackgroundThe effects of spatial filtering and artifacts on electrocorticographic signals
Liu, Y., Coon, W. G., de Pesters, A., Brunner, P., & Schalk, G. (2015) · Journal of Neural Engineering · doi:10.1088/1741-2560/12/5/056008 · Cited by 69
“…band-pass filtered in the broadband gamma frequency band (70–170 Hz), because activity in this frequency band has been shown in many previous ECoG studies to be an accurate measure of task-related activity (Miller et al 2007, Schalk et al 2007, Sinai et al 2009, Pei et al 2011, Pasley et al 2012).”
Auditory processing in the human cortex: An intracranial electrophysiology perspective
Nourski, K. V. (2017) · Laryngoscope Investigative Otolaryngology · doi:10.1002/lio2.73 · Cited by 69
“Using STRF-based reconstruction models, Pasley et al.67 were able to decode speech spectrograms from patterns of Laryngoscope Investigative Otolaryngology 2: August 2017 Nourski: Auditory Processing in the Human Cortex cortical high gamma activity.”
MethodNeural Correlates of Music Listening and Recall in the Human Brain
Ding, Y., Zhang, Y., Zhou, W., Ling, Z., Huang, J., Hong, B., & Wang, X. (2019) · Journal of Neuroscience · doi:10.1523/jneurosci.1468-18.2019 · Cited by 69
“…speech, which contains both speech perception and imagery, researchers found that the models built from an overt speech dataset using high-gamma band activity could be used to reconstruct a covert speech, suggesting a partially share neural substrate (Pasley et al., 2012; Martin et al., 2014).”
BackgroundNeural Encoding of Auditory Features during Music Perception and Imagery
Martin, S., Mikutta, C., Leonard, M. K., Hungate, D., Koelsch, S., Shamma, S., Chang, E. F., Millán, J. D. R., Knight, R. T., & Pasley, B. N. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhx277 · Cited by 61
“Auditory perception has been extensively studied in animal models and humans using spectrotemporal receptive field (STRFs) analysis (Aertsen et al. 1981; Clopton and Backoff 1991; Theunissen et al. 2000; Chi et al. 2005; Pasley et al. 2012), which identifies the time-frequency stimulus features encoded by a neuron or…”
BackgroundMethodInfluentialSelf-citationMégevand, P., Mercier, M. R., Groppe, D. M., Golumbic, E. Z., Mesgarani, N., Beauchamp, M. S., Schroeder, C. E., & Mehta, A. D. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.0555-20.2020 · Cited by 67
Self-citationRapid computations of spectrotemporal prediction error support perception of degraded speech
Sohoglu, E., & Davis, M. H. (2020) · eLife · doi:10.7554/elife.58077 · Cited by 66
Neural encoding of sensory and behavioral complexity in the auditory cortex
Kuchibhotla, K. V., & Bathellier, B. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2018.04.002 · Cited by 66
Neural Networks for Efficient Bayesian Decoding of Natural Images from Retinal Neurons
Parthasarathy, N., Batty, E., Falcon, W., Rutten, T., Rajpal, M., Chichilnisky, E. J., & Paninski, L. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/153759 · Cited by 50
“The most common and well-understood decoding approach, linear regression, has been used in various sensory systems [29, 40].”
BackgroundDetecting the locus of auditory attention based on the spectro-spatial-temporal analysis of EEG
Jiang, Y., Chen, N., & Jin, J. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac975c · Cited by 64
Noe, C., & Fischer‐Baum, S. (2020) · Cognition · doi:10.1016/j.cognition.2019.104162 · Cited by 63
Sound representation in higher language areas during language generation
Magrassi, L., Aromataris, G., Cabrini, A., Annovazzi-Lodi, V., & Moro, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1418162112 · Cited by 63
“These findings are similar to those described in the superior temporal gyrus during listening (1, 6).”
InfluentialMillán, J. D. R. (2013) · Wiley eBooks · doi:10.1002/9781118628522.ch12 · Cited by 63
Cortical encoding of speech enhances task-relevant acoustic information
Rutten, S., Santoro, R., Hervais‐Adelman, A., Formisano, E., & Golestani, N. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0648-9 · Cited by 63
Temporal coding in the auditory cortex
Arnal, L. H., Poeppel, D., & Giraud, A. (2015) · Handbook of clinical neurology · doi:10.1016/b978-0-444-62630-1.00005-6 · Cited by 63
High Accuracy Decoding of Dynamical Motion from a Large Retinal Population
Marre, O., Botella‐Soler, V., Simmons, K. D., Mora, T., Tkačik, G., & Berry, M. J. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004304 · Cited by 62
Invasive recordings in the human auditory cortex
Nourski, K. V., & Howard, M. A. (2015) · Handbook of clinical neurology · doi:10.1016/b978-0-444-62630-1.00013-5 · Cited by 62
Sturm, I., Blankertz, B., Potes, C., Schalk, G., & Curio, G. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00798 · Cited by 61
“The five features were calculated as described in the following sections.”
MethodPotes, C., Gunduz, A., Brunner, P., & Schalk, G. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.04.022 · Cited by 61
“01). al., 2011), or auditory processing (Pasley et al., 2012; Sinai et al., 2009).”
Decoding Cognitive Processes from Neural Ensembles
Wallis, J. D. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.09.002 · Cited by 61
“For example, it has been possible to decode speech from neural activity in the superior temporal cortex evoked by speech [72].”
BackgroundDecoding human brain activity with deep learning
Zheng, X. H., Chen, W., Li, M., Zhang, T., You, Y., & Jiang, Y. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.101730 · Cited by 61
Cooperative cortical network for categorical processing of Chinese lexical tone
Si, X., Zhou, W., & Hong, B. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710752114 · Cited by 60
Roussel, P., Godais, G. L., Bocquelet, F., Palma, M., Hongjie, J., Zhang, S., Giraud, A.-L., Mégevand, P., Miller, K., Gehrig, J., et al. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abb25e · Cited by 49
“Several advances have indeed been made over the past decade regarding the decoding of intracranial brain signals underlying either speech perception (Pasley et al., 2012; Chan et al., 2013; Pasley and Knight, 2013; Fontolan et al., 2014; Hyafil et al., 2015; Yildiz et al., 2016; Akbari et al.,…”
BackgroundNeuroGAN: image reconstruction from EEG signals via an attention-based GAN
Mishra, R. K., Sharma, K., Jha, R. R., & Bhavsar, A. V. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-08178-1 · Cited by 59
“PBD has been already explored using modalities such as f-MRI and ECoG as well [6].”
MethodHigh-frequency neural activity predicts word parsing in ambiguous speech streams
Kösem, A., Basirat, A., Azizi, L., & van Wassenhove, V. (2016) · Journal of Neurophysiology · doi:10.1152/jn.00074.2016 · Cited by 59
Akram, S., Simon, J. Z., & Babadi, B. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2628884 · Cited by 58
“Hence these methods cannot measure changes in the STRF occurring on time scales faster than the order of the integration time, typically a minute [14]–[17].”
EEG-Based Auditory Attention Detection via Frequency and Channel Neural Attention
Cai, S., Su, E., Xie, L., & Li, H. (2021) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2021.3125283 · Cited by 57
The use of intracranial recordings to decode human language: Challenges and opportunities
Martin, S., Millán, J. D. R., Knight, R. T., & Pasley, B. N. (2016) · Brain and Language · doi:10.1016/j.bandl.2016.06.003 · Cited by 56
Self-citationYildiz, I. B., von Kriegstein, K., & Kiebel, S. J. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003219 · Cited by 55
“When we recognize speech, our brains map fast speech sound wave modulations to spectrotemporal auditory representations [3,4].”
BackgroundDecoding selective auditory attention with EEG using a transformer model
Xu, Z., Bai, Y., Zhao, R., Hu, H., Ni, G., & Ming, D. (2022) · Methods · doi:10.1016/j.ymeth.2022.04.009 · Cited by 55
Güçlü, U., & van Gerven, M. A. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003724 · Cited by 55
“The invasive nature of the measurement techniques of these studies has restricted human subjects to particular patient populations [3,4].”
BackgroundFiedler, L., Obleser, J., Lunner, T., & Graversen, C. (2016) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2016.7592020 · Cited by 54
“Previous research has demonstrated the feasibility to extract information on attention in such situations by conventional scalp EEG [4, 5].”
BackgroundWhat can we learn about beat perception by comparing brain signals and stimulus envelopes?
Henry, M. J., Herrmann, B., & Grahn, J. A. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172454 · Cited by 53
Extracting the Auditory Attention in a Dual-Speaker Scenario From EEG Using a Joint CNN-LSTM Model
Kuruvila, I., Muncke, J., Fischer, E., & Hoppe, U. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.700655 · Cited by 53
“They are speech onsets (Howard and Poeppel, 2010), speech envelopes (Aiken and Picton, 2008), speech spectrograms (Pasley et al., 2012) or phonemes (Di Liberto et al., 2015).”
Conserved mechanisms of vocalization coding in mammalian and songbird auditory midbrain
Woolley, S. M. N., & Portfors, C. V. (2013) · Hearing Research · doi:10.1016/j.heares.2013.05.005 · Cited by 52
“Similarly, population activity in the human auditory cortex can be used to accurately identify presented speech segments (Pasley et al., 2012; Mesgarani and Chang, 2012).”
BackgroundAdaptive auditory computations
Shamma, S., & Fritz, J. B. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2014.01.011 · Cited by 52
“For instance, when humans attend to a speaker, they seem to bring to the fore the physiological signals of the target while strongly suppressing those of the background [45 ,46,47 ,48].”
BackgroundSelf-citationTowards direct speech synthesis from ECoG: A pilot study
Herff, C., Johnson, G., Diener, L., Shih, J., Krusienski, D., & Schultz, T. (2016) · 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2016.7591004 · Cited by 41
“were able to reconstruct perceived speech from neural activity [10] and Martin et al [11] showed”
Bortfeld, H. (2018) · Developmental Psychobiology · doi:10.1002/dev.21818 · Cited by 52
“…occurs within and beyond the auditory cortex, neuroimaging with fNIRS provides an additional means of assessing whether auditory information relayed by an implant is delivered to the auditory cortex and beyond (e.g., to language‐specific cortical regions of the brain) (Pasley et al., 2012).”
BackgroundRupp, K., Roos, M. J., Milsap, G., Caceres, C. A., Ratto, C. R., Chevillet, M. A., Crone, N. E., & Wolmetz, M. E. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.12.074 · Cited by 51
Self-citationRecent advances in exploring the neural underpinnings of auditory scene perception
Snyder, J. S., & Elhilali, M. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13317 · Cited by 50
Real-time classification of auditory sentences using evoked cortical activity in humans
Moses, D. A., Leonard, M. K., & Chang, E. F. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aaab6f · Cited by 41
“High gamma power was used because previous research has shown that activity in this band strongly correlates with multi-unit activity [17] and is associated with important speech features [6, 7, 10].”
MethodSelf-citationTowards reconstructing intelligible speech from the human auditory cortex
Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/350124 · Cited by 43
Self-citationZhang, D., Song, H., Xu, R., Zhou, W., Ling, Z., & Hong, B. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2012.12.069 · Cited by 47
“Such procedures have been successfully applied in studies focusing on the primary motor cortex for movement decoding (Leuthardt et al., 2009) and the posterior superior temporal gyrus for speech reconstruction (Chang et al., 2010; Pasley et al., 2012).”
Nourski, K. V., Steinschneider, M., Oya, H., Kawasaki, H., Jones, R., & Howard, M. A. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhs314 · Cited by 48
Induced cortical responses require developmental sensory experience
Yusuf, P. A., Hubka, P., Tillein, J., & Kral, A. (2017) · Brain · doi:10.1093/brain/awx286 · Cited by 48
“, 2015), electrocorticography (ECoG) (Edwards et al., 2009; Pasley et al., 2012; de Pesters et al., 2016), and in intracortical recordings (Morillon et al.”
BackgroundHierarchy of speech-driven spectrotemporal receptive fields in human auditory cortex
Venezia, J. H., Thurman, S. M., Richards, V. M., & Hickok, G. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2018.11.049 · Cited by 47
“, 2016), that speech signals can be reconstructed from patterns of activity in the STG using STM-based STRF models (Pasley et al., 2012), and that shifts in STM tuning within the auditory cortex underlie cortical plasticity associated with priming-induced changes in speech intelligibility (Holdgraf et al.”
Venezia, J. H., Hickok, G. S., & Richards, V. M. (2016) · The Journal of the Acoustical Society of America · doi:10.1121/1.4960544 · Cited by 47
Gao, Y., Wang, Q., Ding, Y., Wang, C., Li, H., Wu, X. H., Qu, T., & Li, L. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00034 · Cited by 47
“To avoid the onset and offset (above 3000-ms) effect (Pasley et al., 2012), the period of interest was defined within the time 800–2800 ms after the mixed-speech presentation onset.”
Shinn‐Cunningham, B. G. (2017) · Journal of Speech Language and Hearing Research · doi:10.1044/2017_jslhr-h-17-0080 · Cited by 46
Hearing in noisy environments: noise invariance and contrast gain control
Willmore, B. D. B., Cooke, J. E., & King, A. J. (2014) · The Journal of Physiology · doi:10.1113/jphysiol.2014.274886 · Cited by 46
“Moreover, in humans, a non-linear representation of sound modulations contributes to noise robustness (Pasley et al. 2012).”
BackgroundNourski, K. V., Steinschneider, M., Rhone, A. E., Oya, H., Kawasaki, H., Howard, M. A., & McMurray, B. (2015) · Brain and Language · doi:10.1016/j.bandl.2015.03.003 · Cited by 46
Reading the Human Brain: How the Mind Became Legible
Rose, N. (2016) · Body & Society · doi:10.1177/1357034x15623363 · Cited by 45
Chronic ambulatory electrocorticography from human speech cortex
Rao, V. R., Leonard, M. K., Kleen, J. K., Lucas, B. A., Mirro, E. A., & Chang, E. F. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.04.008 · Cited by 42
“For example, auditory cortex contains neural populations that are selective to spectrotemporal acoustic features (Pasley et al., 2012), but it is unknown whether or how these encoding mechanisms change over long timescales.”
Self-citationHeelan, C., Lee, J., O’Shea, R., Lynch, L., Brandman, D. M., Truccolo, W., & Nurmikko, A. V. (2019) · Communications Biology · doi:10.1038/s42003-019-0707-9 · Cited by 40
The Listening Zone of Human Electrocorticographic Field Potential Recordings
McCarty, M. J., Woolnough, O., Mosher, J. C., Seymour, J. P., & Tandon, N. (2022) · eNeuro · doi:10.1523/eneuro.0492-21.2022 · Cited by 44
“, 2006), including speech production (Pasley et al., 2012; Cogan et al., 2014), auditory processing (Sinai et al.”
BackgroundNeuroprosthetic Speech: The Ethical Significance of Accuracy, Control and Pragmatics
Rainey, S., Maslen, H., Mégevand, P., Arnal, L. H., Fourneret, É., & Yvert, B. (2019) · Cambridge Quarterly of Healthcare Ethics · doi:10.1017/s0963180119000604 · Cited by 44
“Performance was good: although the reconstructed sounds were not intelligible to human ears, the words they corresponded to could be classified accurately by a speech recognition algorithm.(12)”
BackgroundEEG-based detection of the locus of auditory attention with convolutional neural networksPreprint
Vandecappelle, S., Deckers, L., Das, N., Ansari, A. H., Bertrand, A., & Francart, T. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/475673 · Cited by 44
“A common approach is stimulus reconstruction: the post-stimulus brain activity is used to decode and reconstruct the attended stimulus envelope (O’sullivan et al., 2014; Pasley et al., 2012).”
InfluentialDirect Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2022) · doi:10.1101/2022.08.02.502503 · Cited by 42
Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/ace8be · Cited by 42
Attaheri, A., Choisdealbha, Á. N., Rocha, S., Brusini, P., Liberto, G. M. D., Mead, N., Olawole‐Scott, H., Boutris, P., Gibbon, S., Williams, I., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0313274 · Cited by 43
Differential Coding of Conspecific Vocalizations in the Ventral Auditory Cortical Stream
Fukushima, M., Saunders, R. C., Leopold, D. A., Mishkin, M., & Averbeck, B. B. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.3969-13.2014 · Cited by 42
Nugent, A. C., Benitez Andonegui, A., Holroyd, T., & Robinson, S. E. (2022) · Neuroimage: Reports · doi:10.1016/j.ynirp.2022.100093 · Cited by 41
Decoding intracranial EEG data with multiple kernel learning method
Schrouff, J., Mourão-Miranda, J., Phillips, C. L. M., & Parvizi, J. (2015) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2015.11.028 · Cited by 41
“…in specific band-widths of ECoG signal at each recording site MKL is therefore used here as a tool for feature selection As a proof of concept study, we examined the feasibility of the MKL approach by applying this method to previously published human electrophysiological data (Dastjerdi et al.,…”
Intracerebral evidence of rhythm transform in the human auditory cortex
Nozaradan, S., Mouraux, A., Jonas, J., Colnat‐Coulbois, S., Rossion, B., & Maillard, L. G. (2016) · Brain Structure and Function · doi:10.1007/s00429-016-1348-0 · Cited by 41
“Even though speech constitutes an ecological signal able to elicit robust activations throughout the auditory nervous system (see Pasley et al. 2012; Zion Golumbic et al. 2013; Leonard et al. 2015; Nourski et al. 2015 for recent results on speech encoding within human auditory cortices), its…”
BackgroundLing, S., Lee, A. C. H., Armstrong, B. C., & Nestor, A. (2019) · Human Brain Mapping · doi:10.1002/hbm.24757 · Cited by 39
“…letters from fMRI patterns in visual cortex (Schoenmakers et al., 2013; Shen et al., 2019; Thirion et al., 2006) or visualized their representation through psychophysical methods (Gosselin & Schyns, 2003)—see also complementary work (Pasley et al., 2012) targeting ECoG-based speech reconstruction.”
MethodBeguš, G. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00044 · Cited by 39
Fukushima, M., Chao, Z. C., & Fujii, N. (2015) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2015.03.015 · Cited by 39
“There are, however, several advanced statistical techniques available for estimating it from limited data samples [21–23].”
Kröger, B. J., Kannampuzha, J., & Kaufmann, E. (2014) · EPJ Nonlinear Biomedical Physics · doi:10.1140/epjnbp15 · Cited by 39
“Evidence for a higher-level auditory representation is given by [34].”
BackgroundAuditory attention decoding with EEG recordings using noisy acoustic reference signals
Aroudi, A., Mirkovic, B., Vos, M. D., & Doclo, S. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7471764 · Cited by 38
“Index Terms— auditory attention decoding, noisy acoustic reference, speech envelope, brain computer interface”
BackgroundVocal caricatures reveal signatures of speaker identity
López, S. L., Riera, P., Assaneo, M. F., Eguía, M. C., Sigman, M., & Trevisan, M. A. (2013) · Scientific Reports · doi:10.1038/srep03407 · Cited by 37
Cognitive-motor brain–machine interfaces
Tankus, A., Fried, I., & Shoham, S. (2013) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2013.05.005 · Cited by 37
Neural Speech Decoding During Audition, Imagination and Production
Sharon, R., Narayanan, S. S., Sur, M., & Murthy, A. H. (2020) · IEEE Access · doi:10.1109/access.2020.3016756 · Cited by 37
“While [23], [24] study passive listening, [22] looks at speech production and miming.”
BackgroundThe time-course of cortical responses to speech revealed by fast optical imaging
Toscano, J. C., Anderson, N. D., Fabiani, M., Gratton, G., & Garnsey, S. M. (2018) · Brain and Language · doi:10.1016/j.bandl.2018.06.006 · Cited by 36
“Other ECOG studies have suggested fine-grained encoding of speech sounds in pSTG, even sufficient to reconstruct acoustic representations from the intracranial data (Pasley et al., 2012).”
Das, N., Eyndhoven, S. V., Francart, T., & Bertrand, A. (2017) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081390 · Cited by 35
Auditory attention decoding from electroencephalography based on long short-term memory networks
Lu, Y. H., Wang, M., Yao, L., Shen, H., Wu, W., Zhang, Q., Zhang, L., Chen, M., Liu, H., Peng, R., et al. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.102966 · Cited by 35
Metadata
- Authors
- Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F.
- Affiliations
- University of California, Berkeley (US)
- University of Maryland, College Park (US)
- University of California, San Francisco (US)
- Johns Hopkins University (US)
- Corresponding author
- Pasley, Brian N.
- ORCID
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2012-01-31
- Volume
- 10
- Issue
- 1
- Pages
- e1001251
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC3269422
- OpenAlex
- W2005066437
- Cited by
- 604
- References
- 61
- Citations by year
- 2026: 21 · 2025: 30 · 2024: 28 · 2023: 35 · 2022: 36 · 2021: 32 · 2020: 51 · 2019: 66 · 2018: 42 · 2017: 53 · 2016: 43 · 2015: 51 · 2014: 42 · 2013: 50 · 2012: 24
- FWCI
- 27.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- National Institutes of Health (2r01ns021135-33, 5r00ns065120-04)
- U.S. Public Health Service (P04813)
- National Institute on Deafness and Other Communication Disorders (K99DC010439, K99DC012804, R01DC012379)
- National Institute of Neurological Disorders and Stroke (NS-40596, NS-21135, K99NS065120, R56NS021135, R37NS021135, R01NS040596, R00-NS065120, R01 NS-021135)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neuroscience and Music Perception
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Keywords
- speech perception, auditory cortex, superior temporal gyrus, neural decoding, speech reconstruction, acoustic features, syllable rate, word recognition, nonlinear decoding, speech intelligibility
- MeSH terms
- Algorithms, Auditory Cortex, Brain Mapping, Computer Simulation, Electrodes, Implanted, Electroencephalography, Female, Humans, Male, Models, Biological, Speech Acoustics, Linear Models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.