Towards reconstructing intelligible speech from the human auditory cortex
Open accessPreprint — published as 10.1038/s41598-018-37359-z · Paper page
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Auditory stimulus reconstruction is a technique that finds the best approximation of the acoustic stimulus from the population of evoked neural activity. Reconstructing speech from the human auditory cortex creates the possibility of a speech neuroprosthetic to establish a direct communication with the brain and has been shown to be possible in both overt and covert conditions. However, the low quality of the reconstructed speech has severely limited the utility of this method for brain-computer interface (BCI) applications. To advance the state-of-the-art in speech neuroprosthesis, we combined the recent advances in deep learning with the latest innovations in speech synthesis technologies to reconstruct closed-set intelligible speech from the human auditory cortex. We investigated the dependence of reconstruction accuracy on linear and nonlinear (deep neural network) regression methods and the acoustic representation that is used as the target of reconstruction, including auditory spectrogram and speech synthesis parameters. In addition, we compared the reconstruction accuracy from low and high neural frequency ranges. Our results show that a deep neural network model that directly estimates the parameters of a speech synthesizer from all neural frequencies achieves the highest subjective and objective scores on a digit recognition task, improving the intelligibility by 65% over the baseline method which used linear regression to reconstruct the auditory spectrogram. These results demonstrate the efficacy of deep learning and speech synthesis algorithms for designing the next generation of speech BCI systems, which not only can restore communications for paralyzed patients but also have the potential to transform human-computer interaction technologies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes
Buzsáki, G., Anastassiou, C. A., & Koch, C. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3241 · Cited by 4,225
Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature
Destrieux, C., Fischl, B., Dale, A., & Halgren, E. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.06.010 · Cited by 2,956
The functional role of cross-frequency coupling
Canolty, R. T., & Knight, R. T. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.09.001 · Cited by 2,101
Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex
Ray, S., & Maunsell, J. H. R. (2011) · PLoS Biology · doi:10.1371/journal.pbio.1000610 · Cited by 1,012
Power-Law Scaling in the Brain Surface Electric Potential
Miller, K. J., Sorensen, L. B., Ojemann, J. G., & den Nijs, M. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000609 · Cited by 742
Reconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604
Self-citationHearing in the Mind's Ear: A PET Investigation of Musical Imagery and Perception
Zatorre, R. J., Halpern, A. R., Perry, D. W., Meyer, E., & Evans, A. C. (1996) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1996.8.1.29 · Cited by 439
Brain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297
Mental imagery of speech and movement implicates the dynamics of internal forward models
Tian, X. (2010) · Frontiers in Psychology · doi:10.3389/fpsyg.2010.00166 · Cited by 259
Research works printed but not linked: 65
- 10.1126/science.2063199 · DOI · Paper page
- 10.1098/rspb.1995.0204 · DOI · Paper page
- 10.1152/jn.91128.2008 · DOI · Paper page
- 10.1523/JNEUROSCI.19-18-08036.1999 · DOI · Paper page
- 10.1523/JNEUROSCI.3256-10.2011 · DOI
- 2015, Progress in speech decoding from the electrocorticogram, Biomed. Eng. Lett., 5, 10, 10.1007/s13534-015-0175-1 · DOI
- 2016, Automatic speech recognition from neural signals: a focused review, Front. Neurosci., 10, 429
- 2017, Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication, Brain-Computer Interfaces, 4, 186, 10.1080/2326263X.2017.1330611 · DOI · Paper page
- 10.1016/S0079-6123(05)50034-7 · DOI · Paper page
- 2011, Using the electrocorticographic speech network to control a brain– computer interface in humans, J. Neural Eng., 8, 36004, 10.1088/1741-2560/8/3/036004 · DOI · Paper page
- 10.1016/j.cub.2015.08.030 · DOI · Paper page
- 10.1038/nature11020 · DOI · Paper page
- O’Sullivan, J. A. et al. Attentional Selection in a Cocktail Party Environment Can Be Decoded from Single-Trial EEG. Cereb. Cortex bht355 (2014). · DOI · Paper page
- 10.1073/pnas.1205381109 · DOI · Paper page
- 10.1073/pnas.1617622114 · DOI · Paper page
- 2016, Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity, J. Neural Eng., 13, 56004, 10.1088/1741-2560/13/5/056004 · DOI · Paper page
- 10.1523/JNEUROSCI.2383-16.2017 · DOI
- 2011, Should the parameters of a BCI translation algorithm be continually adapted?, J. Neurosci. Methods, 199, 103, 10.1016/j.jneumeth.2011.04.037 · DOI · Paper page
- 2012, Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, Signal Process. Mag. IEEE, 29, 82, 10.1109/MSP.2012.2205597 · DOI · Paper page
- 2018, Speaker-Independent Speech Separation With Deep Attractor Network, IEEE/ACM Trans. Audio, Speech, Lang. Process., 26, 787, 10.1109/TASLP.2018.2795749 · DOI · Paper page
- 10.1109/ICASSP.2017.7952155 · DOI · Paper page
- 2017, Neural decoding of attentional selection in multi-speaker environments without access to clean sources, J. Neural Eng., 14, 56001, 10.1088/1741-2552/aa7ab4 · DOI · Paper page
- 10.1523/JNEUROSCI.5249-08.2009 · DOI · Paper page
- 10.1073/pnas.1318017111 · DOI
- 10.1016/S1388-2457(00)00545-9 · DOI · Paper page
- 10.1152/jn.90954.2008 · DOI · Paper page
- LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. in Advances in neural information processing systems 396–404 (1990).
- Krizhevsky, A. , Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. in Advances in neural information processing systems 1097–1105 (2012).
- 10.1214/12-AOS1000. 1102.4807 · DOI · Paper page
- Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv Prepr. arXiv1502.03167 (2015).
- 10.1121/1.1945807 · DOI · Paper page
- 2006, Discrimination of speech from nonspeech based on multiscale spectro-temporal modulations, IEEE Trans. Audio. Speech. Lang. Processing, 14, 920, 10.1109/TSA.2005.858055 · DOI · Paper page
- 2016, WORLD: a vocoder-based high-quality speech synthesis system for real-time applications, IEICE Trans. Inf. Syst., 99, 1877
- 10.1016/S0167-6393(98)00085-5 · DOI · Paper page
- 10.1126/science.1127647 · DOI · Paper page
- Paul, D. B. & Baker, J. M. The design for the Wall Street Journal-based CSR corpus. in Proceedings of the workshop on Speech and Natural Language 357–362 (Association for Computational Linguistics, 1992). · DOI · Paper page
- 1996, MOS and pair comparison combined methods for quality evaluation of text-to-speech systems, Acta Acust. united with Acust., 82, 650
- 2016, An Algorithm for Predicting the Intelligibility of Speech Masked by Modulated Noise Maskers, IEEE/ACM Trans. Audio, Speech Lang. Process., 24, 2009, 10.1109/TASLP.2016.2585878 · DOI · Paper page
- 10.1016/S0079-6123(06)65031-0 · DOI · Paper page
- Yang, M. et al. Speech reconstruction from human auditory cortex with deep neural networks. In Sixteenth Annual Conference of the International Speech Communication Association (2015). · DOI
- 2008, Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids, Conf. Proc. IEEE Eng. Med. Biol. Soc., 2008, 4964
- 10.3389/fnhum.2015.00097 · DOI · Paper page
- Herff, C. et al. Towards direct speech synthesis from ECoG: A pilot study. in Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th Annual International Conference of the 1540–1543 (IEEE, 2016). · DOI · Paper page
- 10.1162/neco.2006.18.7.1527 · DOI · Paper page
- 2016, EEG-based prediction of driver’s cognitive performance by deep convolutional neural network, Signal Process. Image Commun., 47, 549, 10.1016/j.image.2016.05.018 · DOI · Paper page
- 10.1097/01.wco.0000218233.60217.84 · DOI · Paper page
- 10.2307/2285453 · doi:10.2307/2285453
- 10.1007/s10827-009-0196-9 · DOI · Paper page
- 10.1016/j.neuron.2007.06.004 · DOI · Paper page
- 10.1016/j.neuroimage.2005.03.013 · DOI · Paper page
- Martin, S. et al. Neural encoding of auditory features during music perception and imagery. Cereb. Cortex 1–12 (2017). · DOI · Paper page
- 10.1038/nn.3905 · DOI · Paper page
- 10.1088/1741-2560/3/1/R02 · DOI · Paper page
- Groppe, D. M. et al. iELVis: An open source MATLAB toolbox for localizing and visualizing human intracranial electrode data. 281, · DOI · Paper page
- 2006, BioImage Suite: An integrated medical image analysis suite: An update, Insight J., 2006, 209
- 10.1093/cercor/bhg087 · DOI · Paper page
- 10.1109/18.119739 · DOI · Paper page
- Coates, A. & Ng, A. Y. Selecting receptive fields in deep networks. in Advances in Neural Information Processing Systems 2528–2536 (2011).
- 2016, TensorFlow: A System for Large-Scale Machine Learning, OSDI, 16, 265
- He, K. , Zhang, X. , Ren, S. & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. in Proceedings of the IEEE international conference on computer vision 1026–1034 (2015). · DOI · Paper page
- 2013, Rectifier nonlinearities improve neural network acoustic models, Proc. icml, 30, 3
- Clevert, D.-A. , Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). arXiv Prepr. arXiv1511.07289 (2015).
- Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv Prepr. arXiv1412.6980 (2014).
- 2016, Spatial resolution dependence on spectral frequency in human speech cortex electrocorticography, J. Neural Eng., 13, 56013, 10.1088/1741-2560/13/5/056013 · DOI · Paper page
- 10.1109/ICASSP.2017.7952275 · DOI
Cited by
Speech synthesis from neural decoding of spoken sentences
Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019) · Nature · doi:10.1038/s41586-019-1119-1 · Cited by 883
Angrick, M., Ottenhoff, M. C., Diener, L., Ivucic, D., Ivucic, G., Goulis, S., Saal, J., Colon, A. J., Wagner, L., Krusienski, D. J., et al. (2021) · Communications Biology · doi:10.1038/s42003-021-02578-0 · Cited by 125
Brain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication
Luo, S., Rabbani, Q., & Crone, N. E. (2022) · Neurotherapeutics · doi:10.1007/s13311-022-01190-2 · Cited by 106
High-resolution neural recordings improve the accuracy of speech decoding
Duraivel, S., Rahimpour, S., Chiang, C.-H., Trumpis, M., Wang, C., Barth, K., Harward, S. C., Lad, S. P., Friedman, A. H., Southwell, D. G., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-42555-1 · Cited by 75
Rainey, S., Martin, S., Christen, A., Mégevand, P., & Fourneret, E. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00218-0 · Cited by 65
Metadata
- Authors
- Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2018-06-19
- OpenAlex
- W2809140860
- Cited by
- 43
- References
- 19
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards reconstructing intelligible speech from the human auditory cortex. https://app.cochat.ai/papers/10.1101/350124
- The paper
- Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N. (2018). Towards reconstructing intelligible speech from the human auditory cortex. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/350124
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.