CoChat

Towards reconstructing intelligible speech from the human auditory cortex

Open access
Akbari, Hassan, Khalighinejad, Bahar, Herrero, Jose L., Mehta, Ashesh D., Mesgarani, Nima · 2018 · bioRxiv (Cold Spring Harbor Laboratory)

Preprint — published as 10.1038/s41598-018-37359-z · Paper page

PreprintCited by 43

The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.

Abstract

Auditory stimulus reconstruction is a technique that finds the best approximation of the acoustic stimulus from the population of evoked neural activity. Reconstructing speech from the human auditory cortex creates the possibility of a speech neuroprosthetic to establish a direct communication with the brain and has been shown to be possible in both overt and covert conditions. However, the low quality of the reconstructed speech has severely limited the utility of this method for brain-computer interface (BCI) applications. To advance the state-of-the-art in speech neuroprosthesis, we combined the recent advances in deep learning with the latest innovations in speech synthesis technologies to reconstruct closed-set intelligible speech from the human auditory cortex. We investigated the dependence of reconstruction accuracy on linear and nonlinear (deep neural network) regression methods and the acoustic representation that is used as the target of reconstruction, including auditory spectrogram and speech synthesis parameters. In addition, we compared the reconstruction accuracy from low and high neural frequency ranges. Our results show that a deep neural network model that directly estimates the parameters of a speech synthesizer from all neural frequencies achieves the highest subjective and objective scores on a digit recognition task, improving the intelligibility by 65% over the baseline method which used linear regression to reconstruct the auditory spectrogram. These results demonstrate the efficacy of deep learning and speech synthesis algorithms for designing the next generation of speech BCI systems, which not only can restore communications for paralyzed patients but also have the potential to transform human-computer interaction technologies.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/350124

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 19 references

Research works printed but not linked: 65

  1. 10.1126/science.2063199 · DOI · Paper page
  2. 10.1098/rspb.1995.0204 · DOI · Paper page
  3. 10.1152/jn.91128.2008 · DOI · Paper page
  4. 10.1523/JNEUROSCI.19-18-08036.1999 · DOI · Paper page
  5. 10.1523/JNEUROSCI.3256-10.2011 · DOI
  6. 2015, Progress in speech decoding from the electrocorticogram, Biomed. Eng. Lett., 5, 10, 10.1007/s13534-015-0175-1 · DOI
  7. 2016, Automatic speech recognition from neural signals: a focused review, Front. Neurosci., 10, 429
  8. 2017, Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication, Brain-Computer Interfaces, 4, 186, 10.1080/2326263X.2017.1330611 · DOI · Paper page
  9. 10.1016/S0079-6123(05)50034-7 · DOI · Paper page
  10. 2011, Using the electrocorticographic speech network to control a brain– computer interface in humans, J. Neural Eng., 8, 36004, 10.1088/1741-2560/8/3/036004 · DOI · Paper page
  11. 10.1016/j.cub.2015.08.030 · DOI · Paper page
  12. 10.1038/nature11020 · DOI · Paper page
  13. O’Sullivan, J. A. et al. Attentional Selection in a Cocktail Party Environment Can Be Decoded from Single-Trial EEG. Cereb. Cortex bht355 (2014). · DOI · Paper page
  14. 10.1073/pnas.1205381109 · DOI · Paper page
  15. 10.1073/pnas.1617622114 · DOI · Paper page
  16. 2016, Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity, J. Neural Eng., 13, 56004, 10.1088/1741-2560/13/5/056004 · DOI · Paper page
  17. 10.1523/JNEUROSCI.2383-16.2017 · DOI
  18. 2011, Should the parameters of a BCI translation algorithm be continually adapted?, J. Neurosci. Methods, 199, 103, 10.1016/j.jneumeth.2011.04.037 · DOI · Paper page
  19. 2012, Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, Signal Process. Mag. IEEE, 29, 82, 10.1109/MSP.2012.2205597 · DOI · Paper page
  20. 2018, Speaker-Independent Speech Separation With Deep Attractor Network, IEEE/ACM Trans. Audio, Speech, Lang. Process., 26, 787, 10.1109/TASLP.2018.2795749 · DOI · Paper page
  21. 10.1109/ICASSP.2017.7952155 · DOI · Paper page
  22. 2017, Neural decoding of attentional selection in multi-speaker environments without access to clean sources, J. Neural Eng., 14, 56001, 10.1088/1741-2552/aa7ab4 · DOI · Paper page
  23. 10.1523/JNEUROSCI.5249-08.2009 · DOI · Paper page
  24. 10.1073/pnas.1318017111 · DOI
  25. 10.1016/S1388-2457(00)00545-9 · DOI · Paper page
  26. 10.1152/jn.90954.2008 · DOI · Paper page
  27. LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. in Advances in neural information processing systems 396–404 (1990).
  28. Krizhevsky, A. , Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. in Advances in neural information processing systems 1097–1105 (2012).
  29. 10.1214/12-AOS1000. 1102.4807 · DOI · Paper page
  30. Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv Prepr. arXiv1502.03167 (2015).
  31. 10.1121/1.1945807 · DOI · Paper page
  32. 2006, Discrimination of speech from nonspeech based on multiscale spectro-temporal modulations, IEEE Trans. Audio. Speech. Lang. Processing, 14, 920, 10.1109/TSA.2005.858055 · DOI · Paper page
  33. 2016, WORLD: a vocoder-based high-quality speech synthesis system for real-time applications, IEICE Trans. Inf. Syst., 99, 1877
  34. 10.1016/S0167-6393(98)00085-5 · DOI · Paper page
  35. 10.1126/science.1127647 · DOI · Paper page
  36. Paul, D. B. & Baker, J. M. The design for the Wall Street Journal-based CSR corpus. in Proceedings of the workshop on Speech and Natural Language 357–362 (Association for Computational Linguistics, 1992). · DOI · Paper page
  37. 1996, MOS and pair comparison combined methods for quality evaluation of text-to-speech systems, Acta Acust. united with Acust., 82, 650
  38. 2016, An Algorithm for Predicting the Intelligibility of Speech Masked by Modulated Noise Maskers, IEEE/ACM Trans. Audio, Speech Lang. Process., 24, 2009, 10.1109/TASLP.2016.2585878 · DOI · Paper page
  39. 10.1016/S0079-6123(06)65031-0 · DOI · Paper page
  40. Yang, M. et al. Speech reconstruction from human auditory cortex with deep neural networks. In Sixteenth Annual Conference of the International Speech Communication Association (2015). · DOI
  41. 2008, Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids, Conf. Proc. IEEE Eng. Med. Biol. Soc., 2008, 4964
  42. 10.3389/fnhum.2015.00097 · DOI · Paper page
  43. Herff, C. et al. Towards direct speech synthesis from ECoG: A pilot study. in Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th Annual International Conference of the 1540–1543 (IEEE, 2016). · DOI · Paper page
  44. 10.1162/neco.2006.18.7.1527 · DOI · Paper page
  45. 2016, EEG-based prediction of driver’s cognitive performance by deep convolutional neural network, Signal Process. Image Commun., 47, 549, 10.1016/j.image.2016.05.018 · DOI · Paper page
  46. 10.1097/01.wco.0000218233.60217.84 · DOI · Paper page
  47. 10.2307/2285453 · doi:10.2307/2285453
  48. 10.1007/s10827-009-0196-9 · DOI · Paper page
  49. 10.1016/j.neuron.2007.06.004 · DOI · Paper page
  50. 10.1016/j.neuroimage.2005.03.013 · DOI · Paper page
  51. Martin, S. et al. Neural encoding of auditory features during music perception and imagery. Cereb. Cortex 1–12 (2017). · DOI · Paper page
  52. 10.1038/nn.3905 · DOI · Paper page
  53. 10.1088/1741-2560/3/1/R02 · DOI · Paper page
  54. Groppe, D. M. et al. iELVis: An open source MATLAB toolbox for localizing and visualizing human intracranial electrode data. 281, · DOI · Paper page
  55. 2006, BioImage Suite: An integrated medical image analysis suite: An update, Insight J., 2006, 209
  56. 10.1093/cercor/bhg087 · DOI · Paper page
  57. 10.1109/18.119739 · DOI · Paper page
  58. Coates, A. & Ng, A. Y. Selecting receptive fields in deep networks. in Advances in Neural Information Processing Systems 2528–2536 (2011).
  59. 2016, TensorFlow: A System for Large-Scale Machine Learning, OSDI, 16, 265
  60. He, K. , Zhang, X. , Ren, S. & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. in Proceedings of the IEEE international conference on computer vision 1026–1034 (2015). · DOI · Paper page
  61. 2013, Rectifier nonlinearities improve neural network acoustic models, Proc. icml, 30, 3
  62. Clevert, D.-A. , Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). arXiv Prepr. arXiv1511.07289 (2015).
  63. Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv Prepr. arXiv1412.6980 (2014).
  64. 2016, Spatial resolution dependence on spectral frequency in human speech cortex electrocorticography, J. Neural Eng., 13, 56013, 10.1088/1741-2560/13/5/056013 · DOI · Paper page
  65. 10.1109/ICASSP.2017.7952275 · DOI

Cited by

Metadata

Authors
Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N.
Venue
bioRxiv (Cold Spring Harbor Laboratory)
Publisher
openRxiv
Published
2018-06-19
OpenAlex
W2809140860
Cited by
43
References
19
Type
posted content
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.