Key considerations in designing a speech brain-computer interface
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
• Speech BCI technologies can be envisioned to restore speech from thoughts in aphasic people with preserved cortical areas. • The inferior frontal cortical region is a good candidate for a speech BCI. • Articulatory-based speech synthesis may be adequate for a speech BCI relying on frontal speech activity decoding. • Microelectrode array recordings is the technology that would provide the most detailed neural activity from speech networks. Restoring communication in case of aphasia is a key challenge for neurotechnologies. To this end, brain-computer strategies can be envisioned to allow artificial speech synthesis from the continuous decoding of neural signals underlying speech imagination. Such speech brain-computer interfaces do not exist yet and their design should consider three key choices that need to be made: the choice of appropriate brain regions to record neural activity from, the choice of an appropriate recording technique, and the choice of a neural decoding scheme in association with an appropriate speech synthesis method. These key considerations are discussed here in light of (1) the current understanding of the functional neuroanatomy of cortical areas underlying overt and covert speech production, (2) the available literature making use of a variety of brain recording techniques to better characterize and address the challenge of decoding cortical speech signals, and (3) the different speech synthesis approaches that can be considered depending on the level of speech representation (phonetic, acoustic or articulatory) envisioned to be decoded at the core of a speech BCI paradigm.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The cortical organization of speech processing
Hickok, G., & Poeppel, D. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2113 · Cited by 5,495 · cited 1×
“Speech processing by the human brain involves a wide cortical network, which has been modeled by two main information streams linking auditory areas of the superior temporal plane to articulatory areas of frontal regions, one ventral and the other dorsal (Hickok and Poeppel, 2004, 2007).”
Neuronal ensemble control of prosthetic devices by a human with tetraplegia
Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., & Donoghue, J. P. (2006) · Nature · doi:10.1038/nature04970 · Cited by 3,295 · cited 1×
“…with the drawback to require invasive recordings, BCI systems based on intracortical signals seem to alleviate the subject fatigue, the external device becoming progressively embodied after a period of training (Hochberg et al., 2006, 2012; Collinger et al., 2013; Wodlinger et al., 2015).”
Farwell, L. A., & Donchin, E. (1988) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(88)90149-6 · Cited by 3,228
Reach and grasp by people with tetraplegia using a neurally controlled robotic arm
Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., Haddadin, S., Liu, J., Cash, S. S., van der Smagt, P., et al. (2012) · Nature · doi:10.1038/nature11076 · Cited by 2,719
Positron emission tomographic studies of the cortical anatomy of single-word processing
Petersen, S. E., Fox, P. T., Posner, M. I., Mintun, M., & Raichle, M. E. (1988) · Nature · doi:10.1038/331585a0 · Cited by 2,593 · cited 2×
“Functional imaging studies have shown that overt word repetition activates motor and premotor cortices bilaterally (Petersen et al., 1988, 1989; Palmer et al., 2001; Peeva et al., 2010; Cogan et al., 2014).”
Hickok, G., & Poeppel, D. (2004) · Cognition · doi:10.1016/j.cognition.2003.10.011 · Cited by 2,212
High-performance neuroprosthetic control by an individual with tetraplegia
Collinger, J. L., Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., McMorland, A. J., Velliste, M., Boninger, M. L., & Schwartz, A. B. (2013) · The Lancet · doi:10.1016/s0140-6736(12)61816-9 · Cited by 1,848 · cited 2×
“Recent advances in motor BCI have shown that, provided careful training, a ten of DoF could indeed be controlled from unit or multiunit activity recorded using microelectrode arrays (Collinger et al., 2013; Wodlinger et al., 2015).”
Functional organization of human sensorimotor cortex for speech articulation
Bouchard, K. E., Mesgarani, N., Johnson, K., & Chang, E. F. (2013) · Nature · doi:10.1038/nature11911 · Cited by 686 · cited 2×
“Several ECoG studies have shown that cortical oscillations are relevant correlates of speech processing (Leuthardt et al., 2011; Pei et al., 2011a; Pasley et al., 2012; Bouchard et al., 2013; Pasley and Knight, 2013; Martin et al., 2014; Mugler et al., 2014) (see also Fig.”
Speech rhythms and multiplexed oscillatory sensory coding in the human brain
Gross, J., Hoogenboom, N., Thut, G., Schyns, P., Panzeri, S., Belin, P., & Garrod, S. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001752 · Cited by 643
Reconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604 · cited 7×
“Such reconstruction was all the more accurate that the number of electrode sites was high, and the most informative sites were found to be in temporal auditory areas (Pasley et al., 2012).”
InfluentialWho “Is Saying What”? Brain-based decoding of human voice and speech
Formisano, E., De Martino, F., Bonte, M., & Goebel, R. (2008) · Science · doi:10.1126/science.1164318 · Cited by 599 · cited 2×
“…single trial fMRI can be used to successfully predict with an accuracy above chance level which of different speech items or types are perceived by subjects from their BOLD activity recorded in auditory areas (Formisano et al., 2008; Evans et al., 2013; Bonte et al., 2014; Correia et al., 2014).”
BackgroundMethodWodlinger, B., Downey, J. E., Tyler-Kabara, E. C., Schwartz, A. B., Boninger, M. L., & Collinger, J. L. (2015) · Journal of Neural Engineering · doi:10.1088/1741-2560/12/1/016011 · Cited by 536 · cited 2×
“Recent advances in motor BCI have shown that, provided careful training, a ten of DoF could indeed be controlled from unit or multiunit activity recorded using microelectrode arrays (Collinger et al., 2013; Wodlinger et al., 2015).”
Sequential Processing of Lexical, Grammatical, and Phonological Information Within Broca’s Area
Sahin, N. T., Pinker, S., Cash, S. S., Schomer, D., & Halgren, E. (2009) · Science · doi:10.1126/science.1174481 · Cited by 452
Tate, M. C., Herbet, G., Moritz-Gasser, S., Tate, J. E., & Duffau, H. (2014) · Brain · doi:10.1093/brain/awu168 · Cited by 377
Representation of the temporal envelope of sounds in the human brain
Giraud, A.-L., Lorenzi, C., Ashburner, J., Wable, J., Johnsrude, I., Frackowiak, R., & Kleinschmidt, A. (2000) · Journal of Neurophysiology · doi:10.1152/jn.2000.84.3.1588 · Cited by 353
Coupled neural systems underlie the production and comprehension of naturalistic narrative speech
Silbert, L. J., Honey, C. J., Simony, E., Poeppel, D., & Hasson, U. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1323812111 · Cited by 352
Jarosiewicz, B., Sarma, A. A., Bacher, D., Masse, N. Y., Simeral, J. D., Sorice, B., Oakley, E. M., Blabe, C., Pandarinath, C., Gilja, V., et al. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aac7328 · Cited by 340
Perrone-Bertolotti, M., Rapin, L., Lachaux, J.-P., Baciu, M., & Lœvenbruck, H. (2014) · Behavioural Brain Research · doi:10.1016/j.bbr.2013.12.034 · Cited by 317
Brain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297 · cited 2×
“Although the design of such decoder and its use in a closed-loop BCI paradigm is still an unsolved issue, a recent study (Herff et al., 2015) reported encouraging results on the offline decoding of EcoG data, with a word-error-rate of 25%.”
Sensory–motor transformations for speech occur bilaterally
Cogan, G. B., Thesen, T., Carlson, C., Doyle, W., Devinsky, O., & Pesaran, B. (2014) · Nature · doi:10.1038/nature12935 · Cited by 273 · cited 2×
“Functional imaging studies have shown that overt word repetition activates motor and premotor cortices bilaterally (Petersen et al., 1988, 1989; Palmer et al., 2001; Peeva et al., 2010; Cogan et al., 2014).”
Pei, X., Barbour, D. L., Leuthardt, E. C., & Schalk, G. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/4/046028 · Cited by 237 · cited 3×
“To a lesser extent, ECoG data could also be used to predict silently articulated or covertly imagined speech not actually overtly pronounced by the subject (Pei et al., 2011a; Ikeda et al., 2014; Martin et al., 2014) (see also Fig.”
Decoding spoken words using local field potentials recorded from the cortical surface
Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., & Greger, B. (2010) · Journal of Neural Engineering · doi:10.1088/1741-2560/7/5/056007 · Cited by 235 · cited 2×
“Discriminating between 2 and 10 words could be achieved above chance level using discrete classification algorithms applied to ECoG neural features (Kellis et al., 2010), indicating that ECoG signals contain information that differs from word to word.”
An event-related fMRI study of overt and covert word stem completion
Palmer, E. D., Rosen, H. J., Ojemann, J. G., Buckner, R. L., Kelley, W. M., & Petersen, S. E. (2001) · NeuroImage · doi:10.1006/nimg.2001.0779 · Cited by 216 · cited 3×
“In general, inner speech has been found to activate similar brain areas but with a lesser amplitude than overt speech across most ventral and dorsal stream areas (Price et al., 1994; Ryding et al., 1996; Palmer et al., 2001; Shuster and Lemieux, 2005).”
An fMRI investigation of covertly and overtly produced mono- and multisyllabic words
SHUSTER, L., & LEMIEUX, S. (2005) · Brain and Language · doi:10.1016/j.bandl.2004.07.007 · Cited by 205 · cited 1×
“In general, inner speech has been found to activate similar brain areas but with a lesser amplitude than overt speech across most ventral and dorsal stream areas (Price et al., 1994; Ryding et al., 1996; Palmer et al., 2001; Shuster and Lemieux, 2005).”
BackgroundPei, X., Leuthardt, E. C., Gaona, C. M., Brunner, P., Wolpaw, J. R., & Schalk, G. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.10.029 · Cited by 204
Decoding spectrotemporal features of overt and covert speech from the human cortex
Martin, S., Brunner, P., Holdgraf, C., Heinze, H.-J., Crone, N. E., Rieger, J., Schalk, G., Knight, R. T., & Pasley, B. N. (2014) · Frontiers in Neuroengineering · doi:10.3389/fneng.2014.00014 · Cited by 200 · cited 6×
“To a lesser extent, ECoG data could also be used to predict silently articulated or covertly imagined speech not actually overtly pronounced by the subject (Pei et al., 2011a; Ikeda et al., 2014; Martin et al., 2014) (see also Fig.”
InfluentialMugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., Shih, J. J., Krusienski, D. J., & Slutzky, M. W. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/035015 · Cited by 192 · cited 3×
“…the speech motor cortex were also used to decode all phonemes of American English using discrete classification with a success rate of about 20% across 4 different subjects, this rate reaching 36% in one subject using 6 electrodes located over the ventral somatosensory region (Mugler et al., 2014).”
The neural correlates of inner speech defined by voxel-based lesion-symptom mapping
Geva, S., Jones, P. S., Crinion, J. T., Price, C. J., Baron, J.-C., & Warburton, E. A. (2011) · Brain · doi:10.1093/brain/awr232 · Cited by 173
Noninvasive brain-computer interface enables communication after brainstem stroke
Sellers, E. W., Ryan, D. B., & Hauser, C. K. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3007801 · Cited by 140
Peeva, M. G., Guenther, F. H., Tourville, J. A., Nieto-Castanon, A., Anton, J.-L., Nazarian, B., & Alario, F.-X. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.12.065 · Cited by 135 · cited 2×
“Functional imaging studies have shown that overt word repetition activates motor and premotor cortices bilaterally (Petersen et al., 1988, 1989; Palmer et al., 2001; Peeva et al., 2010; Cogan et al., 2014).”
Basho, S., Palmer, E. D., Rubio, M. A., Wulfeck, B., & Müller, R.-A. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.01.007 · Cited by 127
Brumberg (2011) · Frontiers in Neuroscience · doi:10.3389/fnins.2011.00065 · Cited by 114 · cited 1×
“%) from signals recorded with this single 2-channel microelectrode in the speech motor cortex of a locked-in syndrome individual (Brumberg et al., 2011).”
Effects of training and motivation on auditory P300 brain-computer interface performance
Baykara, E., Ruf, C. A., Fioravanti, C., Käthner, I., Simon, N., Kleih, S. C., Kübler, A., & Halder, S. (2016) · Clinical Neurophysiology · doi:10.1016/j.clinph.2015.04.054 · Cited by 91 · cited 1×
“Another major limitations of EEG-based BCI systems for communication is that they still require a high level of concentration of the subjects (Käthner et al., 2014; Baykara et al., 2016), imposing a high cognitive workload limiting their easy use over extensive periods of time.”
BackgroundCovert speech arrest induced by rTMS over both motor and nonmotor left hemisphere frontal sites
Aziz-Zadeh, L., Cattaneo, L., Rochat, M., & Rizzolatti, G. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929054021157 · Cited by 90 · cited 2×
“The left inferior frontal cortex has further been shown to be specifically activated during covert word retrieval (Hirshorn and Thompson-Schill, 2006) and to be important for inner speech production as assessed using repetitive transcranial magnetic stimulation (Aziz-Zadeh et al., 2005).”
Neural decoding of single vowels during covert articulation using electrocorticography
Ikeda, S., Shibata, T., Nakano, N., Okada, R., Tsuyuguchi, N., Ikeda, K., & Kato, A. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00125 · Cited by 52 · cited 1×
“To a lesser extent, ECoG data could also be used to predict silently articulated or covertly imagined speech not actually overtly pronounced by the subject (Pei et al., 2011a; Ikeda et al., 2014; Martin et al., 2014) (see also Fig.”
Cited by
Speech synthesis from neural decoding of spoken sentences
Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019) · Nature · doi:10.1038/s41586-019-1119-1 · Cited by 883
Speech synthesis from ECoG using densely connected 3D convolutional neural networks
Angrick, M., Herff, C., Mugler, E., Tate, M. C., Slutzky, M. W., Krusienski, D. J., & Schultz, T. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab0c59 · Cited by 225
Imagined speech can be decoded from low- and cross-frequency intracranial EEG features
Proix, T., Delgado Saa, J., Christen, A., Martin, S., Pasley, B. N., Knight, R. T., Tian, X., Poeppel, D., Doyle, W. K., Devinsky, O., et al. (2022) · Nature Communications · doi:10.1038/s41467-021-27725-3 · Cited by 212
Decoding Covert Speech From EEG-A Comprehensive Review
Panachakel, J. T., & Ramakrishnan, A. G. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.642251 · Cited by 156
Direct Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2022) · doi:10.1101/2022.08.02.502503 · Cited by 42
Metadata
- Authors
- Bocquelet, F., Hueber, T., Girin, L., Chabardès, S., & Yvert, B.
- Venue
- Journal of Physiology-Paris
- Publisher
- Elsevier BV
- Published
- 2016-11
- Volume
- 110
- Issue
- 4
- Pages
- 392-401
- ISSN
- 0928-4257 (ISSN-L)
- OpenAlex
- W2742210132
- Cited by
- 73
- References
- 35
- FWCI
- 1.57
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.