Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
A direct-speech brain-computer interface (DS-BCI) acquires neural signals corresponding to imagined speech, then processes and decodes these signals to produce a linguistic output in the form of phonemes, words, or sentences. Recent research has shown the potential of neurolinguistics to enhance decoding approaches to imagined speech with the inclusion of semantics and phonology in experimental procedures. As neurolinguistics research findings are beginning to be incorporated within the scope of DS-BCI research, it is our view that a thorough understanding of imagined speech, and its relationship with overt speech, must be considered an integral feature of research in this field. With a focus on imagined speech, we provide a review of the most important neurolinguistics research informing the field of DS-BCI and suggest how this research may be utilized to improve current experimental protocols and decoding techniques. Our review of the literature supports a cross-disciplinary approach to DS-BCI research, in which neurolinguistics concepts and methods are utilized to aid development of a naturalistic mode of communication.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Brain–computer interface–based communication in the completely locked-in State
Chaudhary, U., Xia, B., Silvoni, S., Cohen, L. G., & Birbaumer, N. (2017) · PLOS Biology · doi:10.1371/journal.pbio.1002593 · Cited by 235
RetractedThe cortical organization of speech processing
Hickok, G., & Poeppel, D. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2113 · Cited by 5,495
Reach and grasp by people with tetraplegia using a neurally controlled robotic arm
Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., Haddadin, S., Liu, J., Cash, S. S., van der Smagt, P., et al. (2012) · Nature · doi:10.1038/nature11076 · Cited by 2,719 · cited 1×
“Although microelectrode arrays have shown good performance in fields such as neuromotor prostheses (e.g., Hochberg et al., 2012), relatively few studies have utilized them for recording the spiking activity of single or multiple units (SU or MU), i.e., neurons, during imagined speech.”
BackgroundTowards a neural basis of auditory sentence processing
Friederici, A. D. (2002) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01839-8 · Cited by 2,231 · cited 2×
“It has been proposed (Frazier, 1987), and countered (Friederici, 2002), that there is an isolated syntactic processing mechanism that has no relation to semantics or other non-syntactic information.”
BackgroundThe neural correlates of the verbal component of working memory
Paulesu, E., Frith, C. D., & Frackowiak, R. S. J. (1993) · Nature · doi:10.1038/362342a0 · Cited by 2,230 · cited 1×
“Activation of Broca area during imagined speech indicates that this typical language region is associated with its production and is consistent with results from functional imaging studies examining silent articulation (Paulesu et al., 1993).”
ResultPrice, C. J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.04.062 · Cited by 2,159
The spatial and temporal signatures of word production components
Indefrey, P., & Levelt, W. J. M. (2004) · Cognition · doi:10.1016/j.cognition.2002.06.001 · Cited by 2,144
Word length and the structure of short-term memory
Baddeley, A. D., Thomson, N., & Buchanan, M. (1975) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(75)80045-4 · Cited by 1,980
Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing.
Vigneau, M., Beaucousin, V., Hervé, P. Y., Duffau, H., Crivello, F., Houdé, O., Mazoyer, B., & Tzourio-Mazoyer, N. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2005.11.002 · Cited by 1,851
Brain mechanisms linking language and action
Pulvermüller, F. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1706 · Cited by 1,828 · cited 1×
“Results from action word studies have indicated that semantic processing can engage many different cortical areas, with Pulvermu¨ller (2005) stating that this contradicts the view that processing of meaning is concentrated in a single cortical location.”
BackgroundAn integrated theory of language production and comprehension
Pickering, M. J., & Garrod, S. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12001495 · Cited by 1,569
Voxel-based lesion-symptom mapping
Bates, E., Wilson, S. M., Saygin, A. P., Dick, F., Sereno, M. I., Knight, R. T., & Dronkers, N. F. (2003) · Nature Neuroscience · doi:10.1038/nn1050 · Cited by 1,457 · cited 1×
“Investigations into the neural correlates of language function and competence commonly employ functional imaging approaches (see Indefrey and Levelt, 2004), as well as LSM, to determine the links between linguistic pathologies and corresponding lesion sites in aphasics (Bates et al., 2003).”
MethodPulvermüller, F. (1999) · Behavioral and Brain Sciences · doi:10.1017/s0140525x9900182x · Cited by 1,284
Restoring cortical control of functional movement in a human with quadriplegia
Bouton, C. E., Shaikhouni, A., Annetta, N. V., Bockbrader, M. A., Friedenberg, D. A., Nielson, D. M., Sharma, G., Sederberg, P. B., Glenn, B. C., Mysiw, W. J., et al. (2016) · Nature · doi:10.1038/nature17435 · Cited by 924
Badre, D., Poldrack, R. A., Paré-Blagoev, E. J., Insler, R. Z., & Wagner, A. D. (2005) · Neuron · doi:10.1016/j.neuron.2005.07.023 · Cited by 914
Inner Speech: Development, Cognitive Functions, Phenomenology, and Neurobiology
Alderson-Day, B., & Fernyhough, C. (2015) · Psychological Bulletin · doi:10.1037/bul0000021 · Cited by 864 · cited 9×
“Research has demonstrated that imagined speech involves many cognitive functions, including learning (Alderson-Day and Fernyhough, 2015), task production (Dolcos and Albarracin, 2014), and memory (Perrone-Bertolotti et al.”
BackgroundInfluentialComputational neuroanatomy of speech production
Hickok, G. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3158 · Cited by 837 · cited 6×
“These are the word (lemma) level and the phonological level (Hickok, 2012).”
BackgroundInfluentialThe Spatial and Temporal Signatures of Word Production Components: A Critical Update
Indefrey, P. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00255 · Cited by 767 · cited 4×
“EEG studies have been used to study the time courses associated with the processing stages in word production (see Indefrey, 2011 for review).”
BackgroundMethodInfluentialEvolution, brain, and the nature of language
Berwick, R. C., Friederici, A. D., Chomsky, N., & Bolhuis, J. J. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.12.002 · Cited by 583 · cited 2×
“Neuroanatomical Regions Associated with Imagined Speech Production The diagram depicts brain regions typically associated with language function in the left hemisphere (Berwick et al., 2013), with each of the numbered sections indicating one of Brodmann areas (BA).”
BackgroundPandarinath, C., Nuyujukian, P., Blabe, C. H., Sorice, B. L., Saab, J., Willett, F. R., Hochberg, L. R., Shenoy, K. V., & Henderson, J. M. (2017) · eLife · doi:10.7554/elife.18554 · Cited by 566 · cited 3×
“In comparison with many of the previous incarnations of communicative BCI (Chaudhary et al., 2017; Pandarinath et al., 2017), the character of the modality of interaction, i.”
BackgroundResultInfluentialMental Concerts: Musical Imagery and Auditory Cortex
Zatorre, R. J., & Halpern, A. R. (2005) · Neuron · doi:10.1016/j.neuron.2005.06.013 · Cited by 420
The role of inner speech in task switching: A dual-task investigation
Emerson, M. J., & Miyake, A. (2003) · Journal of Memory and Language · doi:10.1016/s0749-596x(02)00511-9 · Cited by 348 · cited 1×
“Although imagined speech is not normally associated with executive control processes, the role of imagined speech in task switching, for example, switching attention across multiple arithmetic problems, has been studied (Emerson and Miyake, 2003).”
BackgroundPerrone-Bertolotti, M., Rapin, L., Lachaux, J.-P., Baciu, M., & Lœvenbruck, H. (2014) · Behavioural Brain Research · doi:10.1016/j.bbr.2013.12.034 · Cited by 317
Miyake, A., Emerson, M. J., Padilla, F., & Ahn, J.-C. (2004) · Acta Psychologica · doi:10.1016/j.actpsy.2003.12.004 · Cited by 317
Auditory imagery: Empirical findings
Hubbard, T. L. (2010) · Psychological Bulletin · doi:10.1037/a0018436 · Cited by 310 · cited 12×
“Indefrey and Lev-elt (2004) reviewed 82 functional imaging studies of single word production, identifying 11 regions in the left hemisphere (posterior IFG, ventral precentral gyrus, SMA, mid-and posterior STG and MTG, posterior temporal fusiform gyrus, anterior insula, thalamus, and medial cerebellum) and four in the…”
BackgroundInfluentialBrain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297 · cited 5×
“, 2013), and syntax (Herff et al., 2015) each showing some potential to aid classification in DS-BCI.”
BackgroundMethodInfluentialMonitoring the Time Course of Phonological Encoding
Wheeldon, L. R., & Levelt, W. J. M. (1995) · Journal of Memory and Language · doi:10.1006/jmla.1995.1014 · Cited by 286
Functional anatomy of inner speech and auditory verbal imagery
McGuire, P. K., Silbersweig, D. A., Murray, R. M., David, A. S., Frackowiak, R. S. J., & Frith, C. D. (1996) · Psychological Medicine · doi:10.1017/s0033291700033699 · Cited by 277
Functional heterogeneity of left inferior frontal cortex as revealed by fMRI
Paulesu, E., Goldacre, B., Scifo, P., Cappa, S. F., Gilardi, M. C., Castiglioni, I., Perani, D., & Fazio, F. (1997) · Neuroreport · doi:10.1097/00001756-199705260-00042 · Cited by 257
A functional study of auditory verbal imagery
SHERGILL, S. S., BULLMORE, E. T., BRAMMER, M. J., WILLIAMS, S. C. R., MURRAY, R. M., & McGUIRE, P. K. (2001) · Psychological Medicine · doi:10.1017/s003329170100335x · Cited by 237
Pei, X., Barbour, D. L., Leuthardt, E. C., & Schalk, G. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/4/046028 · Cited by 237
Alien voices and inner dialogue: towards a developmental account of auditory verbal hallucinations
Fernyhough, C. (2004) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2004.09.001 · Cited by 236
Decoding spoken words using local field potentials recorded from the cortical surface
Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., & Greger, B. (2010) · Journal of Neural Engineering · doi:10.1088/1741-2560/7/5/056007 · Cited by 235
Inner speech slips exhibit lexical bias, but not the phonemic similarity effect
Oppenheim, G. M., & Dell, G. S. (2008) · Cognition · doi:10.1016/j.cognition.2007.02.006 · Cited by 216 · cited 6×
“Limitations to the scope of empirical study in the case of imagined speech has induced the development of methods for indirect study of the phenomenon (Filik and Barber, 2011; Oppenheim and Dell, 2008).”
BackgroundMethodInfluentialComparing cortical activations for silent and overt speech using event-related fMRI
Huang, J., Carr, T. H., & Cao, Y. (2002) · Human Brain Mapping · doi:10.1002/hbm.1060 · Cited by 216
An event-related fMRI study of overt and covert word stem completion
Palmer, E. D., Rosen, H. J., Ojemann, J. G., Buckner, R. L., Kelley, W. M., & Petersen, S. E. (2001) · NeuroImage · doi:10.1006/nimg.2001.0779 · Cited by 216
Wildgruber, D., Ackermann, H., & Grodd, W. (2001) · NeuroImage · doi:10.1006/nimg.2000.0672 · Cited by 206
An fMRI investigation of covertly and overtly produced mono- and multisyllabic words
SHUSTER, L., & LEMIEUX, S. (2005) · Brain and Language · doi:10.1016/j.bandl.2004.07.007 · Cited by 205
Pei, X., Leuthardt, E. C., Gaona, C. M., Brunner, P., Wolpaw, J. R., & Schalk, G. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.10.029 · Cited by 204
Decoding spectrotemporal features of overt and covert speech from the human cortex
Martin, S., Brunner, P., Holdgraf, C., Heinze, H.-J., Crone, N. E., Rieger, J., Schalk, G., Knight, R. T., & Pasley, B. N. (2014) · Frontiers in Neuroengineering · doi:10.3389/fneng.2014.00014 · Cited by 200 · cited 5×
“The current stream of DS-BCI research indicates a trend toward improved classification of imagined speech units for decoded brain activity (González-Castañeda et al., 2017; Martin et al., 2014) and the development of methodologies for continuous decoding of imagined speech (Brumberg et al.”
BackgroundResultInfluentialMugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., Shih, J. J., Krusienski, D. J., & Slutzky, M. W. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/035015 · Cited by 192
Modulation of activity in temporal cortex during generation of inner speech
Shergill, S. S., Brammer, M. J., Fukuda, R., Bullmore, E., Amaro, E., Murray, R. M., & McGuire, P. K. (2002) · Human Brain Mapping · doi:10.1002/hbm.10046 · Cited by 182
The neural correlates of inner speech defined by voxel-based lesion-symptom mapping
Geva, S., Jones, P. S., Crinion, J. T., Price, C. J., Baron, J.-C., & Warburton, E. A. (2011) · Brain · doi:10.1093/brain/awr232 · Cited by 173
Tian, X., & Poeppel, D. (2013) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00381 · Cited by 172 · cited 1×
“It has been suggested that differences between the two conditions may be the result of additional motor elements of imagined speech, which involve the deployment of a somatosensory forward model (Tian and Poeppel, 2013).”
BackgroundWord pair classification during imagined speech using direct brain recordings
Martin, S., Brunner, P., Iturrate, I., Millán, J. D. R., Schalk, G., Knight, R. T., & Pasley, B. N. (2016) · Scientific Reports · doi:10.1038/srep25803 · Cited by 171
Toward a phenomenology of inner speaking
Hurlburt, R. T., Heavey, C. L., & Kelsey, J. M. (2013) · Consciousness and Cognition · doi:10.1016/j.concog.2013.10.003 · Cited by 166 · cited 1×
“Hurlburt, Heavey, and Kelsey (Hurlburt et al., 2013) state that both production and perception of imagined speech exhibit activations in regions such as the IFG, SMA, insula, and posterior STG (Hubbard, 2010; Price, 2012).”
BackgroundMotor movement matters: The flexible abstractness of inner speech
Oppenheim, G. M., & Dell, G. S. (2010) · Memory & Cognition · doi:10.3758/mc.38.8.1147 · Cited by 145
Jones, S. R., & Fernyhough, C. (2007) · Clinical Psychology Review · doi:10.1016/j.cpr.2006.10.001 · Cited by 142 · cited 2×
“A danger associated with these studies is the lack of ecological validity in eliciting imagined speech (Alderson-Day and Fernyhough, 2015) and the failure of researchers to acknowledge the possibility that imagined speech is present during baseline assessments (Jones and Fernyhough, 2007).”
BackgroundSelf-awareness and the left inferior frontal gyrus: Inner speech use during self-related processing
Morin, A., & Michaud, J. (2007) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2007.06.013 · Cited by 141
Hurlburt, R. T., Alderson-Day, B., Kühn, S., & Fernyhough, C. (2016) · PLOS ONE · doi:10.1371/journal.pone.0147932 · Cited by 131
Marvel, C. L., & Desmond, J. E. (2012) · Brain and Language · doi:10.1016/j.bandl.2011.08.005 · Cited by 130
Basho, S., Palmer, E. D., Rubio, M. A., Wulfeck, B., & Müller, R.-A. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.01.007 · Cited by 127 · cited 9×
“Nevertheless, there are significant differences in brain activity between the two processes (e.g., Basho et al., 2007).”
BackgroundInfluentialThe functional neuroanatomy of metrical stress evaluation of perceived and imagined spoken words
Aleman, A. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhh124 · Cited by 119 · cited 6×
“Increased activation has been observed in the left MTG and STG during the production of multisyllabic words in imagined speech trials (Shuster and Lemieux, 2005), and the posterior STG has been implicated inmetric stress evaluation in the phonological loop (Aleman et al., 2005) (see Figure 3).”
BackgroundResultInfluentialBrumberg (2011) · Frontiers in Neuroscience · doi:10.3389/fnins.2011.00065 · Cited by 114 · cited 10×
“The neural correlates of vowels and consonants (Idrees and Farooq, 2016; Pei et al., 2011b; Yoshimura et al., 2016), phonemes (Brumberg et al., 2011; Leuthardt et al., 2011), syllables (Denget al., 2010), whole words (González-Castañeda et al., 2017; Martin et al., 2016), and even sentences…”
BackgroundInfluentialThe brain's conversation with itself: neural substrates of dialogic inner speech
Alderson-Day, B., Weis, S., McCarthy-Jones, S., Moseley, P., Smailes, D., & Fernyhough, C. (2016) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsv094 · Cited by 94 · cited 4×
“Imagined speech’s close association with working memory (Marvel and Desmond, 2012), the range of articulatory forms it can take (Alderson-Day et al., 2015; Deng et al., 2010), and the different neural activations it exhibits in relation to overt speech (Basho et al.”
BackgroundInfluentialDolcos, S., & Albarracin, D. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2048 · Cited by 79
Error biases in inner and overt speech: evidence from tongue twisters
Corley, M., Brocklehurst, P. H., & Moat, H. S. (2011) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0021321 · Cited by 70 · cited 1×
“Several modalities have been developed to facilitate such communication through movement-independent BCI, including BCI-spellers (e.g., D’albis et al., 2012), BCIs based on steady-state visually evoked potential (e.g., Bin et al., 2009), and BCIs based on motor imagery (e.g., Tabar and Halici,…”
MethodGeva, S., Bennett, S., Warburton, E. A., & Patterson, K. (2011) · Aphasiology · doi:10.1080/02687038.2010.511236 · Cited by 67
Inner speech captures the perception of external speech
Scott, M., Yeung, H. H., Gick, B., & Werker, J. F. (2013) · The Journal of the Acoustical Society of America · doi:10.1121/1.4794932 · Cited by 62
Inner Speech during Silent Reading Reflects the Reader's Regional Accent
Filik, R., & Barber, E. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025782 · Cited by 62
Neural decoding of single vowels during covert articulation using electrocorticography
Ikeda, S., Shibata, T., Nakano, N., Okada, R., Tsuyuguchi, N., Ikeda, K., & Kato, A. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00125 · Cited by 52
Brocklehurst, P. H., & Corley, M. (2011) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2010.11.004 · Cited by 46
Cited by
Interface, interaction, and intelligence in generalized brain–computer interfaces
Gao, X., Wang, Y., Chen, X., & Gao, S. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.04.003 · Cited by 304
Imagined speech can be decoded from low- and cross-frequency intracranial EEG features
Proix, T., Delgado Saa, J., Christen, A., Martin, S., Pasley, B. N., Knight, R. T., Tian, X., Poeppel, D., Doyle, W. K., Devinsky, O., et al. (2022) · Nature Communications · doi:10.1038/s41467-021-27725-3 · Cited by 212
Representation of internal speech by single neurons in human supramarginal gyrus
Wandelt, S. K., Bjånes, D. A., Pejsa, K., Lee, B., Liu, C., & Andersen, R. A. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01867-y · Cited by 74
Soroush, P. Z., Herff, C., Ries, S. K., Shih, J. J., Schultz, T., & Krusienski, D. J. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.119913 · Cited by 48
Direct Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2022) · doi:10.1101/2022.08.02.502503 · Cited by 42
Metadata
- Authors
- Cooney, C., Folli, R., & Coyle, D.
- Venue
- iScience
- Publisher
- Elsevier BV
- Published
- 2018-10
- Volume
- 8
- Pages
- 103-125
- ISSN
- 2589-0042 (ISSN-L)
- OpenAlex
- W2893394967
- Cited by
- 106
- References
- 62
- FWCI
- 3.46
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.