Recommendations for promoting user agency in the design of speech neuroprostheses
The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Brain-computer interfaces (BCI) that directly decode speech from brain activity aim to restore communication in people with paralysis who cannot speak. Despite recent advances, neural inference of speech remains imperfect, limiting the ability for speech BCIs to enable experiences such as fluent conversation that promote agency – that is, the ability for users to author and transmit messages enacting their intentions. Here, we make recommendations for promoting agency based on existing and emerging strategies in neural engineering. The focus is on achieving fast, accurate, and reliable performance while ensuring volitional control over when a decoder is engaged, what exactly is decoded, and how messages are expressed. Additionally, alongside neuroscientific progress within controlled experimental settings, we argue that a parallel line of research must consider how to translate experimental successes into real-world environments. While such research will ultimately require input from prospective users, here we identify and describe design choices inspired by human-factors work conducted in existing fields of assistive technology, which address practical issues likely to emerge in future real-world speech BCI applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PENFIELD, W., & BOLDREY, E. (1937) · Brain · doi:10.1093/brain/60.4.389 · Cited by 4,188
Natural speech reveals the semantic maps that tile human cerebral cortex
Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016) · Nature · doi:10.1038/nature17637 · Cited by 1,504
High-performance brain-to-text communication via handwriting
Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M., & Shenoy, K. V. (2021) · Nature · doi:10.1038/s41586-021-03506-2 · Cited by 1,024
Speech synthesis from neural decoding of spoken sentences
Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019) · Nature · doi:10.1038/s41586-019-1119-1 · Cited by 883
Self-citationComputational neuroanatomy of speech production
Hickok, G. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3158 · Cited by 837 · cited 1×
“It is well established that spoken language results from a series of hierarchically-organized neural processing steps (Hickok, 2012).”
BackgroundNeuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria
Moses, D. A., Metzger, S. L., Liu, J. R., Anumanchipalli, G. K., Makin, J. G., Sun, P. F., Chartier, J., Dougherty, M. E., Liu, P. M., Abrams, G. M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2027540 · Cited by 582
Self-citationA high-performance speech neuroprosthesis
Willett, F. R., Kunz, E. M., Fan, C., Avansino, D. T., Wilson, G. H., Choi, E. Y., Kamdar, F., Glasser, M. F., Hochberg, L. R., Druckmann, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06377-x · Cited by 581
A high-performance neuroprosthesis for speech decoding and avatar control
Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., Dougherty, M. E., Liu, J. R., Wu, P., Berger, M. A., et al. (2023) · Nature · doi:10.1038/s41586-023-06443-4 · Cited by 522
Self-citationSemantic reconstruction of continuous language from non-invasive brain recordings
Tang, J., LeBel, A., Jain, S., & Huth, A. G. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01304-9 · Cited by 442
Perspectives of speech language pathologists regarding success versus abandonment of AAC
Johnson, J. M., Inglebret, E., Jones, C., & Ray, J. (2006) · Augmentative and Alternative Communication · doi:10.1080/07434610500483588 · Cited by 307
Brain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297 · cited 1×
“…the majority of speech-BCI research has focused on decoding low-level articulatory and phonological signals within sensorimotor and auditory regions of the cortex (Martin et al., 2014; Herff et al., 2015; Ramsey et al., 2018; Anumanchipalli et al., 2019; Moses et al., 2021; Proix et al., 2022).”
BackgroundAAC for adults with acquired neurological conditions: A review
Beukelman, D. R., Fager, S., Ball, L., & Dietz, A. (2007) · Augmentative and Alternative Communication · doi:10.1080/07434610701553668 · Cited by 264
Real-time decoding of question-and-answer speech dialogue using human cortical activity
Moses, D. A., Leonard, M. K., Makin, J. G., & Chang, E. F. (2019) · Nature Communications · doi:10.1038/s41467-019-10994-4 · Cited by 224
Self-citationHuggins, J. E., Wren, P. A., & Gruis, K. L. (2011) · Amyotrophic Lateral Sclerosis · doi:10.3109/17482968.2011.572978 · Cited by 215
Imagined speech can be decoded from low- and cross-frequency intracranial EEG features
Proix, T., Delgado Saa, J., Christen, A., Martin, S., Pasley, B. N., Knight, R. T., Tian, X., Poeppel, D., Doyle, W. K., Devinsky, O., et al. (2022) · Nature Communications · doi:10.1038/s41467-021-27725-3 · Cited by 212 · cited 3×
“…the majority of speech-BCI research has focused on decoding low-level articulatory and phonological signals within sensorimotor and auditory regions of the cortex (Martin et al., 2014; Herff et al., 2015; Ramsey et al., 2018; Anumanchipalli et al., 2019; Moses et al., 2021; Proix et al., 2022).”
BackgroundAn fMRI investigation of covertly and overtly produced mono- and multisyllabic words
SHUSTER, L., & LEMIEUX, S. (2005) · Brain and Language · doi:10.1016/j.bandl.2004.07.007 · Cited by 205 · cited 1×
“Indeed, evidence from early imaging studies (Murphy et al., 1997; Shuster and Lemieux, 2005) as well as recent electrophysiological recordings (Proix et al., 2022) suggest that articulatory representations are only weakly encoded during imagined speech.”
BackgroundDecoding spectrotemporal features of overt and covert speech from the human cortex
Martin, S., Brunner, P., Holdgraf, C., Heinze, H.-J., Crone, N. E., Rieger, J., Schalk, G., Knight, R. T., & Pasley, B. N. (2014) · Frontiers in Neuroengineering · doi:10.3389/fneng.2014.00014 · Cited by 200
Moorcroft, A., Scarinci, N., & Meyer, C. (2018) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2018.1499135 · Cited by 161
Translational opportunities and challenges of invasive electrodes for neural interfaces
Shen, K., Chen, O., Edmunds, J. L., Piech, D. K., & Maharbiz, M. M. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01021-5 · Cited by 160
Metzger, S. L., Liu, J. R., Moses, D. A., Dougherty, M. E., Seaton, M. P., Littlejohn, K. T., Chartier, J., Anumanchipalli, G. K., Tu-Chan, A., Ganguly, K., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33611-3 · Cited by 153
Self-citationCerebral areas associated with motor control of speech in humans
Murphy, K., Corfield, D. R., Guz, A., Fink, G. R., Wise, R. J. S., Harrison, J., & Adams, L. (1997) · Journal of Applied Physiology · doi:10.1152/jappl.1997.83.5.1438 · Cited by 149
Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids
Ramsey, N. F., Salari, E., Aarnoutse, E. J., Vansteensel, M. J., Bleichner, M. G., & Freudenburg, Z. V. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.10.011 · Cited by 124
Decoding Inner Speech Using Electrocorticography: Progress and Challenges Toward a Speech Prosthesis
Martin, S., Iturrate, I., Millán, J. D. R., Knight, R. T., & Pasley, B. N. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00422 · Cited by 115
Linse, K., Aust, E., Joos, M., & Hermann, A. (2018) · Frontiers in Neurology · doi:10.3389/fneur.2018.00603 · Cited by 84
Brain2Char: a deep architecture for decoding text from brain recordings
Sun, P., Anumanchipalli, G. K., & Chang, E. F. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc742 · Cited by 77 · cited 1×
“…LMs within speech BCIs is analogous to an “autocorrect” function, seeking to improve neural inference either directly at the neural-decoding stage (Sun et al., 2020), retrospectively by interpreting decoded word probabilities such that final output sentences conform with highly probable sequences…”
BackgroundSelf-citationBrumberg, J. S., Krusienski, D. J., Chakrabarti, S., Gunduz, A., Brunner, P., Ritaccio, A. L., & Schalk, G. (2016) · PLOS ONE · doi:10.1371/journal.pone.0166872 · Cited by 71
Performance assessment in brain-computer interface-based augmentative and alternative communication
Thompson, D. E., Blain‐Moraes, S., & Huggins, J. E. (2013) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-12-43 · Cited by 65
Brain-Computer Interfaces for Communication: Preferences of Individuals With Locked-in Syndrome
Branco, M. P., Pels, E. G. M., Sars, R. H., Aarnoutse, E. J., Ramsey, N. F., Vansteensel, M. J., & Nijboer, F. (2021) · Neurorehabilitation and Neural Repair · doi:10.1177/1545968321989331 · Cited by 61
Peters, B., Bieker, G., Heckman, S. M., Huggins, J. E., Wolf, C., Zeitlin, D., & Fried-Oken, M. (2015) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2014.03.037 · Cited by 44
Neurotechnology ethics and relational agency
Goering, S., Brown, T., & Klein, E. P. (2021) · Philosophy Compass · doi:10.1111/phc3.12734 · Cited by 36
Nijboer, F., Plass-Oude Bos, D., Blokland, Y., van Wijk, R., & Farquhar, J. (2014) · Brain-Computer Interfaces · doi:10.1080/2326263x.2013.877210 · Cited by 35
NeuroVAD: Real-Time Voice Activity Detection from Non-Invasive Neuromagnetic Signals
Dash, D., Ferrari, P., Dutta, S., & Wang, J. (2020) · Sensors · doi:10.3390/s20082248 · Cited by 32
Demonstration of a portable intracortical brain-computer interface
Weiss, J. M., Gaunt, R. A., Franklin, R., Boninger, M. L., & Collinger, J. L. (2019) · Brain-Computer Interfaces · doi:10.1080/2326263x.2019.1709260 · Cited by 32
Control and Ownership of Neuroprosthetic Speech
Maslen, H., & Rainey, S. (2020) · Philosophy & Technology · doi:10.1007/s13347-019-00389-0 · Cited by 23
Vansteensel, M. J., Branco, M. P., Leinders, S., Freudenburg, Z. V., Schippers, A., Geukes, S. H., Gaytant, M. A., Gosselaar, P. H., Aarnoutse, E. J., & Ramsey, N. F. (2022) · Neurorehabilitation and neural repair · doi:10.1177/15459683221125788 · Cited by 23
Online internal speech decoding from single neurons in a human participant
Wandelt, S. K., Bjånes, D. A., Pejsa, K., Lee, B., Liu, C., & Andersen, R. A. (2022) · doi:10.1101/2022.11.02.22281775 · Cited by 17 · cited 1×
“Additionally, alternate strategies that decode imagined speech from phonological (Brumberg et al., 2016) or somatosensory (Wandelt et al., 2022) representations have shown promise.”
BackgroundHill, K. J., Huggins, J. E., & Woodworth, C. (2020) · PM&R · doi:10.1002/pmrj.12525 · Cited by 10
Improving Patient Safety and Patient-Provider Communication: A Clinical Forum
Hurtig, R. R., Alper, R. M., Bryant, K., Davidson, K. R., & Bilskemper, C. (2019) · Perspectives of the ASHA Special Interest Groups · doi:10.1044/2019_persp-19-00134 · Cited by 10
Building communication neurotechnology for high stakes communications
Chandler, J. A., der Loos, K. I. V., Boehnke, S. E., Beaudry, J. S., Buchman, D. Z., & Illes, J. (2021) · Nature reviews. Neuroscience · doi:10.1038/s41583-021-00517-w · Cited by 8
Research works printed but not linked: 2
- Boy, 2017, The handbook of human-machine interaction: a human-centered design approach, 10.1201/9781315557380 · doi:10.1201/9781315557380
- Zickler, 2009, BCI applications for people with disabilities: defining user needs and user requirements, Assist. Technol. Res. Ser., 25, 185, 10.3233/978.1.60750.042.1.185 · doi:10.3233/978.1.60750.042.1.185
Cited by
Silva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103
“A potential safeguard would be to chain any speech decoding to an initial speech-detection module, trained to identify volitional speech attempts 218 .”
Self-citationThe ethical significance of user-control in AI-driven speech-BCIs: a narrative review
van Stuijvenberg, O. C., Samlal, D. P. S., Vansteensel, M. J., Broekman, M. L. D., & Jongsma, K. R. (2024) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2024.1420334 · Cited by 13
Metadata
- Authors
- Sankaran, N., Moses, D., Chiong, W., & Chang, E. F.
- Venue
- Frontiers in Human Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2023-10-18
- Volume
- 17
- ISSN
- 1662-5161
- Cited by
- 14
- References
- 39
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.