Biosignal-Based Spoken Communication: A Survey
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Speech is a complex process involving a wide range of biosignals, including but not limited to acoustics. These biosignals-stemming from the articulators, the articulator muscle activities, the neural pathways, and the brain itself-can be used to circumvent limitations of conventional speech processing in particular, and to gain insights into the process of speech production in general. Research on biosignal-based speech processing is a wide and very active field at the intersection of various disciplines, ranging from engineering, computer science, electronics and machine learning to medicine, neuroscience, physiology, and psychology. Consequently, a variety of methods and approaches have been used to investigate the common goal of creating biosignal-based speech processing devices for communication applications in everyday situations and for speech rehabilitation, as well as gaining a deeper understanding of spoken communication. This paper gives an overview of the various modalities, research approaches, and objectives for biosignal-based spoken communication.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743
Brain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927
Event-related EEG/MEG synchronization and desynchronization: basic principles
Pfurtscheller, G., & Lopes da Silva, F. H. (1999) · Clinical Neurophysiology · doi:10.1016/s1388-2457(99)00141-8 · Cited by 7,328 · cited 1×
“, event-related (de)synchronization) [51].”
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Jöbsis, F. F. (1977) · Science · doi:10.1126/science.929199 · Cited by 3,899
(1992) · doi:10.5555/176313.176316 · Cited by 2,892
Nunez, P. L., & Srinivasan, R. (2006) · doi:10.1093/acprof:oso/9780195050387.001.0001 · Cited by 2,531
Price, C. J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.04.062 · Cited by 2,159
Picton, T. W., Bentin, S., Berg, P., Donchin, E., Hillyard, S. A., Johnson, R., Miller, G. A., Ritter, W., Ruchkin, D. S., Rugg, M. D., et al. (2000) · Psychophysiology · doi:10.1111/1469-8986.3720127 · Cited by 2,157 · cited 1×
“, event-related potentials, ERPs) to overcome its comparatively low signal-to-noise ratio [50].”
BackgroundA new strategy for multifunction myoelectric control
Hudgins, B., Parker, P., & Scott, R. N. (1993) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.204774 · Cited by 2,149 · cited 1×
“Modern approaches now use time-domain features [39] similar to the Hudgins feature set [80] and spectral”
MethodThe spatial and temporal signatures of word production components
Indefrey, P., & Levelt, W. J. M. (2004) · Cognition · doi:10.1016/j.cognition.2002.06.001 · Cited by 2,144
Natural speech reveals the semantic maps that tile human cerebral cortex
Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016) · Nature · doi:10.1038/nature17637 · Cited by 1,504 · cited 1×
“have investigated this decoding approach using fMRI for decoding three Dutch vowels [121], nouns [122], and functional representations of natural speech [123].”
MethodPredicting Human Brain Activity Associated with the Meanings of Nouns
Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008) · Science · doi:10.1126/science.1152876 · Cited by 1,180 · cited 1×
“have investigated this decoding approach using fMRI for decoding three Dutch vowels [121], nouns [122], and functional representations of natural speech [123].”
MethodInner Speech: Development, Cognitive Functions, Phenomenology, and Neurobiology
Alderson-Day, B., & Fernyhough, C. (2015) · Psychological Bulletin · doi:10.1037/bul0000021 · Cited by 864 · cited 1×
“, selftalk, verbal thinking, inner voice, inner dialogue) [12], we adopt Vygotsky’s model [13] that defines inner speech as an internalized process in which one thinks in pure meanings.”
MethodNeural mechanisms underlying auditory feedback control of speech
Tourville, J. A., Reilly, K. J., & Guenther, F. H. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.09.054 · Cited by 649
Maynard, E. M., Nordhausen, C. T., & Normann, R. A. (1997) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/s0013-4694(96)95176-0 · Cited by 640 · cited 1×
“croelectrodes and microarrays, such as the Utah array [52], provide unparalleled spatial and temporal resolution down to single neuron action potentials.”
BackgroundReconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604 · cited 1×
“tening [127].”
Who “Is Saying What”? Brain-based decoding of human voice and speech
Formisano, E., De Martino, F., Bonte, M., & Goebel, R. (2008) · Science · doi:10.1126/science.1164318 · Cited by 599 · cited 1×
“have investigated this decoding approach using fMRI for decoding three Dutch vowels [121], nouns [122], and functional representations of natural speech [123].”
MethodAudio-visual speech modeling for continuous speech recognition
Dupont, S., & Luettin, J. (2000) · IEEE Transactions on Multimedia · doi:10.1109/6046.865479 · Cited by 598
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · cited 4×
“4% word accuracy for the recognition of 333 word classes without use of a language model applying television broadcasts data [26].”
BackgroundMethodInfluentialHigh-frequency gamma oscillations and human brain mapping with electrocorticography
Crone, N. E., Sinai, A., & Korzeniewska, A. (2006) · Progress in Brain Research · doi:10.1016/s0079-6123(06)59019-3 · Cited by 526
Denby, B., Schultz, T., Honda, K., Hueber, T., Gilbert, J. M., & Brumberg, J. S. (2010) · Speech Communication · doi:10.1016/j.specom.2009.08.002 · Cited by 487 · cited 6×
“In a closed-loop paradigm, speakers can rely on synthetic speech for auditory feedback and exploit it to regulate their own production, as in [140] for SSI and [57] for BCI.”
BackgroundInfluentialSelf-citationBrain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297 · cited 4×
“Alternatively, the ability to decode basic units of speech, such as formants or phones [98], will enable the creation of generative models that are not limited to a fixed vocabulary.”
BackgroundMethodInfluentialSelf-citationDecoding spoken words using local field potentials recorded from the cortical surface
Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., & Greger, B. (2010) · Journal of Neural Engineering · doi:10.1088/1741-2560/7/5/056007 · Cited by 235 · cited 1×
“[131]; microelectrodes: [55], [56]), syllables (EEG: [88]; ECoG: [132]), and words (ECoG: [133]).”
Decoding spectrotemporal features of overt and covert speech from the human cortex
Martin, S., Brunner, P., Holdgraf, C., Heinze, H.-J., Crone, N. E., Rieger, J., Schalk, G., Knight, R. T., & Pasley, B. N. (2014) · Frontiers in Neuroengineering · doi:10.3389/fneng.2014.00014 · Cited by 200
Mugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., Shih, J. J., Krusienski, D. J., & Slutzky, M. W. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/035015 · Cited by 192
Self-citationSpeech motor control: acoustic goals, saturation effects, auditory feedback and internal models.
Perkell, J., Matthies, M., Lane, H., Guenther, F., Wilhelms-Tricarico, R., Wozniak, J., & Guiod, P. (1997) · Speech Communication · doi:10.1016/s0167-6393(97)00026-5 · Cited by 177 · cited 1×
“to the absence of auditory feedback that is critical for normal speech production [154], [155].”
BackgroundWord pair classification during imagined speech using direct brain recordings
Martin, S., Brunner, P., Iturrate, I., Millán, J. D. R., Schalk, G., Knight, R. T., & Pasley, B. N. (2016) · Scientific Reports · doi:10.1038/srep25803 · Cited by 171
Brumberg (2011) · Frontiers in Neuroscience · doi:10.3389/fnins.2011.00065 · Cited by 114 · cited 4×
“[131]; microelectrodes: [55], [56]), syllables (EEG: [88]; ECoG: [132]), and words (ECoG: [133]).”
MethodInfluential
Cited by
Speech synthesis from ECoG using densely connected 3D convolutional neural networks
Angrick, M., Herff, C., Mugler, E., Tate, M. C., Slutzky, M. W., Krusienski, D. J., & Schultz, T. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab0c59 · Cited by 225
Self-citationHerff, C., Diener, L., Angrick, M., Mugler, E., Tate, M. C., Goldrick, M. A., Krusienski, D. J., Slutzky, M. W., & Schultz, T. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01267 · Cited by 118
Self-citationBio-Signals in Medical Applications and Challenges Using Artificial Intelligence
Swapna, M., Viswanadhula, U. M., Aluvalu, R., Vardharajan, V., & Kotecha, K. (2022) · Journal of Sensor and Actuator Networks · doi:10.3390/jsan11010017 · Cited by 60
Metadata
- Authors
- Schultz, T., Wand, M., Hueber, T., Krusienski, D. J., Herff, C., & Brumberg, J. S.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2017-12
- Volume
- 25
- Issue
- 12
- Pages
- 2257-2271
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W2768153200
- Cited by
- 201
- References
- 29
- FWCI
- 15.64
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.