Streaming End-to-end Speech Recognition for Mobile Devices
Open accessThe DOI matches its Crossref record; 42 of the 45 references it cites were found in a registry or an index.
Abstract
End-to-end (E2E) models, which directly predict output character sequences given input speech, are good candidates for on-device speech recognition. E2E models, however, present numerous challenges: In order to be truly useful, such models must decode speech utterances in a streaming fashion, in real time; they must be robust to the long tail of use cases; they must be able to leverage user-specific context (e.g., contact lists); and above all, they must be extremely accurate. In this work, we describe our efforts at building an E2E speech recog-nizer using a recurrent neural network transducer. In experimental evaluations, we find that the proposed approach can outperform a conventional CTC-based model in terms of both latency and accuracy in a number of evaluation categories.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“which is comparable to state-of-the-art speech recognition systems (e.g. word error rates of 4–5% 14,15), putting it well within the range of usability.”
From High-performance brain-to-text communication via handwriting (2021) · cites this paper as [15]
In the citing paper
“Importantly, a 24% word error rate is probably not yet sufficiently low for everyday use (for example, compared with a 4–5% word error rate for state-of-the-art speech-to-text systems23,37).”
From A high-performance speech neuroprosthesis (2023) · cites this paper as [37]
In the citing paper
“Recently, real-time ASR systems have been developed either using on-device computing or in a cloud-assisted manner [161], [162].”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [162]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 2,557
Sequence Transduction with Recurrent Neural Networks
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,170
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472618 · Cited by 1,096
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models
Chiu, C.-C., Sainath, T. N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R. J., Rao, K., Gonina, E., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462105 · Cited by 1,043
Self-citationWeighted finite-state transducers in speech recognition
Mohri, M., Pereira, F., & Riley, M. (2002) · Computer Speech & Language · doi:10.1006/csla.2001.0184 · Cited by 885
Japanese and Korean voice search
Schuster, M., & Nakajima, K. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6289079 · Cited by 862
Joint CTC-attention based end-to-end speech recognition using multi-task learning
Kim, S., Hori, T., & Watanabe, S. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953075 · Cited by 844
EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding
Miao, Y., Gowayyed, M., & Metze, F. (2015) · 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404790 · Cited by 707
Convolutional neural networks for small-footprint keyword spotting
Sainath, T. N., & Parada, C. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-352 · Cited by 521
Self-citationBa, J. L., Kiros, J., & Hinton, G. E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06450 · Cited by 440
Rao, K., Sak, H., & Prabhavalkar, R. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268935 · Cited by 341
Self-citationParallel WaveNet: Fast High-Fidelity Speech Synthesis
van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., van den Driessche, G., Lockhart, E., Cobo, L. C., Stimberg, F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10433 · Cited by 330
Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304
Chan, W., Jaitly, N., Le, Q. V., & Vinyals, O. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.01211 · Cited by 284
An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model
Kannan, A., Wu, Y., Nguyen, P., Sainath, T. N., Chen, Z., & Prabhavalkar, R. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462682 · Cited by 253
Self-citation“Your Word is my Command”: Google Search by Voice: A Case Study
Schalkwyk, J., Beeferman, D., Beaufays, F., Byrne, B., Chelba, C., Cohen, M., Kamvar, M., & Strope, B. (2010) · Advances in Speech Recognition · doi:10.1007/978-1-4419-5951-5_4 · Cited by 232
Kim, C., Misra, A., Chin, K., Hughes, T., Narayanan, A., Sainath, T. N., & Bacchiani, M. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1510 · Cited by 218
Self-citationPersonalized speech recognition on mobile devices
McGraw, I., Prabhavalkar, R., Alvarez, R., Arenas, M. G., Rao, K., Rybach, D., Alsharif, O., Sak, H., Gruenstein, A., Beaufays, F., et al. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472820 · Cited by 161
Self-citationDeep Context: End-to-end Contextual Speech Recognition
Pundak, G., Sainath, T. N., Prabhavalkar, R., Kannan, A., & Zhao, D. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639034 · Cited by 158
Self-citationConvolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting
Arık, S. Ö., Kliegl, M., Child, R., Hestness, J., Gibiansky, A., Fougner, C., Prenger, R., & Coates, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1737 · Cited by 148
Lower Frame Rate Neural Network Acoustic Models
Pundak, G., & Sainath, T. N. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-275 · Cited by 128
Self-citationBringing contextual information to google speech recognition
Aleksic, P., Ghodsi, M., Michaely, A., Allauzen, C., Hall, K., Roark, B., Rybach, D., & Moreno, P. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-177 · Cited by 91
Self-citationStreaming small-footprint keyword spotting using sequence-to-sequence models
He, Y., Prabhavalkar, R., Rao, K., Li, W., Bakhtin, A., & Mcgraw, I. C. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268974 · Cited by 85
Self-citationContextual Speech Recognition in End-to-end Neural Network Systems Using Beam Search
Williams, I., Kannan, A., Aleksic, P. S., Rybach, D., & Sainath, T. N. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2416 · Cited by 82
Self-citationAn RNN Model of Text Normalization
Sproat, R., & Jaitly, N. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-35 · Cited by 63
Recent Advances in Google Real-Time HMM-Driven Unit Selection Synthesizer
Gonzalvo, X., Tazari, S., Chan, C.-A., Becker, M., Gutkin, A., & Silén, H. (2016) · Interspeech · doi:10.21437/interspeech.2016-264 · Cited by 62
End-to-end Streaming Keyword Spotting
Alvarez, R., & Park, H.-J. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683557 · Cited by 62
Embedded speech recognition applications in mobile phones: Status, trends, and challenges
Cohen, J. (2008) · 2008 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2008.4518869 · Cited by 60
Composition-based on-the-fly rescoring for salient n-gram biasing
Hall, K., Cho, E., Allauzen, C., Beaufays, F., Coccaro, N., Nakajima, K., Riley, M., Brian Roark, A., Rybach, D., & Zhang, L. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-340 · Cited by 52
Self-citationSemi-supervised Training for End-to-end Models via Weak Distillation
Li, B., Sainath, T. N., Pang, R., & Wu, Z. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682172 · Cited by 46
Self-citationOn the Efficient Representation and Execution of Deep Acoustic Models
Alvarez, R., Prabhavalkar, R., & Bakhtin, A. (2016) · Interspeech 2016 · doi:10.21437/interspeech.2016-128 · Cited by 44
Self-citationSpeechalator: two-way speech-to-speech translation on a consumer PDA
Waibel, A., Badran, A., Black, A. W., Frederking, R., Gates, D., Lavie, A., Levin, L. S., Lenzo, K., Tomokiyo, L. M., Reichert, J., et al. (2003) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2003-147 · Cited by 41
Sequence-based class tagging for robust transcription in ASR
Vasserman, L., Schogol, V., & Hall, K. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-178 · Cited by 17
Reducing the computational complexity for whole word models
Soltau, H., Liao, H., & Sak, H. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268917 · Cited by 14
Improving the efficiency of forward-backward algorithm using batched computation in TensorFlow
Sim, K. C., Narayanan, A., Bagby, T., Sainath, T. N., & Bacchiani, M. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268944 · Cited by 10
Self-citation(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1
Research works printed but not linked: 3
- graves, 2012, Speech recognition with deep neural networks, Proc ICASSP
- chiu, 2017, Monotonic chunkwise alignments, Proc ICLR
- 10.1109/ICASSP.2012.6289079 · DOI · Paper page
Cited by
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150
Self-citationConformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.08100 · Cited by 3,150
Self-citationHigh-performance brain-to-text communication via handwriting
Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M., & Shenoy, K. V. (2021) · Nature · doi:10.1038/s41586-021-03506-2 · Cited by 1,024
“The character error rate decreased to 0.89% and the word error rate decreased to 3.4% averaged across all days, which is comparable to state-of-the-art speech recognition systems with word error rates of 4–5% 14,15 , putting it well within the range of usability.”
BackgroundInfluentialZhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774
A high-performance speech neuroprosthesis
Willett, F. R., Kunz, E. M., Fan, C., Avansino, D. T., Wilson, G. H., Choi, E. Y., Kamdar, F., Glasser, M. F., Hochberg, L. R., Druckmann, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06377-x · Cited by 581
Metadata
- Authors
- He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., Liang, Q., Bhatia, D., Shangguan, Y., Li, B., Pundak, G., Sim, K. C., Bagby, T., Chang, S.-Y., Rao, K., & Gruenstein, A.
- Affiliations
- Google (United States) (US)
- ORCID
- Venue
- ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 6381-6385
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2962760690
- Cited by
- 601
- References
- 45
- Citations by year
- 2026: 11 · 2025: 36 · 2024: 37 · 2023: 86 · 2022: 89 · 2021: 164 · 2020: 140 · 2019: 37
- FWCI
- 41.13
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- end-to-end speech recognition, streaming speech recognition, on-device speech recognition, mobile devices, RNN-T, real-time decoding, latency, contact lists
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.