Personalization of end-to-end speech recognition on mobile devices for named entities
Open accessAbstract
We study the effectiveness of several techniques to personalize end-to-end speech models and improve the recognition of proper names relevant to the user. These techniques differ in the amounts of user effort required to provide supervision, and are evaluated on how they impact speech recognition performance. We propose using keyword-dependent precision and recall metrics to measure vocabulary acquisition performance. We evaluate the algorithms on a dataset that we designed to contain names of persons that are difficult to recognize. Therefore, the baseline recall rate for proper names in this dataset is very low: 2.4%. A data synthesis approach we developed brings it to 48.6%, with no need for speech input from the user. With speech input, if the user corrects only the names, the name recall rate improves to 64.4%. If the user corrects all the recognition errors, we achieve the best recall of 73.5%. To eliminate the need to upload user data and store personalized models on a server, we focus on performing the entire personalization workflow on a mobile device.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In information theory, compression guarantees are instance specific and depend on the entropy of the underlying distribution.
“However usually in information theory, the compression guarantees are instance specific and depend on the entropy of the underlying distribution [140].”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [140]
Section 3.2.2 discusses a Fisher-information-based regularizer for personalizing speech-recognition models, not information-theoretic compression guarantees. A search for entropy, compression, and distribution-dependent guarantees found no passage in our copy addressing the claim; page numbers were unavailable. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2014) · Cited by 33,419 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
(2013) · Cited by 3,740 · see citing works
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973 · see citing works
Self-citationSequence Transduction with Recurrent Neural NetworksPreprint
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,320 · see citing works
Audio augmentation for speech recognition
Ko, T., Peddinti, V., Povey, D., & Khudanpur, S. P. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-711 · Cited by 1,130 · see citing works
Streaming End-to-end Speech Recognition for Mobile Devices
He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682336 · Cited by 601 · see citing works
Self-citationSpeaker adaptation of neural network acoustic models using i-vectors
Saon, G., Soltau, H., Nahamoo, D., & Picheny, M. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707705 · Cited by 586 · see citing works
On Using Monolingual Corpora in Neural Machine TranslationPreprint
Gülçehre, Ç., Fırat, O., Xu, K., Cho, K., Barrault, L., Lin, H.-C., Bougares, F., Schwenk, H., & Bengio, Y. (2015) · HAL · doi:10.48550/arxiv.1503.03535 · Cited by 470 · see citing works
Rao, K., Sak, H., & Prabhavalkar, R. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268935 · Cited by 351 · see citing works
Unsupervised spoken keyword spotting via segmental DTW on Gaussian posteriorgrams
Zhang, Y., & Glass, J. R. (2009) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2009.5372931 · Cited by 331 · see citing works
Multi-style training for robust isolated-word speech recognition
Lippmann, R. P., Martin, E., & Paul, D. (2005) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1987.1169544 · Cited by 313 · see citing works
Cold Fusion: Training Seq2Seq Models Together with Language Models
Sriram, A., Jun, H., Satheesh, S., & Coates, A. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1392 · Cited by 278 · see citing works
Swietojanski, P., & Renals, S. (2014) · 2014 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2014.7078569 · Cited by 218 · see citing works
Improving DNN speaker independence with I-vector inputs
Senior, A., & Moreno, I. L. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853591 · Cited by 192 · see citing works
Personalized speech recognition on mobile devices
Mcgraw, I. C., Prabhavalkar, R., Álvarez, R., Arenas, M. G., Rao, K., Rybach, D., Alsharif, O., Sak, H., Gruenstein, A., Beaufays, F., et al. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472820 · Cited by 164 · see citing works
Self-citationLi, B., & Sim, K. C. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-214 · Cited by 155 · see citing works
Self-citationA Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition
Toshniwal, S., Kannan, A., Chiu, C., Wu, Y., Sainath, T. N., & Livescu, K. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639038 · Cited by 149 · see citing works
Lower Frame Rate Neural Network Acoustic Models
Pundak, G., & Sainath, T. N. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-275 · Cited by 132 · see citing works
Cluster adaptive training for deep neural network
Tan, T., Qian, Y., Yin, M., et al. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178787 · Cited by 73 · see citing works
Factorized Hidden Layer Adaptation for Deep Neural Network Based Acoustic Modeling
Samarakoon, L., & Sim, K. C. (2016) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2016.2601146 · Cited by 73 · see citing works
Self-citationMulti-basis adaptive neural network for rapid adaptation in speech recognition
Wu, C., & Gales, M. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178785 · Cited by 70 · see citing works
Improved recognition of contact names in voice commands
Aleksic, P. S., Allauzen, C., Elson, D. K., et al. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178957 · Cited by 52 · see citing works
Domain Adaptation Using Factorized Hidden Layer for Robust Automatic Speech Recognition
Sim, K. C., Narayanan, A., Misra, A., et al. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2246 · Cited by 33 · see citing works
Self-citationEfficient Implementation of Recurrent Neural Network Transducer in Tensorflow
Bagby, T., Rao, K., & Sim, K. C. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639690 · Cited by 29 · see citing works
Self-citationStreaming End-to-end Speech Recognition For Mobile DevicesPreprint
He, Y., Sainath, T. N., Prabhavalkar, R., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.06621 · Cited by 23 · see citing works
Adaptation of Deep Neural Network Acoustic Models for Robust Automatic Speech Recognition
Sim, K. C., Qian, Y., Mantena, G., Samarakoon, L., Kundu, S., & Tan, T. (2017) · Springer eBooks · doi:10.1007/978-3-319-64680-0_9 · Cited by 14 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Keith Hall, Eunjoon Cho, Cyril Allauzen, Franc ¸oise Beaufays, Noah Coccaro, Kaisuke Nakajima, Michael Riley, Brian Roark, David Rybach, and Linda Zhang, "Composition-based on-the-fly rescoring for salient n- gram biasing," in Sixteenth Annual Conference of the In- ternational Speech Communication Association, 2015, pp. 1418-1422.
- Tom Bagby, Kanishka Rao, and Khe Chai Sim, "Effi- cient implementation of recurrent neural network trans- ducer in TensorFlow," in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 506-512.
Cited by
Sorted by citations
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772 · see citing works
Attack of the tails: Yes, you really can backdoor federated learning
Wang, H., Sreenivasan, K., Rajput, S., Vishwakarma, H., Agarwal, S., Sohn, J.-Y., Lee, K., & Papailiopoulos, D. (2020) · arXiv · doi:10.48550/arxiv.2007.05084 · Cited by 406 · see citing works
Human-centered neural reasoning for subjective content processing: Hate speech, emotions, and humor
Kazienko, P., Bielaniewicz, J., Gruza, M., Kanclerz, K., Karanowski, K., Miłkowski, P., & Kocoń, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.010 · Cited by 38 · see citing works
On-device personalization of automatic speech recognition models for disordered speech
Tomanek, K., Beaufays, F., Cattiau, J., Chandorkar, A., & Sim, K. C. (2021) · arXiv · doi:10.48550/arxiv.2106.10259 · Cited by 14 · see citing works
Self-citation
- Authors
- Sim, K. C., Beaufays, F., Benard, A., Guliani, D., Kabel, A., Khare, N., Lucassen, T., Zadrazil, P., Zhang, H., Johnson, L., Motta, G., & Zhou, L.
- Affiliations
- Google (United States) (US)
- Venue
- 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
- Publisher
- IEEE
- Published
- 2019-12
- Pages
- 23-30
- OpenAlex
- W2996545246
- Cited by
- 54
- References
- 29
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 9 · 2023: 14 · 2022: 5 · 2021: 11 · 2020: 8
- FWCI
- 2.25
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- end-to-end speech recognition, personalization, named entity recognition, proper names, mobile devices, vocabulary acquisition, data synthesis, user corrections
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
