An Investigation into On-Device Personalization of End-to-End Automatic Speech Recognition Models
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Speaker-independent speech recognition systems trained with data from many users are generally robust against speaker variability and work well for a large population of speakers. However, these systems do not always generalize well for users with very different speech characteristics. This issue can be addressed by building personalized systems that are designed to work well for each specific user. In this paper, we investigate the idea of securely training personalized end-to-end speech recognition models on mobile devices so that user data and models never leave the device and are never stored on a server. We study how the mobile training environment impacts performance by simulating on-device data consumption. We conduct experiments using data collected from speech impaired users for personalization. Our results show that personalization achieved 63.7\% relative word error rate reduction when trained in a server environment and 58.1% in a mobile environment. Moving to on-device personalization resulted in 18.7% performance degradation, in exchange for improved scalability and data privacy. To train the model on device, we split the gradient computation into two and achieved 45% memory reduction at the expense of 42% increase in training time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
Self-citationListen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
Federated Optimization: Distributed Machine Learning for On-Device Intelligence
Konečný, J., McMahan, H. B., Ramage, D., & Richtárik, P. (2016) · arXiv · doi:10.48550/arxiv.1610.02527 · Cited by 1,563
Federated Learning for Mobile Keyboard Prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., & Ramage, D. (2018) · arXiv · doi:10.48550/arxiv.1811.03604 · Cited by 1,148 · cited 1×
“Federated learning has been successfully applied to mobile keyboard prediction and query suggestions [17, 18].”
MethodSelf-citationApplied Federated Learning: Improving Google Keyboard Query Suggestions
Yang, T., Andrew, G., Eichner, H., Sun, H., Li, W., Kong, N., Ramage, D., & Beaufays, F. (2018) · arXiv · doi:10.48550/arxiv.1812.02903 · Cited by 684 · cited 1×
“Federated learning has been successfully applied to mobile keyboard prediction and query suggestions [17, 18].”
MethodSelf-citationStreaming End-to-end Speech Recognition for Mobile Devices
He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682336 · Cited by 601
Self-citationNeural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304 · cited 1×
“Automatic speech recognition (ASR) systems are trained with thousands of hours of speech data from many users to be robust against speaker variability and achieve state-of-the-art performance [1, 2, 3].”
BackgroundSwietojanski, P., & Renals, S. (2014) · 2014 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2014.7078569 · Cited by 218
Li, B., & Sim, K. C. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-214 · Cited by 155
Self-citation
Cited by
Federated Learning for Healthcare Informatics
Xu, J., Glicksberg, B. S., Su, C., Walker, P., Bian, J., & Wang, F. (2019) · arXiv · doi:10.48550/arxiv.1911.06270 · Cited by 1,538
“Although there are some general work about federated learning personalization [63, 109], for healthcare informatics, how to combining the medical domain knowledge and make the global model be personalized for every medical institutions or wearable devices is another open question.”
BackgroundFederated Learning for Healthcare Informatics
Xu, J., Glicksberg, B. S., Su, C., Walker, P., Bian, J., & Wang, F. (2021) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-020-00082-4 · Cited by 1,542
“Although there are some general work about federated learning personalization [48, 94], for healthcare informatics, how to combining the medical domain knowledge and make the global model be personalized for every medical institutions or wearable devices is another open question.”
BackgroundImproving Federated Learning Personalization via Model Agnostic Meta Learning
Jiang, Y., Konečný, J., Rush, K., & Kannan, S. (2019) · arXiv · doi:10.48550/arxiv.1909.12488 · Cited by 381
“Existing FL personalization work directly takes a converged initial model and conducts personalization evaluation via gradient descent (Beaufays et al., 2019).”
BackgroundSurvey of Personalization Techniques for Federated Learning
Kulkarni, V., Kulkarni, M., & Pant, A. (2020) · 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4) · doi:10.1109/worlds450073.2020.9210355 · Cited by 338
Electrical Load Forecasting Using Edge Computing and Federated Learning
Taik, A., & Cherkaoui, S. (2020) · ICC 2020 - 2020 IEEE International Conference on Communications (ICC) · doi:10.1109/icc40277.2020.9148937 · Cited by 252
Metadata
- Authors
- Sim, K. C., Zadrazil, P., & Beaufays, F.
- Venue
- Interspeech 2019
- Publisher
- ISCA
- Published
- 2019-09-15
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2972895620
- Cited by
- 51
- References
- 10
- FWCI
- 3.41
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Investigation into On-Device Personalization of End-to-End Automatic Speech Recognition Models. https://app.cochat.ai/papers/10.21437/interspeech.2019-1752
- The paper
- Sim, K. C., Zadrazil, P., & Beaufays, F. (2019). An Investigation into On-Device Personalization of End-to-End Automatic Speech Recognition Models. Interspeech 2019. https://doi.org/10.21437/interspeech.2019-1752
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.