The DOI matches its Crossref record; 18 of the 21 references it cites were found in a registry or an index.
Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets
Abstract
While Deep Neural Networks (DNNs) have achieved tremendous success for large vocabulary continuous speech recognition (LVCSR) tasks, training of these networks is slow. One reason is that DNNs are trained with a large number of training parameters (i.e., 10–50 million). Because networks are trained with a large number of output targets to achieve good performance, the majority of these parameters are in the final weight layer. In this paper, we propose a low-rank matrix factorization of the final weight layer. We apply this low-rank technique to DNNs for both acoustic modeling and language modeling. We show on three different LVCSR tasks ranging between 50–400 hrs, that a low-rank factorization reduces the number of parameters of the network by 30–50%. This results in roughly an equivalent reduction in training time, without a significant loss in final recognition accuracy, compared to a full-rank representation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“[213] further shows that INT4 inference of ResNet50 could bring an additional 50-60% speedup on NVIDIA T4 and RTX, compared to its INT8 counterpart, emphasizing the importance of using lower-bit precision to maximize efficiency.”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [213]
In the citing paper
“Memory required to store the weight matrices can be reduced by using a small projection layer and a small layer before the output layer, or factorizing a large weight into the product of two smaller matrices [35].”
From Automatic Speech Recognition With Very Large Conversational Finnish and Estonian Vocabularies (2017) · cites this paper as [35]
Citations
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationConvolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL · doi:10.5555/303568.303704 · Cited by 4,646
Deep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · Cited by 1,839
Self-citationConnectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
Deep convolutional neural networks for LVCSR
Sainath, T. N., Mohamed, A., Kingsbury, B. E. D., & Ramabhadran, B. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639347 · Cited by 1,083
Self-citationConversational speech transcription using context-dependent deep neural networks
Seide, F. T. B., Li, G., & Yu, D. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-169 · Cited by 877
Seide, F. T. B., Li, G., Xie, C., & Yu, D. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2011.6163899 · Cited by 648
Parallel stochastic gradient algorithms for large-scale matrix completion
Recht, B., & Ré, C. (2013) · Mathematical Programming Computation · doi:10.1007/s12532-013-0053-8 · Cited by 378
Introduction to Linear Algebra
DeFranza, J., & Gagliardi, D. (2008) · Cited by 378
Dimension Reduction and Coefficient Estimation in Multivariate Linear Regression
Yuan, M., Ekici, A., Lu, Z., & Monteiro, R. D. C. (2007) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2007.00591.x · Cited by 310
Research works printed but not linked: 1
- sainath, 2013, Exploring convolutional neural networks for large vo-cabulary speech recognition, Submitted to ICASSP
Cited by
Sorted by citations
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Deep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,789
Metadata
- Authors
- Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B.
- Affiliations
- IBM (United States) (US)
- Venue
- 2013 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 6655-6659
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2058641082
- Cited by
- 593
- References
- 21
- Citations by year
- 2026: 17 · 2025: 38 · 2024: 36 · 2023: 40 · 2022: 53 · 2021: 64 · 2020: 71 · 2019: 52 · 2018: 54 · 2017: 48 · 2016: 55 · 2015: 37 · 2014: 24 · 2013: 4
- FWCI
- 25.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- low-rank factorization, deep neural network training, large vocabulary continuous speech recognition, parameter reduction, acoustic modeling, language modeling, training time reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
