Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets
Abstract
While Deep Neural Networks (DNNs) have achieved tremendous success for large vocabulary continuous speech recognition (LVCSR) tasks, training of these networks is slow. One reason is that DNNs are trained with a large number of training parameters (i.e., 10–50 million). Because networks are trained with a large number of output targets to achieve good performance, the majority of these parameters are in the final weight layer. In this paper, we propose a low-rank matrix factorization of the final weight layer. We apply this low-rank technique to DNNs for both acoustic modeling and language modeling. We show on three different LVCSR tasks ranging between 50–400 hrs, that a low-rank factorization reduces the number of parameters of the network by 30–50%. This results in roughly an equivalent reduction in training time, without a significant loss in final recognition accuracy, compared to a full-rank representation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
INT4 inference of ResNet50 can bring an additional 50-60% speedup on NVIDIA T4 and RTX compared to its INT8 counterpart.
“[213] further shows that INT4 inference of ResNet50 could bring an additional 50-60% speedup on NVIDIA T4 and RTX, compared to its INT8 counterpart, emphasizing the importance of using lower-bit precision to maximize efficiency.”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [213]
Memory required to store the weight matrices can be reduced by using a small projection layer and a small layer before the output layer, or by factorizing a large weight into the product of two smaller matrices.
“Memory required to store the weight matrices can be reduced by using a small projection layer and a small layer before the output layer, or factorizing a large weight into the product of two smaller matrices [35].”
From Automatic Speech Recognition With Very Large Conversational Finnish and Estonian Vocabularies (2017) · cites this paper as [35]
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
Self-citationConvolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL · doi:10.5555/303568.303704 · Cited by 4,637 · see citing works
Deep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · Cited by 1,839 · see citing works
Self-citationConnectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115 · see citing works
Deep convolutional neural networks for LVCSR
Sainath, T. N., Mohamed, A., Kingsbury, B. E. D., & Ramabhadran, B. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639347 · Cited by 1,066 · see citing works
Self-citationConversational speech transcription using context-dependent deep neural networks
Seide, F. T. B., Li, G., & Yu, D. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-169 · Cited by 877 · see citing works
Seide, F. T. B., Li, G., Xie, C., & Yu, D. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2011.6163899 · Cited by 641 · see citing works
Parallel stochastic gradient algorithms for large-scale matrix completion
Recht, B., & Ré, C. (2013) · Mathematical Programming Computation · doi:10.1007/s12532-013-0053-8 · Cited by 378 · see citing works
Introduction to Linear Algebra
DeFranza, J., & Gagliardi, D. (2008) · Cited by 378 · see citing works
Dimension Reduction and Coefficient Estimation in Multivariate Linear Regression
Yuan, M., Ekici, A., Lu, Z., & Monteiro, R. D. C. (2007) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2007.00591.x · Cited by 310 · see citing works
To recognize shapes, first learn to generate images
Hinton, G. E. (2007) · Progress in brain research · doi:10.1016/s0079-6123(06)65034-6 · Cited by 304 · see citing works
Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine
Dahl, G. E., Ranzato, M., Mohamed, A., & Hinton, G. E. (2010) · Cited by 268 · see citing works
Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling
Kingsbury, B. E. D. (2009) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2009.4960445 · Cited by 234 · see citing works
Self-citationKingsbury, B. E. D., Sainath, T. N., & Soltau, H. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-3 · Cited by 222 · see citing works
Self-citationMaking Deep Belief Networks effective for large vocabulary continuous speech recognition
Sainath, T. N., Kingsbury, B. E. D., Ramabhadran, B., Fousek, P., Novak, P., & Mohamed, A. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2011.6163900 · Cited by 192 · see citing works
Self-citationDeep Neural Network Language Models
Arısoy, E., Sainath, T. N., Kingsbury, B. E. D., & Ramabhadran, B. (2012) · Cited by 174 · see citing works
Self-citationThe IBM Attila speech recognition toolkit
Soltau, H., Saon, G., & Kingsbury, B. E. D. (2010) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2010.5700829 · Cited by 119 · see citing works
Self-citationPerformance prediction for exponential language models
Chen, S. F. (2009) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1620754.1620820 · Cited by 55 · see citing works
Flagged first, then by citations
3 of the 21 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576 · see citing works
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · see citing works
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Deep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,785 · see citing works
Collaborative Deep Learning for Recommender Systems
Wang, H., Wang, N., & Yeung, D.-Y. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783273 · Cited by 1,657 · see citing works
Do Deep Nets Really Need to be Deep?Preprint
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,592 · see citing works
A survey on deep learning for big data
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.10.006 · Cited by 1,226 · see citing works
Speech Recognition Using Deep Neural Networks: A Systematic Review
Nassif, A. B., Shahin, I. M. A., Attili, I. B., Azzeh, M., & Shaalan, K. F. (2019) · IEEE Access · doi:10.1109/access.2019.2896880 · Cited by 1,159 · see citing works
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · see citing works
“…architectures : One line of work has focused on optimizing the NN model architecture in terms of its micro-architecture [100, 110, 125, 163, 164, 205, 245, 272] (e.g., kernel types such as depth-wise convolution or low-rank factorization) as well as its macro-architecture [99, 100, 103, 109,…”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2976475 · Cited by 996 · see citing works
Sak, H., Senior, A., & Beaufays, F. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.1128 · Cited by 859 · see citing works
A Survey of Model Compression and Acceleration for Deep Neural Networks
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2017) · arXiv · doi:10.48550/arxiv.1710.09282 · Cited by 857 · see citing works
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765695 · Cited by 817 · see citing works
Collaborative filtering and deep learning based recommendation system for cold start items
Jian, W., He, J., Chen, K., Zhou, Y., & Tang, Z. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.09.040 · Cited by 713 · see citing works
Tensorizing Neural NetworksPreprint
Novikov, A., Podoprikhin, D., Osokin, A., & Vetrov, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06569 · Cited by 656 · see citing works
Speaker adaptation of neural network acoustic models using i-vectors
Saon, G., Soltau, H., Nahamoo, D., & Picheny, M. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707705 · Cited by 586 · see citing works
A comprehensive survey on model compression and acceleration
Choudhary, T., Mishra, V. K., Goswami, A., & Jagannathan, S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09816-7 · Cited by 519 · see citing works
Communication-efficient edge ai: algorithms and systems
Shi, Y., Yang, K., Jiang, T., Zhang, J., & Letaief, K. B. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3007787 · Cited by 488 · see citing works
BlockDrop: Dynamic Inference Paths in Residual Networks
Wu, Z., Nagarajan, T., Kumar, A., Rennie, S., Davis, L. S., Grauman, K., & Feris, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00919 · Cited by 465 · see citing works
Group sparse regularization for deep neural networks
Scardapane, S., Comminiello, D., Hussain, A., & Uncini, A. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.02.029 · Cited by 455 · see citing works
Fast inference of deep neural networks in FPGAs for particle physics
Duarte, J., Han, S., Harris, P., Jindariani, S., Kreinar, E., Kreis, B., Ngadiuba, J., Maurizio Pierini, B., Rivera, R., Tran, N., et al. (2018) · Journal of Instrumentation · doi:10.1088/1748-0221/13/07/p07027 · Cited by 450 · see citing works
Recent advances in convolutional neural network acceleration
Zhang, Q., Zhang, M., Chen, T., Sun, Z., Ma, Y., & Yu, B. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.09.038 · Cited by 440 · see citing works
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423 · see citing works
Deep Neural Network Approaches to Speaker and Language Recognition
Richardson, F. S., Reynolds, D. A., & Dehak, N. (2015) · IEEE Signal Processing Letters · doi:10.1109/lsp.2015.2420092 · Cited by 404 · see citing works
Fast ConvNets Using Group-Wise Brain Damage
Lebedev, V., & Lempitsky, V. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.280 · Cited by 397 · see citing works
Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., & Feris, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.126 · Cited by 386 · see citing works
fastmachinelearning/hls4ml: coris
Vloncar, Summers, S., Duarte, J., Tran, N., Kreis, B., Jngadiub, Ghielmetti, N., Hoang, D., Kreinar, E., Lin, K., et al. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5680908 · Cited by 363 · see citing works
DeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications
Huynh, L. N., Lee, Y., & Balan, R. K. (2017) · Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3081333.3081360 · Cited by 329 · see citing works
An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections
Cheng, Y., Yu, F. X., Feris, R., Kumar, S., Choudhary, A., & Chang, S.-F. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.327 · Cited by 302 · see citing works
Model Compression for Deep Neural Networks: A Survey
Li, Z., Li, H., & Meng, L. (2023) · Computers · doi:10.3390/computers12030060 · Cited by 313 · see citing works
Learning small-size DNN with output-distribution-based criteria
Li, J., Zhao, R., Huang, J.-T., & Gong, Y. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-432 · Cited by 252 · see citing works
Towards Bayesian Deep Learning: A Framework and Some Existing Methods
Wang, H., & Yeung, D. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2606428 · Cited by 254 · see citing works
A Survey on Bayesian Deep Learning
Wang, H., & Yeung, D.-Y. (2021) · ACM Computing Surveys · doi:10.1145/3409383 · Cited by 247 · see citing works
Multichannel Signal Processing With Deep Neural Networks for Automatic Speech Recognition
Sainath, T. N., Weiss, R. J., Wilson, K., Li, B., Narayanan, A., Variani, E., Bacchiani, M., Izhak Shafran, J., Senior, A., Chin, K. K., et al. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2672401 · Cited by 231 · see citing works
Self-citationMethods for Pruning Deep Neural Networks
Vadera, S., & Ameen, S. (2022) · IEEE Access · doi:10.1109/access.2022.3182659 · Cited by 229 · see citing works
Yang, Z., Moczulski, M., Denil, M., De Freitas, N., Song, L., & Wang, Z. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.173 · Cited by 215 · see citing works
Structured Transforms for Small-Footprint Deep LearningPreprint
Sindhwani, V., Sainath, T. N., & Kumar, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.01722 · Cited by 200 · see citing works
Self-citation"Learning-Compression" Algorithms for Neural Net Pruning
Carreira-Perpiñán, M. Á., & Idelbayev, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00890 · Cited by 198 · see citing works
Centripetal SGD for Pruning Very Deep Convolutional Networks With Complicated Structure
Ding, X., Ding, G., Guo, Y., & Han, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00508 · Cited by 198 · see citing works
Liao, H., McDermott, E., & Senior, A. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707758 · Cited by 194 · see citing works
Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
Verelst, T., & Tuytelaars, T. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00239 · Cited by 174 · see citing works
Personalized speech recognition on mobile devices
Mcgraw, I. C., Prabhavalkar, R., Álvarez, R., Arenas, M. G., Rao, K., Rybach, D., Alsharif, O., Sak, H., Gruenstein, A., Beaufays, F., et al. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472820 · Cited by 164 · see citing works
Xue, J., Li, J., Yu, D., Seltzer, M., & Gong, Y. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854828 · Cited by 162 · see citing works
Sparse low rank factorization for deep neural network compression
Swaminathan, S., Garg, D., Kannan, R., & Andrès, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.035 · Cited by 148 · see citing works
Ultimate tensorization: compressing convolutional and FC layers alikePreprint
Garipov, T., Podoprikhin, D., Novikov, A., & Vetrov, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03214 · Cited by 141 · see citing works
Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
Ye, J., Wang, L., Li, G., Chen, D., Zhe, S., Chu, X., & Xu, Z. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00977 · Cited by 143 · see citing works
OpenEI: An Open Framework for Edge Intelligence
Zhang, X., Wang, Y., Lu, S., Liu, L., xu, L., & Shi, W. (2019) · 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00182 · Cited by 139 · see citing works
“Sainath [27] uses a low-rank matrix factorization on the final weight layer of a DNN for acoustic modeling.”
MethodDeep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
Wu, J., Yılmaz, E., Zhang, M., Li, H., & Tan, K. C. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00199 · Cited by 136 · see citing works
Lower Frame Rate Neural Network Acoustic Models
Pundak, G., & Sainath, T. N. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-275 · Cited by 132 · see citing works
Self-citationLow-Rank Compression of Neural Nets: Learning the Rank of Each Layer
Idelbayev, Y., & Carreira-Perpiñán, M. Á. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00807 · Cited by 132 · see citing works
A survey on optimization techniques for edge artificial intelligence (ai)
Surianarayanan, C., Lawrence, J. J., Chelliah, P. R., Prakash, E., & Hewage, C. (2023) · Sensors · doi:10.3390/s23031279 · Cited by 131 · see citing works
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122 · see citing works
Deep Model Compression: Distilling Knowledge from Noisy Teachers
Sau, B. B., & Balasubramanian, V. N. (2016) · arXiv.org · doi:10.48550/arxiv.1610.09650 · Cited by 196 · see citing works
Building DNN acoustic models for large vocabulary speech recognition
Maas, A. L., Qi, P., Xie, Z., Hannun, A., Lengerich, C. T., Jurafsky, D. S., & Ng, A. Y. (2016) · Computer Speech & Language · doi:10.1016/j.csl.2016.06.007 · Cited by 120 · see citing works
Reshaping deep neural network for fast decoding by node-pruning
He, T., Fan, Y., Qian, Y., Tan, T., & Yu, K. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853595 · Cited by 119 · see citing works
Neural Network Adaptive Beamforming for Robust Multichannel Speech Recognition
Li, B., Sainath, T. N., Weiss, R. J., Wilson, K., & Bacchiani, M. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-173 · Cited by 115 · see citing works
Self-citationA Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
Katare, D., Perino, D., Nurmi, J., Warnier, M., Janssen, M., & Ding, A. (2023) · IEEE Communications Surveys and Tutorials · doi:10.1109/comst.2023.3302474 · Cited by 112 · see citing works
Unsupervised Anomaly Detection and Localization Based on Deep Spatiotemporal Translation Network
Ganokratanaa, T., Aramvith, S., & Sebe, N. (2020) · IEEE Access · doi:10.1109/access.2020.2979869 · Cited by 112 · see citing works
Compressed Time Delay Neural Network for Small-Footprint Keyword Spotting
Sun, M., Snyder, D., Gao, Y., Nagaraja, V., Rodehorst, M., Panchapagesan, S., Ström, N., Matsoukas, S., & Vitaladevuni, S. N. P. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-480 · Cited by 111 · see citing works
Direct Shape Regression Networks for End-to-End Face Alignment
Miao, X., Zhen, X., Liu, X., Deng, C. D., Athitsos, V., & Huang, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00529 · Cited by 111 · see citing works
Compressing recurrent neural network with tensor train
Tjandra, A., Sakti, S., & Nakamura, S. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966420 · Cited by 110 · see citing works
Efficient Training of Very Deep Neural Networks for Supervised Hashing
Zhang, Z., Chen, Y., & Saligrama, V. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.165 · Cited by 109 · see citing works
The IBM 2016 English Conversational Telephone Speech Recognition System
Saon, G., Sercu, T., Rennie, S. J., & Kuo, H.-K. J. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1460 · Cited by 106 · see citing works
Deep neural networks compression: A comparative survey and choice recommendations
Marinò, G. C., Petrini, A., Malchiodi, D., & Frasca, M. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.11.072 · Cited by 104 · see citing works
On Complex Valued Convolutional Neural NetworksPreprint
Guberman, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.09046 · Cited by 104 · see citing works
A survey on deep matrix factorizations
Handschutter, P. D., Gillis, N., & Siebert, X. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100423 · Cited by 102 · see citing works
Building Competitive Direct Acoustics-to-Word Models for English Conversational Speech Recognition
Audhkhasi, K., Kingsbury, B. E. D., Ramabhadran, B., Saon, G., & Picheny, M. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461935 · Cited by 100 · see citing works
Self-citationDP Compress: A Model Compression Scheme for Generating Efficient Deep Potential Models
Lu, D., Jiang, W., Chen, Y., Zhang, L., Jia, W., Wang, H., & Chen, M. (2022) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.2c00102 · Cited by 100 · see citing works
Large vocabulary automatic speech recognition for children
Liao, H., Pundak, G., Siohan, O., Carroll, M. K., Coccaro, N., Jiang, Q.-M., Sainath, T. N., Senior, A., Beaufays, F., & Bacchiani, M. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-373 · Cited by 99 · see citing works
Self-citationPrabhavalkar, R., Alsharif, O., Bruguier, A., & McGraw, L. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472823 · Cited by 97 · see citing works
Is Attention Better Than Matrix Decomposition?Preprint
Geng, Z., Guo, M.-H., Chen, H., Li, X., Wei, K., & Lin, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04553 · Cited by 127 · see citing works
Deep learning: from speech recognition to language and multimodal processing
Deng, L. (2016) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2015.22 · Cited by 95 · see citing works
Random Feature Expansions for Deep Gaussian Processes
Cutajar, K., Bonilla, E. V., Pietro Michiardi, A., & Filippone, M. (2016) · Graduate School and Research Center in Digital Science (EURECOM) · doi:10.48550/arxiv.1610.04386 · Cited by 95 · see citing works
Neural Tensor Factorization for Temporal Interaction Learning
Wu, X., Shi, B., Dong, Y., Huang, C., & Chawla, N. V. (2019) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3289600.3290998 · Cited by 92 · see citing works
Parameter-Efficient Model Adaptation for Vision Transformers
He, X., Li, C., Zhang, P., Yang, J., & Wang, X. E. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25160 · Cited by 90 · see citing works
Lee, J., Lee, S., Lee, J., Sathyanarayana, S. D., Lim, H., Lee, J., Zhu, X., Ramakrishnan, S., Grunwald, D. C., Lee, K., et al. (2020) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3386901.3388911 · Cited by 83 · see citing works
AI on the edge: a comprehensive review
Su, W., Li, L., Liu, F., He, M., & Liang, X. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10141-4 · Cited by 86 · see citing works
SNR-Based Progressive Learning of Deep Neural Network for Speech Enhancement
Gao, T., Du, J., Dai, L.-R., & Lee, C. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-224 · Cited by 82 · see citing works
Cheng, X., Zhang, L., Tang, Y., Liu, Y., Wu, H., & He, J. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3149337 · Cited by 84 · see citing works
Model Compression Applied to Small-Footprint Keyword Spotting
Tucker, G., Wu, M., Sun, M., Panchapagesan, S., Fu, G.-S., & Vitaladevuni, S. N. P. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1393 · Cited by 81 · see citing works
Deep Model Compression and Architecture Optimization for Embedded Systems: A Survey
Berthelier, A., Château, T., Duffner, S., García, C., & Blanc, C. (2020) · Journal of Signal Processing Systems · doi:10.1007/s11265-020-01596-1 · Cited by 81 · see citing works
Collaborative Recurrent Autoencoder: Recommend while Learning to Fill in the BlanksPreprint
Wang, H., Shi, X., & Yeung, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00454 · Cited by 79 · see citing works
Neuroevolution of self-interpretable agents
Tang, Y., Nguyen, D., & Ha, D. (2020) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3377930.3389847 · Cited by 78 · see citing works
Feature selection using Deep Neural Networks
Roy, D., Murty, K. S. R., & Mohan, C. K. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280626 · Cited by 77 · see citing works
On-Device Deep Learning for Mobile and Wearable Sensing Applications: A Review
İncel, Ö. D., & Bursa, S. Ö. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3240854 · Cited by 75 · see citing works
[Invited Paper] A Survey of Product Quantization
Matsui, Y., Uchida, Y., Jeǵou, H., & Satoh, S. (2017) · ITE Transactions on Media Technology and Applications · doi:10.3169/mta.6.2 · Cited by 74 · see citing works
A survey of model compression techniques: past, present, and future
Liu, D., Zhu, Y., Liu, Z., Liu, Y., Han, C., Tian, J., Li, R., & Wei, Y. (2025) · Frontiers in Robotics and AI · doi:10.3389/frobt.2025.1518965 · Cited by 87 · see citing works
Compressing deep neural networks using a rank-constrained topology
Nakkiran, P., Álvarez, R., Prabhavalkar, R., & Parada, C. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-351 · Cited by 72 · see citing works
Knowledge distillation for small-footprint highway networks
Lu, L., Guo, M., & Renals, S. J. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953072 · Cited by 71 · see citing works
Lightweight and Efficient End-To-End Speech Recognition Using Low-Rank Transformer
Winata, G. I., Cahyawijaya, S., Lin, Z., Liu, Z., & Fung, P. (2019) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp40776.2020.9053878 · Cited by 90 · see citing works
Rapid adaptation for deep neural networks through multi-task learning
Huang, Z., Li, J., Siniscalchi, S. M., Chen, I., Wu, J. Y., & Lee, C. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-719 · Cited by 64 · see citing works
An Improved Deep Computation Model Based on Canonical Polyadic Decomposition
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2017) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2017.2701797 · Cited by 64 · see citing works
Progressive Blockwise Knowledge Distillation for Neural Network Acceleration
Wang, H., Zhao, H., Li, X., & Tan, X. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/384 · Cited by 63 · see citing works
A Survey on Model Compression and Acceleration for Pretrained Language Models
Xu, C., & McAuley, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26255 · Cited by 70 · see citing works
Mean-normalized stochastic gradient for large-scale deep learning
Wiesler, S., Richard, A., Schlüter, R., & Ney, H. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853582 · Cited by 61 · see citing works
ApNet: Approximation-Aware Real-Time Neural Network
Bateni, S., & Liu, C. (2018) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss.2018.00017 · Cited by 62 · see citing works
Low-Memory Neural Network Training: A Technical ReportPreprint
Sohoni, N. S., Aberger, C. R., Leszczynski, M., Zhang, J., & Ré, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.10631 · Cited by 71 · see citing works
A Tensor-Train Deep Computation Model for Industry Informatics Big Data Feature Learning
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2791423 · Cited by 59 · see citing works
Extracting deep neural network bottleneck features using low-rank matrix factorization
Zhang, Y., Chuangsuwanich, E., & Glass, J. R. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853583 · Cited by 58 · see citing works
- Authors
- Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B.
- Affiliations
- IBM (United States) (US)
- Venue
- 2013 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 6655-6659
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2058641082
- Cited by
- 593
- References
- 21
- Citations by year
- 2026: 17 · 2025: 38 · 2024: 36 · 2023: 40 · 2022: 53 · 2021: 64 · 2020: 71 · 2019: 52 · 2018: 54 · 2017: 48 · 2016: 55 · 2015: 37 · 2014: 24 · 2013: 4
- FWCI
- 25.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- low-rank factorization, deep neural network training, large vocabulary continuous speech recognition, parameter reduction, acoustic modeling, language modeling, training time reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
