Factorization tricks for LSTM networks

“The partition of layer into parallel groups have been introduced by Krizhevsky et al. (2012) in AlexNet, where some convolutional layers have been divided into two groups to split the model between two GPUs.”

“To address this issue, the LSTM cell has been introduced by Hochreiter & Schmidhuber (1997), with the following recurrent computations: LSTM : (1) where x t is input, h t is cell’s state, and c t is cell’s memory.”

“The idea of factorization of large convolutinal layer into the stack of layers with smaller filters was used, for example, in VGG networks (Simonyan & Zisserman, 2014), and in ResNet “bottleneck design” (He et al., 2016).”

“This multi-group approach was extended to the extreme in Xception architecture by Chollet (2016).”

“Learning long-range dependencies with Recurrent Neural Networks (RNN) is challenging due to the vanishing and exploding gradient problems (Bengio et al., 1994; Pascanu et al., 2013).”

“In case of LSTM networks, ConvLSTM (Shi et al., 2015), has been introduced to better exploit possible spatiotemporal correlations, which is conceptually similar to grouping.”

“LSTM networks (Hochreiter & Schmidhuber, 1997) have been successfully used in language modeling (Jozefowicz et al., 2016; Shazeer et al., 2017), speech recognition (Xiong et al., 2016), machine translation (Wu et al., 2016), and many other tasks.”

“Learning long-range dependencies with Recurrent Neural Networks (RNN) is challenging due to the vanishing and exploding gradient problems (Bengio et al., 1994; Pascanu et al., 2013).”

“We consider LSTM cell with projection of size p , LSTMP, where Equation 1 is computed as follows (Sak et al., 2014; Zaremba et al., 2014).”

“We consider LSTM cell with projection of size p , LSTMP, where Equation 1 is computed as follows (Sak et al., 2014; Zaremba et al., 2014).”

“As a baseline, we used BIGLSTM model without CNN inputs described by Jozefowicz et al. (2016).”

“LSTM networks (Hochreiter & Schmidhuber, 1997) have been successfully used in language modeling (Jozefowicz et al., 2016; Shazeer et al., 2017), speech recognition (Xiong et al., 2016), machine translation (Wu et al., 2016), and many other tasks.”

“Denil et al. (2013) have shown that it is possible to train several different deep architectures by learning only a small number of weights and predicting the rest.”

“Multi-group convnets have been widely used to reduce network weights and required compute, for example by Esser et al. (2016).”

“We test F-LSTM and G-LSTM architectures on the task of language modeling using One Billion Word Benchmark (Chelba et al., 2013).”

“LSTM networks (Hochreiter & Schmidhuber, 1997) have been successfully used in language modeling (Jozefowicz et al., 2016; Shazeer et al., 2017), speech recognition (Xiong et al., 2016), machine translation (Wu et al., 2016), and many other tasks.”

“, 2017) or multi-tower (Ciregan et al., 2012) models, the key differences are: (1) input to different groups is different and assumed independent, and (2) instead of computing ensemble output, it is concatenated into independent pieces.”

“Recent work has achieved significant improvements in computational efficiency through factorization tricks [18] and conditional computation [26], while also improving model performance in case of the latter.”

“…Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang et al., 2017; Kanai et…”

“…2013) 51.3 24 GPUs LSTM-2048-512 (Jozefowicz et al., 2016) 43.7 32 GPUs 2-layer LSTM-8192-1024 (Jozefowicz et al., 2016) 30.6 32 GPUs BIG GLSTM-G4 (Kuchaiev & Ginsburg, 2017) 23.3 ∗ 8 GPUs LSTM-2048 (Grave et al., 2016a) 43.9 1 GPU 2-layer LSTM-2048 (Grave et al., 2016a) 39.8 1 GPU GCNN-13 38.1 1…”

“In the NLP literature, the most relevant work can be group LSTMs (Kuchaiev and Ginsburg, 2017; Gao et al., 2018), which employs the idea of group convolution (Zhang et al., 2017, 2018) into Recurrent Neural Networks (RNN).”

“Factorized LSTM [53] fac-torizes each weight matrix of an LSTM into two smaller matrices and feeds them with two subvectors.”

“The weight matrix of LSTM can be factorized into matrices, and results can be concatenated for faster convergence and parameter reduction [90].”

“We use the idea from group convolutions and the recent separable-LSTM paper [12] to further reduce this size by factoring the final 1× 1 convolution, and we call the result a super-separable convolution.”

“, 2017) and to speed up recurrent networks (Diamos et al., 2016; Shazeer et al., 2017; Kuchaiev and Ginsburg, 2017).”

“Kuchaiev & Ginsburg (2017); Neil et al. (2016); Koutnik et al. (2014) proposed to partition the hidden layer into separated modules with different updates.”

“, 2017) and to speed up recurrent networks (Diamos et al., 2016; Shazeer et al., 2017; Kuchaiev and Ginsburg, 2017).”

“Recent work, through factorization tricks [148] and conditional computation [149], has achieved substantial improvements in computational efficiency while also improving model output in the case of the latter. authors have presented two simple ways of reducing the number of parameters and speeding up…”

“Still, they use CIFG and group-LSTM (GLSTM) [32] for approximation.”

“In this work, we extend idea of the in-training compression method proposed in [9] by implementing low-rank units on the transformer model [5], which is suitable for effectively shrinking the whole network size and at the same time, reducing the computational cost in training and evaluation, with improvements in the e…”

“For 4K word-piece models, GLSTM is in general on-par with its P 4K-L counterpart.”

“Although some recent works have somewhat optimized the efficiency through factorization techniques [22], conditional operations [23], etc.”

“Second, approaches based on sequence-to-sequence DL models suffer from the exposure bias [36] and disappearance of the gradient [23, 24], resulting in poor effectiveness in generating a long sequence of tokens as an assertion.”

“The Residual Network (ResNet-50) is a little different from a normal CNN and it can handle the problem of the vanishing gradient [15].”

“While factorisation and conditional computational approaches [59, 60] have improved the efficiency of the model, however, the underlying issue of computing it sequentially persists.”

“In the RNN literature, there is only one paper (Kuchaiev and Ginsburg, 2017), to our best knowledge, exploiting the group strategy.”

“[42,43] proposed factorization tricks, and conditional computation, respectively, notably increase the computational efficiency.”

“term dependencies, and poor sequential computation efficiency [19], [20], may hinder the detection performance of these models.”

“Another approach enforces low-rank structure on the layers, resulting in lower computational cost both for training and inference[5, 6].”

“Hyperparameters : Table 6 shows the hyperparameters used for training the MNIST-LSTM baseline and the MNIST network compressed using pruning, LMF, KPLSTM and a smaller baseline with the number of parameters equivalent to the compressed network.”

“To mitigate the complexity issues, [24] applies a matrix factorization method to the parameters of the LSTM network.”

“Kuchaiev and Ginsburg (2017) focus on how to reduce the computational cost through patitioning or factorizing LSTM matrix.”

“Because these types of networks have millions of parameters in their architecture, a large computation time is required using multi-GPU systems [41].”

“al., 2011; Hutchinson et al., 2015), but these are less powerful than the RNN. A somewhat different use of low-rank factorization has previously been used to reduce the parameter count in an LSTM LM (Kuchaiev and Ginsburg, 2017), finding that the reduced number of parameters leads to faster training. There is a long hi…”

“Low-rank factorization (Kuchaiev and Ginsburg, 2017; Noach and Goldberg, 2020; Hsu et al., 2022) is a widely used model compression technique.”

“For such data, Transformers quickly run out of memory, as they scale quadratically in complexity with sequence length (Vaswani et al., 2017), while LSTMs require significantly larger hidden states (Kuchaiev & Ginsburg, 2017).”

“While factorization and conditional computational approaches (Shazeer et al. 2017; Kuchaiev and Ginsburg 2017) have improved the efficiency of the model, yet the underlying issue of computing it sequentially persists.”

“Classical neural network like LSTM has many variants such as time-frequency LSTM [49], grid LSTM [31], LSTM with layer normalization [5] and dropout [52], phased LSTM [42], group LSTM [35], etc. Graphi is a generic execution engine, optimized for all variations without specializations.”

“Nevertheless, recent research has demonstrated considerable enhancements in computational efficiency via factorization methodologies (Kuchaiev and Ginsburg 2017) and conditional computation techniques (Shazeer et al. 2017).”

“To reduce the number of parameters and accelerate the training, inspired by the LSTM factorization technique [57], we propose to divide the sequence x t , t = 1 , 2 , . . . , μ into independent groups.”

“However, they struggled with scaling to very long sequences and large datasets due to efficiency limitations [87, 160].”

“Finally, these inputs with 300-dimensional vectors are passed to the encoder to train the model with the RNN [44] LSTM [45–47] and layers structured in the form of a linear stack.”

“and when to erase or write to it. However, LSTMs are computational costly and memory-excessive. Therefore, there are different approaches in the literature to overcome this problem. For instance, in [10], two different possibilities are introduced to make LSTM cells more computationally efficient and to reduce the numb…”

“In the work [41], the large weight matrix in the LSTM model is simply factorized into two small matrices and up to 4.5 × parameters reduction is achieved.”

“Low Rank Matrix Factorization (LMF) [8] expresses a larger matrix A of dimension m × n as a product of two smaller matrices U and V of dimension m × d and d × n , respectively.”

“The core idea of MMLSTMs is to divide the output features into two parts and generate them independently with two LSTMs. Reference [33] also splits the output of the LSTM into several groups and utilizes the LSTM with matrix factorization to obtain the output of each group by separated input…”

“[10] explores the independence of neurons in the recurrent layers of LSTM and achieves state-of-the-art results while obtaining 2.”

“Koutnik et al. (2014); Neil et al. (2016); Kuchaiev & Ginsburg (2017) put forward to partition the hidden layer into separated modules as independent feature groups.”

“More detailed analysis, e.g., the accuracy of the JVHW estimates, is shown in the Appendix E. best known model for PTB [48] best known model for 1BW [26] memory length k e s ti m a t e d c ond . e n t r opy ˆ H ( X The performance of ˆ H opt and ˆ H emp in terms of Shannon entropy estimation is…”

“A variant of the locally-connected RNN is the “grouped” RNN [8, 20, 42], where a single layer is replaced by several smaller ones in parallel.”

“Low-rank matrix factorization based approaches for LSTMs and RNNs are presented in [19] and [20], respectively.”

“Kuchaiev and Ginsburg [29] simply factorize the large weight matrix in the LSTM model into two small matrices and achieve up to 4.5 × parameters reduction.”

“Aside from pruning we also experimented with matrix and tensor decomposition techniques (Lebedev et al., 2015; Kim et al., 2016; Kuchaiev and Ginsburg, 2018) which resulted in a similar reduction in the number of model parameters.”

“Moreover, RNN and LSTM models require a considerable amount of time when trained with long timeseries data [25], [26], [27].”

“This paper introduces one such scheme – Hybrid Matrix Factorization.”

“The research in neural network (NN) compression can be roughly categorized into 4 topics: pruning [21], structured matrix based techniques [1], quantization [2, 5] and tensor decomposition [8, 14].”

“…further improve generalization abilities while simultaneously enabling the recurrent unit to learn representations at very high dimensional space, we propose to use grouped linear transformation (GLT) instead of standard linear transformation for recurrent connections (Kuchaiev and Ginsburg, 2017).”

“Additionally, recent works in matrix factorization [58, 60, 33, 78] apply weight factorization to the model parameter before the model is trained and yield a comparable or even better result with fewer parameters and computational cost.”

“In order to overcome such an issue, various attempts have been made to increase the efficiency [17, 29].”

“The summaries generated from the Long Short Term Memory (LSTM) [20] decoder were employed for semantic and contextual similarity matching with the image captions.”

“First, inspired by (Kuchaiev and Ginsburg, 2017), we disentangle the hidden state of LSTM into several independently updated small hidden states, which reduces the number of parameters.”

“With the same setting for low-rank factorization, F-LSTM is also used to train the network.”

“e results to the original application of our model, which achieved perplexity of 23.7 on the Penn Treebank dataset, and the state-of-the-art performance benchmarks on the Google Billion Words dataset [7, 23, 14] (perplexity of 43.8, 28.0, and 24.29, respectively). We note that our model’s low perplexity score cannot b…”

“Factorized LSTM [3] applies SVD structure on LSTM’s weight matrices with little performance regression while using only 1/3 parameters.”

“…pruning, other approaches also make significant contribution to LSTMs compression, including distillation (Tian et al., 2017), matrix factorization (Kuchaiev & Ginsburg, 2017), parameter sharing (Lu et al., 2016), group Lasso regularization (Wen et al., 2017), weight quantization (Zen et al.,…”

“Low rank factorization of neural network weights in general has been the subject of many other works (Denil et al., 2013; Sainath et al., 2013; Ba & Caruana, 2014; Kuchaiev & Ginsburg, 2017).”

“This method produces impressive results with the compressed model is not only smaller and faster but also able to outperform the uncompressed model (Winata et al. 2020; Cahyawijaya 2021; Kuchaiev and Ginsburg 2017).”

“Low-rank Matrix Factorization (LMF) (Kuchaiev & Ginsburg, 2017; Chen et al., 2018; Grachev et al., 2017; Thakker et al., 2019; Thakker et al., 2020a) can also be categorized under this topic.”

“Models trained on large, diverse benchmark corpora such as the Bil-lion Word Corpus and Wikitext-103 have seen re-ported perplexities as low as 23.7 and 37.2, respectively (Kuchaiev and Ginsburg, 2017; Dauphin et al., 2017).”

“Recently, Kronecker Products (KP) were used to compress IoT applications by 15-38x compression factors (Thakker et al., 2019d;b) and achieves better accuracy than pruning (Zhu & Gupta, 2017), lowrank matrix factorization (LMF) and small baseline (SB).”

“Low-rank techniques were also used to accelerate neural network training and reduce the number of parameters [13,18,29,40,50,51].”

“r approach to reduce the size of the hidden layers [4, 38]. Recent work has achieved signifi- cant improvements in computational efficiency through factorization tricks [35] and conditional computation [20]. Recently, factorization was also employed at the embedding layer of large pretrained language models such as BERT…”

“To solve this problem, a lot of work has achieved significant improvement in computational efficiency through factorization tricks [31] and conditional computation [50].”

“The number of parameters in LSTM layer can be calculated as: 4 × h × (( d +1)+ h ) [45], where h is the size of hidden state and d is the input size.”

“Methods that apply low-rank matrix factorization to RNNs are somewhat newer, and were first explored by Kuchaiev and Ginsburg (2017).”

“(Kuchaiev and Ginsburg, 2017) proposed group-wise RNN as a special form of ensembled RNNs.”

“3 on the Google Billion Words dataset [17].”

“For comparison, we note that the architecture used here has also achieved perplexity score of 23.7 on the Penn Treebank dataset and 24.3 on the Google Billion Words dataset [17].”

“We use the idea form [15] and apply to our model.”

“Group linear transformations (GLT), originally introduced to improve the efficiency of the LSTM, also sparsify the connections in fully connected layers and significantly reduce computational costs (Kuchaiev & Ginsburg, 2017; Mehta et al., 2018).”

“Moreover, the work in [39] tackles the challenge of training large LSTM networks by proposing Factorized LSTM (F-LSTM), which decomposes the LSTM weight matrix ( W ) into the product of two smaller matrices ( W 1 and W 2 ), reducing the total parameter count and computational complexity.”

“, Convolutional LSTM [14], Dynamic Cortex Memory [15] and Group LSTM [16].”

“Karmaşıklığı ve öğrenilecek parametre sayısını daha da düşürmek amacıyla, UKSB parametrelerine matris ayrıştırması yöntemi [8] uygulanmaktadır.”

“Recurrent neural networks (RNNs) and especially long short-term memory networks (LSTMs) [HS97], designed to work with sequential data, were widely successful in speech recognition [BK15] and machine translation [AR15].”

“Processing long temporal sequences presents a significant challenge for many deep learning algorithms [1], [2].”

“The implemented deep learning for sentiment analysis is RNN with LSTM [40] and Deep Chrome Convolutional Neural Network (DCCNN) will be applied on the above three datasets and it has shown improved performance than other traditional machine algorithms like SVM, Naïve Bayes, Random Forest, and…”

“Low-rank techniques were also used to accelerate neural network training and reduce the number of parameters [8], [11], [19], [28], [33], [34].”

“In this paper, we propose combining traditional feature parameters with identity vectors as input [2].”

“Kuchaiev and Ginsburg [26] introduce a Matrix Factorization (MF) to further reduce the overall complexity of the network, in which the weight matrices of each gate unit of the network are decomposed into two matrices.”