Factorization tricks for LSTM networks

“The idea of factorization of large convolutinal layer into the stack of layers with smaller filters was used, for example, in VGG networks (Simonyan & Zisserman, 2014), and in ResNet “bottleneck design” (He et al., 2016).”

“The partition of layer into parallel groups have been introduced by Krizhevsky et al. (2012) in AlexNet, where some convolutional layers have been divided into two groups to split the model between two GPUs.”

“To address this issue, the LSTM cell has been introduced by Hochreiter & Schmidhuber (1997), with the following recurrent computations: LSTM : (1) where x t is input, h t is cell’s state, and c t is cell’s memory.”

“This multi-group approach was extended to the extreme in Xception architecture by Chollet (2016).”

“Learning long-range dependencies with Recurrent Neural Networks (RNN) is challenging due to the vanishing and exploding gradient problems (Bengio et al., 1994; Pascanu et al., 2013).”

“In case of LSTM networks, ConvLSTM (Shi et al., 2015), has been introduced to better exploit possible spatiotemporal correlations, which is conceptually similar to grouping.”

“LSTM networks (Hochreiter & Schmidhuber, 1997) have been successfully used in language modeling (Jozefowicz et al., 2016; Shazeer et al., 2017), speech recognition (Xiong et al., 2016), machine translation (Wu et al., 2016), and many other tasks.”

“Learning long-range dependencies with Recurrent Neural Networks (RNN) is challenging due to the vanishing and exploding gradient problems (Bengio et al., 1994; Pascanu et al., 2013).”

“We consider LSTM cell with projection of size p , LSTMP, where Equation 1 is computed as follows (Sak et al., 2014; Zaremba et al., 2014).”

“Recent advancements in factorization strategies [14] and conditional computation [15] have significantly enhanced computational efficiency while maintaining model performance.”

“Language modeling requires memory to learn long term dependency to predict the next word in a sequence [61].”

“Recent work has improved computing efficiency by employing factorization methods [8] and conditional computation [13] , as well as enhancing model performance in the latter scenario.”

“The closest work that we know of is in Kuchaiev & Ginsburg (2017), which factorizes the LSTM cell matrices during training in a highly optimized framework; however, they do so at the cost of an increase of perplexity.”

“[20] present two simple ways of reducing the number of parameters and accelerating the training of large LSTM networks, which are matrix factorization and partitioning of LSTM matrix into the independent groups.”

“The idea is that LSTM can be divided into several sub-LSTMs (Kuchaiev and Ginsburg, 2017).”

“In fact, prior work on tensorized RNNs and LSTMs (Kuchaiev and Ginsburg 2017; Novikov et al. 2015) supports this generalization.”

“Recent research has made significant improvements in computational efficiency through factorization techniques [3] and conditional computation [4] , while also improving the performance of the model in the latter case.”

“The number of parameters in LSTM layer can be calculated as: 4 × H × (( d + 1) + H ) [29], where H is the size of latent feature and d is the input size.”

“To address this, techniques have been developed to reduce training time [28], [29].”