Compressing Language Models using Doped Kronecker Products
Open accessAbstract
Kronecker Products (KP) have been used to compress IoT RNN Applications by 15-38x compression factors, achieving better results than traditional compression methods. However when KP is applied to large Natural Language Processing tasks, it leads to significant accuracy loss (approx 26%). This paper proposes a way to recover accuracy otherwise lost when applying KP to large NLP tasks, by allowing additional degrees of freedom in the KP matrix. More formally, we propose doping, a process of adding an extremely sparse overlay matrix on top of the pre-defined KP structure. We call this compression method doped kronecker product compression. To train these models, we present a new solution to the phenomenon of co-matrix adaption (CMA), which uses a new regularization scheme called co matrix dropout regularization (CMR). We present experimental results that demonstrate compression of a large language model with LSTM layers of size 25 MB by 25x with 1.4% loss in perplexity score. At 25x compression, an equivalent pruned network leads to 7.9% loss in perplexity score, while HMD and LMF lead to 15% and 27% loss in perplexity score respectively.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The method is called doped Kronecker product (DKP) compression using co-matrix adaptation (CMA) that follows the co-matrix row dropout regularization (CMR) scheme.
“The newly introduced method is called the doped Kronecker product (DKP) compression by using the co-matrix adaptation (CMA) that follows the comatrix row dropout regularization (CMR) scheme.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Thakker et al., 2020
The Conclusion (p. 4) calls the technique doped Kronecker product (DKP) compression and says it uses co-matrix row dropout regularization (CMR). But it describes co-matrix adaptation (CMA) as a training issue that CMR manages, not as a method DKP uses or a step that follows CMR. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThakker et al. (2020) devised a method to recover the accuracy loss by the KP structure for natural language processing.
“In (Thakker et al., 2020), a method is devised to recover the accuracy loss by the KP structure for natural language processing.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Thakker et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv.org · doi:10.48550/arxiv.1409.2329 · Cited by 3,086 · see citing works
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and ActivationsPreprint
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · Journal of machine learning research · doi:10.48550/arxiv.1609.07061 · Cited by 2,196 · see citing works
Deep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104 · see citing works
Self-citationTo prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805 · see citing works
BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1Preprint
Courbariaux, M., & Bengio, Y. (2016) · arXiv (Cornell University) · Cited by 484 · see citing works
Weighted-Entropy-Based Quantization for Deep Neural Networks
Park, E., Ahn, J., & Yoo, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.761 · Cited by 274 · see citing works
Structured Transforms for Small-Footprint Deep LearningPreprint
Sindhwani, V., Sainath, T. N., & Kumar, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.01722 · Cited by 200 · see citing works
Compressing recurrent neural network with tensor train
Tjandra, A., Sakti, S., & Nakamura, S. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966420 · Cited by 110 · see citing works
Factorization tricks for LSTM networks
Kuchaiev, O., & Ginsburg, B. (2017) · arXiv · doi:10.48550/arxiv.1703.10722 · Cited by 99 · cited 4× · see citing works
“Recently, Kronecker Products (KP) were used to compress IoT applications by 15-38x compression factors (Thakker et al., 2019d;b) and achieves better accuracy than pruning (Zhu & Gupta, 2017), lowrank matrix factorization (LMF) and small baseline (SB).”
InfluentialCompression of recurrent neural networks for efficient language modeling
Grachev, A. M., Ignatov, D. I., & Savchenko, A. V. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.03.057 · Cited by 42 · see citing works
Compressing RNNs for IoT devices by 15-38x using Kronecker Products
Thakker, U., Beu, J., Gope, D., Zhou, C., Fedorov, I., Dasika, G., & Mattina, M. (2019) · arXiv · doi:10.48550/arxiv.1906.02876 · Cited by 30 · see citing works
Self-citationRun-Time Efficient RNN Compression for Inference on Edge Devices
Thakker, U., Beu, J. G., Gope, D., Dasika, G. S., & Mattina, M. (2019) · 2019 2nd Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) · doi:10.1109/emc249363.2019.00013 · Cited by 21 · see citing works
Self-citationTernary Hybrid Neural-Tree Networks for Highly Constrained IoT ApplicationsPreprint
Gope, D., Dasika, G. S., & Mattina, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01531 · Cited by 19 · see citing works
Self-citationSkipping RNN State Updates without Retraining the Original Model
Tao, J., Thakker, U., Dasika, G. S., & Beu, J. G. (2019) · doi:10.1145/3362743.3362965 · Cited by 19 · see citing works
Self-citationTernary MobileNets via Per-Layer Hybrid Filter Banks
Gope, D., Beu, J. G., Thakker, U., & Mattina, M. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00362 · Cited by 16 · see citing works
Self-citationDeepTwist: Learning Model Compression via Occasional Weight DistortionPreprint
Lee, D., Kapoor, P., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12823 · Cited by 16 · see citing works
Measuring scheduling efficiency of RNNs for NLP applicationsPreprint
Thakker, U., Dasika, G. S., Beu, J. G., & Mattina, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03302 · Cited by 14 · see citing works
Self-citationRetraining-Based Iterative Weight Quantization for Deep Neural NetworksPreprint
Lee, D., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11233 · Cited by 11 · see citing works
Pushing the limits of RNN Compression
Thakker, U., Fedorov, I., Beu, J. G., Gope, D., Zhou, C., Dasika, G. S., & Mattina, M. (2019) · Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) · doi:10.1109/emc2-nips53020.2019.00012 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
1 of the 20 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR) , 2016.
Cited by
Sorted by citations
A review on TinyML: State-of-the-art and prospects
Ray, P. P. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.11.019 · Cited by 460 · see citing works
Federated and edge learning for large language models
Piccialli, F., Chiaro, D., Qi, P., Bellandi, V., & Damiani, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102840 · Cited by 49 · see citing works
Run-Time Efficient RNN Compression for Inference on Edge Devices
Thakker, U., Beu, J. G., Gope, D., Dasika, G. S., & Mattina, M. (2019) · 2019 2nd Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) · doi:10.1109/emc249363.2019.00013 · Cited by 21 · see citing works
Self-citationKronecker Decomposition for GPT Compression
Edalati, A., Tahaei, M. S., Rashid, A., Nia, V. P., Clark, J. J., & Rezagholizadeh, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.24 · Cited by 21 · see citing works
Tahaei, M. S., Charlaix, E., Nia, V. P., Ghodsi, A., & Rezagholizadeh, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.154 · Cited by 12 · see citing works
Tahaei, M. S., Charlaix, E., Nia, V. P., Ghodsi, A., & Rezagholizadeh, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06243 · Cited by 14 · see citing works
Rank and run-time aware compression of NLP Applications
Thakker, U., Beu, J. G., Gope, D., Dasika, G. S., & Mattina, M. (2020) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2020.sustainlp-1.2 · Cited by 10 · see citing works
Self-citationCompressing RNNs to Kilobyte Budget for IoT Devices Using Kronecker Products
Thakker, U., Fedorov, I., Zhou, C., Gope, D., Mattina, M., Dasika, G., & Beu, J. (2021) · ACM Journal on Emerging Technologies in Computing Systems · doi:10.1145/3440016 · Cited by 8 · see citing works
Self-citation
- Authors
- Thakker, U., Whatmough, P. N., Liu, Z.-G., Mattina, M., & Beu, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2001.08896
- OpenAlex
- W3002071578
- Cited by
- 7
- References
- 20
- Citations by year
- 2024: 1 · 2022: 2 · 2021: 2 · 2020: 1 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- LSTM, network pruning, low-rank factorization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
