CoChat
Sign Up
PreprintCited by 1Advanced Neural Network ApplicationsTopic ModelingDomain Adaptation and Few-Shot Learning

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications

Open access
Jaiswal, Ajay, Wang, Yifan, Yin, Lu, Liu, Shiwei, Chen, Runjin, Zhao, Jiawei, Grama, Ananth, Tian, Yuandong +1 more·2024·arXiv
DOI 10.48550/arxiv.2407.11239arXiv 2407.11239
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models' (LLMs) weight matrices can often be expressed in low-rank form with potential to relax memory and compute resource requirements. Unlike prior efforts that focus on developing novel matrix decompositions, in this work we study the non-uniform low-rank properties of weight matrices in LLMs through the lens of stabilizing gradient subspace. First, we provide a theoretical framework to understand the stabilization of gradient subspaces through Hessian analysis. Second, we empirically establish an important relationship between gradient dynamics and low-rank expressiveness of weight matrices. Our findings reveal that different LLM components exhibit varying levels of converged low-rank structures, necessitating variable rank reduction across them to minimize drop in performance due to compression. Drawing on this result, we present Weight Low-Rank Projection(WeLore) that unifies weight compression and memory-efficient fine-tuning into one, in a data-agnostic and one-shot manner. When used as a compression technique, WeLore categorizes weight matrices into Low-rank Components (LRCs) and Non-Low-rank Components (N-LRCs) and suitably encodes them for minimum performance loss. Our gradient dynamics perspective illustrates that LRCs tend to have better fine-tuning capabilities and their standalone fine-tuning can closely mimic and sometimes outperform the training loss trajectory and performance of full fine-tuning with notable memory and compute footprint reduction. Codes are available at https://github.com/VITA-Group/WeLore.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.11239

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 53

  1. Biderman, D., Ortiz, J. G., Portes, J., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673, 2024a. · arXiv
  2. Biderman, D., Portes, J., Ortiz, J. J. G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C., and Cunningham, J. P. Lora learns less and forgets less, 2024b. URL https://arxiv.org/ abs/2405.09673. · arXiv
  3. Brock, A., Lim, T., Ritchie, J. M., and Weston, N. Freeze- out: Accelerate training by progressively freezing layers. arXiv preprint arXiv:1706.04983, 2017. · arXiv
  4. Chen, D., Bolton, J., and Manning, C. D. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv:1606.02858, 2016. · arXiv
  5. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314. · arXiv
  6. Diao, S., Huang, Z., Xu, R., Li, X., Lin, Y., Zhou, X., and Zhang, T. Black-box prompt learning for pre-trained lan- guage models. arXiv preprint arXiv:2201.08531, 2022. · arXiv
  7. Ergen, T. and Pilanci, M. Revealing the structure of deep neural networks via convex duality, 2021. URL https: //arxiv.org/abs/2002.09773.
  8. Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference on Machine Learning, pp. 10323- 10337. PMLR, 2023.
  9. Galanti, T., Siegel, Z. S., Gupte, A., and Poggio, T. Sgd and weight decay secretly minimize the rank of your neural network, 2024. URL https://arxiv.org/abs/ 2206.05794.
  10. Hambardzumyan, K., Khachatrian, H., and May, J. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.00121, 2021. · arXiv
  11. Hao, Y., Cao, Y., and Mou, L. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293, 2024. · arXiv
  12. Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024. · arXiv
  13. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
  14. Hsu, Y.-C., Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022a. · arXiv
  15. Hsu, Y.-C., Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization, 2022b. URL https://arxiv.org/ abs/2207.00112. · arXiv
  16. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021a. · arXiv
  17. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021b. URL https://arxiv. org/abs/2106.09685.
  18. Iyyer, M., Boyd-Graber, J. L., Claudino, L. M. B., Socher, R., and Daumé, H. A neural network for factoid ques- tion answering over paragraphs. In Conference on Empirical Methods in Natural Language Processing, 2014. URL https://api.semanticscholar. org/CorpusID:216034672.
  19. Jain, S., Keshava, V., Sathyendra, S. M., Fernandes, P., Liu, P., Neubig, G., and Zhou, C. Multi-dimensional evaluation of text summarization with in-context learning. arXiv preprint arXiv:2306.01200, 2023. · arXiv
  20. Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z., and Yang, Y. Compressing llms: The truth is rarely pure and never simple. arXiv preprint arXiv:2310.01382, 2023. · arXiv
  21. Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z., and Yang, Y. Compressing llms: The truth is rarely pure and never simple, 2024. URL https://arxiv.org/ abs/2310.01382. · arXiv
  22. Ji, Z. and Telgarsky, M. Directional convergence and align- ment in deep learning, 2020. URL https://arxiv. org/abs/2006.06657.
  23. Jiang, K., Wu, D., and Jiang, H. Freebaseqa: A new factoid qa data set matching trivia-style question- answer pairs with freebase. In North American Chap- ter of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar. org/CorpusID:174800890.
  24. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. · arXiv
  25. Kaushal, A., Vaidhya, T., and Rish, I. Lord: Low rank decomposition of monolingual code llms for one-shot compression. arXiv preprint arXiv:2309.14021, 2023a. · arXiv
  26. Kaushal, A., Vaidhya, T., and Rish, I. Lord: Low rank decomposition of monolingual code llms for one-shot compression, 2023b. URL https://arxiv.org/ abs/2309.14021. · arXiv
  27. Le, T. and Jegelka, S. Training invariances and the low- rank phenomenon: beyond linear networks, 2022. URL https://arxiv.org/abs/2201.11968. · arXiv
  28. Li, S., Yuan, G., Dai, Y., Zhang, Y., Wang, Y., and Tang, X. Smartfrz: An efficient training framework using attention- based layer freezing. arXiv preprint arXiv:2401.16720, 2024. · arXiv
  29. Li, Y., Yu, Y., Zhang, Q., Liang, C., He, P., Chen, W., and Zhao, T. Losparse: Structured compression of large lan- guage models based on low-rank and sparse approxima- tion, 2023. URL https://arxiv.org/abs/2306. 11222.
  30. Lialin, V., Shivagunde, N., Muckatira, S., and Rumshisky, A. Relora: High-rank training through low-rank up- dates, 2023a. URL https://arxiv.org/abs/ 2307.05695.
  31. Lialin, V., Shivagunde, N., Muckatira, S., and Rumshisky, A. Stack more layers differently: High-rank training through low-rank updates. arXiv preprint arXiv:2307.05695, 2023b. · arXiv
  32. Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024. · arXiv
  33. Liu, Y., Agarwal, S., and Venkataraman, S. Autofreeze: Automatically freezing model blocks to accelerate fine- tuning. arXiv preprint arXiv:2102.01386, 2021. · arXiv
  34. Meng, F., Wang, Z., and Zhang, M. Pissa: Principal sin- gular values and singular vectors adaptation of large lan- guage models, 2024. URL https://arxiv.org/ abs/2404.02948. · arXiv
  35. Ongie, G. and Willett, R. The role of linear layers in nonlinear interpolating networks, 2022. URL https: //arxiv.org/abs/2202.00856.
  36. Renduchintala, A., Konuk, T., and Kuchaiev, O. Tied-lora: Enhacing parameter efficiency of lora with weight tying. arXiv preprint arXiv:2311.09578, 2023. · arXiv
  37. Saha, R., Srivastava, V., and Pilanci, M. Matrix compression via randomized low rank and low precision factorization. Advances in Neural Information Processing Systems, 36, 2023.
  38. Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285, 2023. · arXiv
  39. Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023. · arXiv
  40. Tian, Y., Yu, L., Chen, X., and Ganguli, S. Understanding self-supervised learning with dual deep networks, 2021. URL https://arxiv.org/abs/2010.00578. · arXiv
  41. Timor, N., Vardi, G., and Shamir, O. Implicit regularization towards rank minimization in relu networks, 2022. URL https://arxiv.org/abs/2201.12760. · arXiv
  42. Wang, H., Agarwal, S., Tanaka, Y., Xing, E., Papailiopoulos, D., et al. Cuttlefish: Low-rank model training without all the tuning. Proceedings of Machine Learning and Systems, 5, 2023a.
  43. Wang, H., Agarwal, S., U-chupala, P., Tanaka, Y., Xing, E. P., and Papailiopoulos, D. Cuttlefish: Low-rank model training without all the tuning, 2023b. URL https: //arxiv.org/abs/2305.02538.
  44. Wang, X., Zheng, Y., Wan, Z., and Zhang, M. Svd- llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024. · arXiv
  45. Xia, W., Qin, C., and Hazan, E. Chain of lora: Efficient fine- tuning of language models via residual learning. arXiv preprint arXiv:2401.04151, 2024. · arXiv
  46. Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Y., Jia, Y., Pechenizkiy, M., Liang, Y., Wang, Z., and Liu, S. Out- lier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175, 2023. · arXiv
  47. Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023. · arXiv
  48. Zhang, L., Zhang, L., Shi, S., Chu, X., and Li, B. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303, 2023. · arXiv
  49. Zhao, J., Zhang, Y., Chen, B., Schäfer, F., and Anandkumar, A. Inrank: Incremental low-rank learning. arXiv preprint arXiv:2306.11250, 2023. · arXiv
  50. Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient llm train- ing by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024a. · arXiv
  51. Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient llm training by gradient low-rank projection, 2024b. URL https: //arxiv.org/abs/2403.03507.
  52. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. · arXiv
  53. Zhong, Z., Friedman, D., and Chen, D. Factual probing is [mask]: Learning vs. learning to recall. arXiv preprint arXiv:2104.05240, 2021. · arXiv

Cited by

Sorted by citations

Authors
Jaiswal, A., Wang, Y., Yin, L., Liu, S., Chen, R., Zhao, J., Grama, A., Tian, Y., & Wang, Z.
Affiliations
  • The University of Texas at Austin (US)
  • Purdue University West Lafayette (US)
  • University of Surrey (GB)
  • University of Oxford (GB)
  • California Institute of Technology (US)
  • Meta (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.11239
arXiv
2407.11239
OpenAlex
W4403753878
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2145346)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
large language models, weight compression, Hessian analysis, gradient dynamics, memory-efficient fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review