CoChat
Sign Up
PreprintCited by 4Topic ModelingNatural Language Processing TechniquesAdvanced Neural Network Applications

The DOI matches its DataCite record.

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression

Open access
Saxena, Utkarsh, Saha, Gobinda, Choudhary, Sakshi, Roy, Kaushik·2024·arXiv
DOI 10.48550/arxiv.2408.05646arXiv 2408.05646
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) represent a groundbreaking advancement in the domain of natural language processing due to their impressive reasoning abilities. Recently, there has been considerable interest in increasing the context lengths for these models to enhance their applicability to complex tasks. However, at long context lengths and large batch sizes, the key-value (KV) cache, which stores the attention keys and values, emerges as the new bottleneck in memory usage during inference. To address this, we propose Eigen Attention, which performs the attention operation in a low-rank space, thereby reducing the KV cache memory overhead. Our proposed approach is orthogonal to existing KV cache compression techniques and can be used synergistically with them. Through extensive experiments over OPT, MPT, and Llama model families, we demonstrate that Eigen Attention results in up to 40% reduction in KV cache sizes and up to 60% reduction in attention operation latency with minimal drop in performance. Code is available at https://github.com/UtkarshSaxena1/EigenAttn.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.05646

Cites

Statistics

Figures

Figure 1 Comparison between (a) Standard Attention and (b) Eigen Attention. Eigen Attention utilizes lower dimensional ( r ≪ d ) query, key, and value projection matrices than the standard attention operation, leading to KV cache compression and compute FLOPs benefits.
Figure 1 Comparison between (a) Standard Attention and (b) Eigen Attention. Eigen Attention utilizes lower dimensional ( r ≪ d ) query, key, and value projection matrices than the standard attention operation, leading to KV cache compression and compute FLOPs benefits.CC-BY · From the PDF
Figure 2 Eigenvalue spectrum analysis for OPT-30b model. (a), (b) The Y-axis is the normalized cumulative eigenvalue value after performing SVD on the key value representation matrix, and the X-axis is an index of the largest eigenvalue. (c), (d) Dimensions of the low-rank matrices with normalized cumulative eigenvalue of 0.9.
Figure 2 Eigenvalue spectrum analysis for OPT-30b model. (a), (b) The Y-axis is the normalized cumulative eigenvalue value after performing SVD on the key value representation matrix, and the X-axis is an index of the largest eigenvalue. (c), (d) Dimensions of the low-rank matrices with normalized cumulative eigenvalue of 0.9.CC-BY · From the PDF
Figure 3 PPL on Wikitext with different KV cache sizes in GB ( n = 2048) obtained via different quantization precision and group size. For Eigen Attention, we compress the KV cache to 0.6x and then apply quantization.
Figure 3 PPL on Wikitext with different KV cache sizes in GB ( n = 2048) obtained via different quantization precision and group size. For Eigen Attention, we compress the KV cache to 0.6x and then apply quantization.CC-BY · From the PDF
Figure 4 Ablation Study. (a) Average accuracy (AvgAcc) on zero-shot tasks trained on Llama-2-7b with an increasing number of calibration samples. (b) Perplexity (PPL) vs fine-tuning steps on the C4 dataset for MPT and Llama family of models.
Figure 4 Ablation Study. (a) Average accuracy (AvgAcc) on zero-shot tasks trained on Llama-2-7b with an increasing number of calibration samples. (b) Perplexity (PPL) vs fine-tuning steps on the C4 dataset for MPT and Llama family of models.CC-BY · From the PDF
Figure 5 Layerwise rank assignment for key and value determined by Eigen Attention for OPT-30b with 40% compressed KV cache.
Figure 5 Layerwise rank assignment for key and value determined by Eigen Attention for OPT-30b with 40% compressed KV cache.CC-BY · From the PDF
Figure 6 Comparison between (a) Standard Attention and (b) Eigen Attention for LLMs with RoPE (Touvron et al., 2023). Eigen Attention stores low dimensional representation of key and value in KV cache and applies an additional transformation before applying RoPE.
Figure 6 Comparison between (a) Standard Attention and (b) Eigen Attention for LLMs with RoPE (Touvron et al., 2023). Eigen Attention stores low dimensional representation of key and value in KV cache and applies an additional transformation before applying RoPE.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 40

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  2. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Preprint, arXiv:2403.09054. · arXiv
  3. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query trans- former models from multi-head checkpoints. arXiv preprint arXiv:2305.13245. · arXiv
  4. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150. · arXiv
  5. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439.
  6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. · arXiv
  7. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm con- text window beyond 2 million tokens. arXiv preprint arXiv:2402.13753. · arXiv
  8. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. Preprint, arXiv:2104.08758. · arXiv
  9. Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. 2022. Rank diminishing in deep neural networks. In Proceedings of the 36th International Conference on Neural In- formation Processing Systems, NIPS '22, Red Hook, NY, USA.
  10. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: To- wards 10 million context length llm inference with kv cache quantization. Preprint, arXiv:2401.18079. · arXiv
  11. Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022. Language model compression with weighted low-rank factorization. In International Conference on Learning Representa- tions.
  12. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  13. Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. 2024. Gear: An efficient kv cache compression recipe for near-lossless generative inference of llm. Preprint, arXiv:2403.05527. · arXiv
  14. Ayush Kaushal, Tejas Vaidhya, and Irina Rish. 2023. Lord: Low rank decomposition of monolingual code llms for one-shot compression. Preprint, arXiv:2309.14021. · arXiv
  15. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023a. Ef- ficient memory management for large language model serving with pagedattention. Preprint, arXiv:2309.06180. · arXiv
  16. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023b. Effi- cient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  17. Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. LoSparse: Structured compression of large language models based on low-rank and sparse approximation. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 20336-20350.
  18. PMLR.
  19. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before genera- tion. arXiv preprint arXiv:2404.14469. · arXiv
  20. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for llm compression and acceleration. Preprint, arXiv:2306.00978. · arXiv
  21. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyril- lidis, and Anshumali Shrivastava. 2024. Scis- sorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36. Llama-3. Introducing meta llama 3: The most capable openly available llm to date. Accessed: 2010-06-07.
  22. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. arXiv preprint arXiv:1609.07843. · arXiv
  23. Yue Niu, Saurav Prakash, and Salman Avestimehr. 2024. Atp: Enabling fast llm serving via attention on top principal keys. Preprint, arXiv:2403.02352. · arXiv
  24. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  25. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022. Efficiently scaling transformer in- ference. Preprint, arXiv:2211.05102. · arXiv
  26. Ofir Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. · arXiv
  27. Gobinda Saha, Isha Garg, and Kaushik Roy. 2021. Gra- dient projection memory for continual learning. In International Conference on Learning Representa- tions.
  28. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commu- nications of the ACM, 64(9):99-106.
  29. Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150. · arXiv
  30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models. Preprint, arXiv:2307.09288. · arXiv
  31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Hug- gingface's transformers: State-of-the-art natural lan- guage processing. Preprint, arXiv:1910.03771. · arXiv
  32. Haoyi Wu and Kewei Tu. 2024. Layer-condensed kv cache for efficient inference of large language models. Preprint, arXiv:2405.10637. · arXiv
  33. June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024. No token left be- hind: Reliable kv cache compression via importance- aware mixed precision quantization. Preprint, arXiv:2402.18096. · arXiv
  34. Hao Yu and Jianxin Wu. 2023. Compressing trans- formers: Features are low-rank, but weights are not! Proceedings of the AAAI Conference on Artificial Intelligence, 37(9):11007-11015. · DOI
  35. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. · arXiv
  36. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2024a. Soaring from 4k to 400k: Extending llm's context with activation beacon. arXiv preprint arXiv:2401.03462. · arXiv
  37. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. · arXiv
  38. Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. 2024b. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069. · arXiv
  39. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuan- dong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H 2 o: Heavy-hitter ora- cle for efficient generative inference of large language models. Preprint, arXiv:2306.14048. · arXiv
  40. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. Kivi : Plug-and-play 2bit kv cache quantization with streaming asymmetric quantiza- tion. · DOI · Paper page

Cited by

Sorted by citations

See all 4

Metadata

Authors
Saxena, U., Saha, G., Choudhary, S., & Roy, K.
Affiliations
Purdue University West Lafayette (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
15332-15344
DOI
10.48550/arxiv.2408.05646
arXiv
2408.05646
OpenAlex
W4404781139
Cited by
4
References
0
Citations by year
2026: 3 · 2025: 1
FWCI
1.18
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation
  • U.S. Department of Energy
  • Semiconductor Research Corporation
  • Defense Advanced Research Projects Agency
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
key-value cache compression, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Eigen Attention: Attention in Low-Rank Space for KV Cache Compression. https://app.cochat.ai/papers/10.48550/arxiv.2408.05646
The paper
Saxena, U., Saha, G., Choudhary, S., & Roy, K. (2024). Eigen Attention: Attention in Low-Rank Space for KV Cache Compression. arXiv. https://doi.org/10.48550/arxiv.2408.05646