Low-Rank Bottleneck in Multi-head Attention Models
Open accessAbstract
Attention based Transformer architecture has enabled significant advances in the field of natural language processing. In addition to new pre-training techniques, recent improvements crucially rely on working with a relatively larger embedding dimension for tokens. Unfortunately, this leads to models that are prohibitively large to be employed in the downstream tasks. In this paper we identify one of the important factors contributing to the large embedding size requirement. In particular, our analysis highlights that the scaling between the number of heads and the size of each head in the current architecture gives rise to a low-rank bottleneck in attention heads, causing this limitation. We further validate this in our experiments. As a solution we propose to set the head size of an attention unit to input sequence length, and independent of the number of heads, resulting in multi-head attention layers with provably more expressive power. We empirically show that this allows us to train models with a relatively smaller embedding dimension and with better performance scaling.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The same rank-based constraint governs the capacity of linear or multi-head attention modules: the output context matrix has rank at most rank(K) ≤ d_k, which directly limits their expressivity.
“Finally, the same rank-based constraint governs the capacity of linear or multi-head attention modules. In such architectures, the output context matrix has rank at most rank ( 𝐾 ) ≤ 𝑑 𝑘 , which directly limits their expressivity.”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Bhojanapalli et al. 2020
The Introduction (p. 2) and §2.3 (p. 6) say that small per-head projection dimensions create a low-rank bottleneck that limits the expressive power of multi-head attention. They do not state that the output context matrix has rank at most rank(K) ≤ d_k, or establish the claim for linear attention; the rank constraint they identify is on the projection matrices in each head. (LLM, conf=0.86)
Experimental — accuracy not yet measuredRecent analyses identify a 'low-rank bottleneck' as a capacity-limiting effect in Transformers.
“Recent analyses identify this 'low-rank bottleneck' as a capacity-limiting effect in Transformers (Bhojanapalli et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Bhojanapalli et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 29 · Cited by 46
This paper’s reference list could not be split into single entries.
- Authors
- Bhojanapalli, S., Yun, C., Rawat, A. S., Reddi, S. J., & Kumar, S.
- Affiliations
- Google (United States) (US)
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2002.07028
- OpenAlex
- W3006610127
- Cited by
- 46
- References
- 29
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 6 · 2023: 10 · 2022: 12 · 2021: 5 · 2020: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multi-head attention, embedding dimension, transformer models, model compression, scaling laws
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
