ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needs
Open accessPreprint — published as 10.1109/tai.2025.3605569
Abstract
Conventional Low-Rank Adaptation (LoRA) methods employ a fixed rank, imposing uniform adaptation across transformer layers and attention heads despite their heterogeneous learning dynamics. This paper introduces Adaptive Rank Dynamic LoRA (ARD-LoRA), a novel framework that automates rank allocation through learnable scaling factors. These factors are optimized via a meta-objective balancing task performance and parameter efficiency, incorporating $\ell_1$ sparsity for minimal rank and Total Variation regularization for stable rank transitions. ARD-LoRA enables continuous, differentiable, per-head rank adaptation. Experiments on LLAMA-3.1-70B and PaliGemma-2 demonstrate ARD-LoRA's efficacy, achieving up to 99.3% of full fine-tuning performance with only 0.32% trainable parameters, outperforming strong baselines like DoRA and AdaLoRA. Furthermore, it reduces multimodal adaptation memory by 41%. These results establish dynamic, fine-grained rank allocation as a critical paradigm for efficient foundation model adaptation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1× · see citing works
“1-70B [ 10] and PaliGemma-2 [11] has advanced AI but posed challenges for downstream task adaptation.”
The approximation of one matrix by another of lower rank
Eckart, C., & Young, G. (1936) · Psychometrika · doi:10.1007/bf02288367 · Cited by 4,497 · cited 2× · see citing works
“The Eckart–Young–Mirsky [37], [38] theorem tells us that the best rank-r approximation of Δ W l,h in the Frobenius norm is obtained by truncating the SVD to its first r singular values and corresponding singular vectors.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00686 · Cited by 3,703 · cited 1× · see citing works
“We evaluate on the following benchmarks. a) Language tasks : MMLU [30], BigBench–Hard [31], and GSM8K [32] for reasoning and knowledge-intensive tasks. b) Vision-language tasks : VQAv2 [33], Visual dialog [34], and GQA [35].”
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1× · see citing works
“PEFT methods have evolved from adapter layers [18] and prompt tuning [19] to LoRA [12] and its quantized version QLoRA [20].”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 7× · see citing works
“Dynamic Rank Head-Specific Multimodal LoRA [12] × × QLoRA [20] × × AdaLoRA [21] DyLoRA [22] × IncreLoRA [23] SoRA [24] × DoRA [25] × × MoLA [13] × × DPD-LoRA [29] × × ARD-LoRA (Ours) Note: Symbol legend: This article introduces adaptive rank dynamic LoRA (ARD-LoRA), a framework that formulates rank…”
InfluentialParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 1× · see citing works
“PEFT methods have evolved from adapter layers [18] and prompt tuning [19] to LoRA [12] and its quantized version QLoRA [20].”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · cited 1× · see citing works
“However, conventional LoRA employs a static, uniform rank, disregarding the heterogeneous adaptation needs of transformer layers and attention heads [13], [14].”
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 1× · see citing works
“Upper layers (17–24) show a more uniform rank distribution across heads, suggesting development of holistic reasoning capabilities requiring balanced adaptation.”
SYMMETRIC GAUGE FUNCTIONS AND UNITARILY INVARIANT NORMS
Mirsky, L. (1960) · The Quarterly Journal of Mathematics · doi:10.1093/qmath/11.1.50 · Cited by 835 · cited 2× · see citing works
“The Eckart–Young–Mirsky [37], [38] theorem tells us that the best rank-r approximation of Δ W l,h in the Frobenius norm is obtained by truncating the SVD to its first r singular values and corresponding singular vectors.”
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · cited 1× · see citing works
“Multimodal models exhibit asymmetric adaptation needs between vision and language components [15].”
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 1× · see citing works
“To assess the efficacy of ARD-LoRA, we conduct comprehensive experiments evaluating: 1) performance and parameter efficiency on language tasks; 2) adaptation dynamics in vision-language models; and 3) computational and memory overhead.”
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 2× · see citing works
“Out-of-distribution (OOD) sig-nifies performance on related, unseen datasets (MMLU-tuned model on BigBench–Hard [31]).”
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · cited 8× · see citing works
“Dynamic Rank Head-Specific Multimodal LoRA [12] × × QLoRA [20] × × AdaLoRA [21] DyLoRA [22] × IncreLoRA [23] SoRA [24] × DoRA [25] × × MoLA [13] × × DPD-LoRA [29] × × ARD-LoRA (Ours) Note: Symbol legend: This article introduces adaptive rank dynamic LoRA (ARD-LoRA), a framework that formulates rank…”
InfluentialMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · cited 1× · see citing works
“We evaluate on the following benchmarks. a) Language tasks : MMLU [30], BigBench–Hard [31], and GSM8K [32] for reasoning and knowledge-intensive tasks. b) Vision-language tasks : VQAv2 [33], Visual dialog [34], and GQA [35].”
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445 · cited 1× · see citing works
“We evaluate on the following benchmarks. a) Language tasks : MMLU [30], BigBench–Hard [31], and GSM8K [32] for reasoning and knowledge-intensive tasks. b) Vision-language tasks : VQAv2 [33], Visual dialog [34], and GQA [35].”
Valipour, M., Rezagholizadeh, M., Kobyzev, I., et al. (2022) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2210.07558 · Cited by 337 · cited 5×
“Dynamic Rank Head-Specific Multimodal LoRA [12] × × QLoRA [20] × × AdaLoRA [21] DyLoRA [22] × IncreLoRA [23] SoRA [24] × DoRA [25] × × MoLA [13] × × DPD-LoRA [29] × × ARD-LoRA (Ours) Note: Symbol legend: This article introduces adaptive rank dynamic LoRA (ARD-LoRA), a framework that formulates rank…”
InfluentialMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 1× · see citing works
“We evaluate on the following benchmarks. a) Language tasks : MMLU [30], BigBench–Hard [31], and GSM8K [32] for reasoning and knowledge-intensive tasks. b) Vision-language tasks : VQAv2 [33], Visual dialog [34], and GQA [35].”
PaliGemma 2: A Family of Versatile VLMs for Transfer
Steiner, A., Pinto, A. S., Tschannen, M., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2412.03555 · Cited by 260 · cited 1×
“1-70B [ 10] and PaliGemma-2 [11] has advanced AI but posed challenges for downstream task adaptation.”
Sparse Low-rank Adaptation of Pre-trained Language Models
Ding, N., Lv, X.-T., Wang, Q., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2311.11696 · Cited by 151 · cited 2×
“Dynamic Rank Head-Specific Multimodal LoRA [12] × × QLoRA [20] × × AdaLoRA [21] DyLoRA [22] × IncreLoRA [23] SoRA [24] × DoRA [25] × × MoLA [13] × × DPD-LoRA [29] × × ARD-LoRA (Ours) Note: Symbol legend: This article introduces adaptive rank dynamic LoRA (ARD-LoRA), a framework that formulates rank…”
Higher Layers Need More LoRA Experts
Gao, C.-Y., Chen, K., Rao, J., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2402.08562 · Cited by 116 · cited 1×
“Other specialized techniques include dynamic rank-selective LoRA [26], QDyLoRA [27], and layer-specific meth-ods like MoLA [13], AlphaLoRA [28], and DPD-LoRA [29] for multimodality.”
Low-Rank Adaptation for Foundation Models: A Comprehensive Review
Yang, M.-L., Chen, J.-L., Tao, J.-K., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2501.00365 · Cited by 67 · cited 1×
“Fixed-rank LoRA can thus be inefficient: high ranks may waste 19%–27% GPU memory [16], while low ranks can degrade performance by 12%– 15% [17].”
DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution
Mao, Y., Huang, K.-Y., Guan, C.-H., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2405.17357 · Cited by 67 · cited 7×
“…Rank Head-Specific Multimodal LoRA [12] × × QLoRA [20] × × AdaLoRA [21] DyLoRA [22] × IncreLoRA [23] SoRA [24] × DoRA [25] × × MoLA [13] × × DPD-LoRA [29] × × ARD-LoRA (Ours) Note: Symbol legend: This article introduces adaptive rank dynamic LoRA (ARD-LoRA), a framework that formulates rank…”
InfluentialIncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning
Zhang, F. F., Li, L.-Z., Chen, J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.12043 · Cited by 54 · cited 5×
“Dynamic Rank Head-Specific Multimodal LoRA [12] × × QLoRA [20] × × AdaLoRA [21] DyLoRA [22] × IncreLoRA [23] SoRA [24] × DoRA [25] × × MoLA [13] × × DPD-LoRA [29] × × ARD-LoRA (Ours) Note: Symbol legend: This article introduces adaptive rank dynamic LoRA (ARD-LoRA), a framework that formulates rank…”
InfluentialAdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., & Zhao, T. (2023) · arXiv · doi:10.48550/arxiv.2303.10512 · Cited by 47 · cited 6× · see citing works
“Dynamic Rank Head-Specific Multimodal LoRA [12] × × QLoRA [20] × × AdaLoRA [21] DyLoRA [22] × IncreLoRA [23] SoRA [24] × DoRA [25] × × MoLA [13] × × DPD-LoRA [29] × × ARD-LoRA (Ours) Note: Symbol legend: This article introduces adaptive rank dynamic LoRA (ARD-LoRA), a framework that formulates rank…”
InfluentialQDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
Rajabzadeh, H., Valipour, M., Zhu, T.-S., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2402.10462 · Cited by 36 · cited 1×
“Other specialized techniques include dynamic rank-selective LoRA [26], QDyLoRA [27], and layer-specific meth-ods like MoLA [13], AlphaLoRA [28], and DPD-LoRA [29] for multimodality.”
Lu, H.-D., Zhao, C., Xue, M., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2412.01004 · Cited by 31 · cited 3×
“Other specialized techniques include dynamic rank-selective LoRA [26], QDyLoRA [27], and layer-specific meth-ods like MoLA [13], AlphaLoRA [28], and DPD-LoRA [29] for multimodality.”
Flagged first, then by citations
18 of the 44 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
DTL: Parameter- and Memory-Efficient Disentangled Vision Learning
Fu, M., Zhu, K., Ding, Z., & Wu, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3616318 · Cited by 1 · see citing works
“Variants like AdaLoRA [33], ARD-LoRA [34], Delta-LoRA [35], and SuperLoRA [36] further optimized rank allocation and model flexibility.”
Nawalny, M., Łępicki, M. Z., Latkowski, T., et al. (2025) · Applied Sciences · doi:10.3390/app16010217
FI-LoRA: Dynamic and Lightweight Low-Rank Adaptation Guided by Fisher Information
Zeng, C., & Zhao, S. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3673210
“Veryrecently,methodssuchasFR-LoRA[26] haveintegratedFisherInformationwithLoRAformultilingual continuallearning,whileARD-LoRA[27]andDoRA[28] investigatedynamicrankallocationorweightdecompositionto enhanceadaptation.”
- Authors
- Shinwari, H. U. K., & Usama, M.
- Affiliations
- San Francisco Bay University (US)
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Volume
- 7
- Issue
- 3
- Pages
- 1666-1676
- ISSN
- 2691-4581 (ISSN-L)
- arXiv
- 2506.18267
- OpenAlex
- W4413979290
- Semantic Scholar
- 2ad32392ae5d905ef328d453d537b39f899a57db
- Cited by
- 3
- References
- 44
- Citations by year
- 2026: 1 · 2025: 2
- FWCI
- 1.21
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- low-rank adaptation, parameter-efficient fine-tuning, foundation models, total variation regularization, memory reduction, Llama 3.1, PaliGemma
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
