The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.
FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning
Open accessAbstract
Large language models (LLMs) have demonstrated great capabilities in various natural language understanding and generation tasks.These pre-trained LLMs can be further improved for specific downstream tasks by fine-tuning.However, the adoption of LLM in real-world applications can be hindered by privacy concerns and the resource-intensive nature of model training and fine-tuning.When multiple entities have similar interested tasks but cannot directly share their local data due to privacy regulations, federated learning (FL) is a mainstream solution to leverage the data of different entities.Besides avoiding direct data sharing, FL can also achieve rigorous data privacy protection, model intelligent property protection, and model customization via composition with different techniques.Despite the aforementioned advantages of FL, fine-tuning LLMs in FL settings still lacks adequate support from the existing frameworks and, therefore, faces challenges in optimizing the consumption of significant communication and computational resources, preparing various data for different tasks, and satisfying diverse information protection demands.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
FederatedScope-LLM (FS-LLM) is an open-source package built on FederatedScope for end-to-end federated fine-tuning of large language models, with benchmarking datasets, PEFT algorithms, and efficiency-oriented training operators. Experiments show federated PEFT, especially LoRA, can approach centralized fine-tuning while greatly reducing communication and computation costs, and also expose open challenges in privacy-preserving, personalized, and hyperparameter-optimized federated LLM fine-tuning.
Key findings
- FS-LLM provides three main modules: LLM-Benchmarks for federated fine-tuning datasets and evaluation, LLM-AlgZoo for PEFT and no-full-model algorithms, and LLM-Trainer for acceleration/resource-efficient operators and extensibility.
- In LLaMA-7B experiments, LoRA outperformed P-tuning and prompt tuning across all three tasks under federated settings: Fed-CodeAlpaca 13.29±0.10 vs 9.71±0.66 and 9.63±0.36; Fed-Dolly 46.57±0.24 vs 41.50±0.32 and 40.72±0.64; Fed-GSM8K-3 14.25±1.37 vs 11.75±0.39 and 9.86±0.59. Federated fine-tuning also substantially outperformed local fine-tuning and was competitive with global/centralized fine-tuning.
- PEFT algorithms had similar GPU memory usage (13,440–13,538 MB) but very different message sizes: full fine-tuning 12,852.01 MB, LoRA 21.40 MB, P-tuning 256.48 MB, prompt tuning 0.17 MB. At 100 Mbps, per-round transmission time ranged from about 0.01 seconds with prompt tuning to 40 seconds with P-tuning.
- FedOT without full model access remained competitive at 20% layer dropping (Fed-CodeAlpaca 7.14±2.75, Fed-Dolly 44.88±0.75, Fed-GSM8K-3 9.02±0.71), but at 50% dropping performance collapsed (0.16±0.15, 37.01±2.34, 2.98±0.98). pFedMe did not outperform FedAvg under any tested data distribution, and validation loss ranks were inconsistent with evaluation-score ranks in FedHPO.
Methods
- The package is built on FederatedScope and automates dataset preprocessing, federated fine-tuning execution/simulation, and performance evaluation. It includes curated federated datasets: Fed-CodeAlpaca (9 clients, ~8.0k samples), Fed-Dolly (8 clients, ~15.0k samples), and Fed-GSM8K-3 (3 clients, ~7.5k samples), paired with HumanEval, HELM, and GSM8K-test evaluations.
- Main experiments fine-tune LLaMA-7B with LoRA, P-tuning, and prompt tuning under global, federated, and local scenarios using FedAvg; OPT-2.7B is tested with LoRA. FedOT uses the first and last two LLaMA-7B layers as the adapter and an emulator compressed by dropping 20% or 50% of remaining layers. pFL compares pFedMe and FedAvg with LoRA under Dirichlet-split Fed-CodeAlpaca variants (α from 0.05 to 50.0).
- Experimental protocol: 500 communication rounds, 30 local update steps, batch size 1, three random seeds, average evaluation score with standard deviation. Evaluation score is Pass@1 for HumanEval, a mixture of 16 HELM subtask metrics, and accuracy for GSM8K-test. Hardware includes Nvidia A100 80GB and V100 32GB machines.
Limitations
- Stated by the paper: prefix-tuning was excluded from experiments because its implementation contained unresolved issues; OPT-2.7B failed some HELM subtasks due to exceeded input length; pFedMe underperformed FedAvg, partly because half-precision training restricted the usable hyperparameter space; FedOT has a trade-off between compression rate and model performance; validation loss may not reliably indicate generalization performance; cross-device federated LLM fine-tuning is left as future work.
- Evident from the text: the main benchmark tables report only LLaMA-7B and OPT-2.7B, with additional LLM results deferred to the full version; the paper also notes unresolved compatibility issues between efficient training operators and pFL algorithms, and between shared-base-LLM adapter designs and some pFL algorithms.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 26 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, from the table, we can observe that the computation time varies due to different GPUs, with almost a twofold difference.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, pFedMe surprisingly does not outperform FedAvg under any data distribution, which shows different results with previous language models in pFL-bench [8].”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, there are large differences in the message sizes, which in turn lead to large variations in the transmission time.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All algorithms with federated fine-tuning can significantly outperform those under the local scenario”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, we observe that the performance of fine-tuned LLMs in FL is highly dependent on the choice of hyperparameters.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > E Study of FedHPO for LLMs
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which shows the performance of fine-tuning with FedAvg gradually approaches that under the global scenario as data heterogeneity decreases.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, we reveal a significant discrepancy between the ranks of validation loss and the ranks of final generalization performance in evaluation tasks during fine-tuning.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“fine-tuning with different PEFT algorithms has negligible impact on the GPU memory consumption.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, OPT-2.7Bfails in the evaluation of some subtasks in HELM due to the exceeded input length, which limits the scope of the application of the previous-generation language models with fewer parameters.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“FedOTstill achieves competitive performance compared to some of those PEFT algorithms, even though the clients cannot access the full model.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- It may state two separate claims as one. 13
- Probably not a substantial claim. 9
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the results show that when the dropping rate increases to 50% from 20%, the model loses a large amount of knowledge from the pre-training stage, almost fails to retain the capacity of the CoT and code generation, and hardly acquire new knowledge from finetuning.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“There is a trade-off between the compression rate and the performance of LLMs: increasing the compression rate enhances the privacy of LLMs but degrades their performance.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though the model performance of OPT-2.7B is improved with federated fine-tuning compared to those under the local scenario, it is not significant.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for a client with 100Mbps bandwidth, the transmission time (upload and download) per round ranges from about 0.01 seconds (with prompt tuning) to 40 seconds (with P-tuning) when using different PEFT algorithms.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“one feasible solution is federated learning (FL) [2,28,36,41,50,66], which allows multiple entities to learn model collaboratively without sharing data.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 1 Introduction
What the sources say
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- Rests on an assumption: Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FedOT sacrifices communication efficiency for model performance, making it effective in scenarios where clients cannot access the full model.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“These algorithms perform fine-tuning by only training (additional) modules with limited parameters, known as adapters, but keep other parameters frozen.Compared to full-parameter finetuning in FL, clients only need to transmit the adapters in each communication round, which reduces the transmission time to tens of seconds or even seconds.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 4.1 Reducing Communication & Computation Costs with PEFT Algorithms
What the sources say
- Not checked yetcites [22]
- Not checked yetcites [34] source
- Not checked yetcites [38]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To conduct the experiments uniformly and fairly, we fix the FL-specific hyperparameters and the hyperparameters that have a large impact on the computation cost for all experiments.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 6.1 Effectiveness of PEFT Algorithms in FS-LLM
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“No existing FL package contains comprehensive and efficient implementations of LLM finetuning algorithms and a standardized benchmark for comparing the model performance, communication cost, and computation overhead when federated fine-tuning LLMs.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 1 Introduction
What the sources say
- Not checked yetcites [4]
- Not checked yetcites [54]
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All the experiments are conducted on the machines with the same hardware configuration three times with different random seeds.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 6.1 Effectiveness of PEFT Algorithms in FS-LLM
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Though these algorithms fine-tune a small number of parameters of LLMs, they can still be computationally expensive.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Appendix A The Scope of FS-LLM
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Compared to full-parameter fine-tuning, these PEFT algorithms significantly reduce memory consumption, training time, and communication costs for fine-tuning LLMs.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Appendix A The Scope of FS-LLM
What the sources say
- Not checked yetcites LoRA [22]
- Not checked yetcites prefix-tuning [34]
- Not checked yetcites P-tuning [38]
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Communication and computation resources of clients in FL are usually limited.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We accommodate both closed-source and open-source LLMs through a unified interface, built on FS event-driven architecture that simplifies the extension and customization of federated fine-tuning algorithms.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 4.3 Extendability by Unified Interfaces for Federated Fine-tuning
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Note that there is generally a distribution shift between the fine-tuning and evaluation datasets, making it much more challenging than those in other domains of FL [6,18,61].”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 3.2 Federated LLM Fine-tuning Evaluation
What the sources say
- Not checked yetcites [6]
- Not checked yetcites [18]
- Not checked yetcites [61]
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure the consistency of the evaluation results, we containerize the runtime environment of these evaluation tasks to docker images for conveniently assessing the performance of the federated LLM fine-tuning.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 3.2 Federated LLM Fine-tuning Evaluation
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“we exclude prefix-tuning from our experiments, because its implementation in [40] contains unresolved issues when we were preparing this package”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 6.1 Effectiveness of PEFT Algorithms in FS-LLM
What the sources say
- Not checked yetcites [40]
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This poses a great challenge for finding the optimal hyperparameters, as it requires a fine-grained and exhaustive search over the hyperparameter space.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, we introduce LLM-Benchmarks, the first convenient and fair module to evaluate federated LLM fine-tuning.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 3 LLM-Benchmarks: A Complete Pipeline for Benchmarking
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“It is unclear whether the existing algorithms for solving advanced FL problems, such as personalized FL (pFL) [8,58] and federated hyperparameter optimization (FedHPO) [62], are still effective with different federated fine-tuning algorithms for LLMs.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 1 Introduction
What the sources say
- Not checked yetcites [8]
- Not checked yetcites [58]
- Not checked yetcites [62]
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“Fe-dOT safeguards both the intelligent property of the model providers and the data privacy of the clients, while leveraging the distributed data for adapting LLMs to specific domains.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 4.2 Fine-tuning without Accessing Full Model
What the sources say
- Not checked yetcites [64] source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“For instance, pFL [58] and FedHPO [62] are two advanced FL tasks that can significantly improve model performance in FL.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 5.2 Training Paradigm for Extendability towards Advanced FL Tasks
What the sources say
- Not checked yetcites [58]
- Not checked yetcites [62]
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With such an implementation, by adding a pFL hook-like function, we can achieve the cooperation of any PEFT algorithms with the implemented pFL algorithms, which provides strong extensibility for federated fine-tuning LLMs in personalized settings.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 5.2 Training Paradigm for Extendability towards Advanced FL Tasks
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearThe quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Because pre-trained LLMs are of great intelligent property value and may not belong to clients, it might be necessary to let clients conduct federated fine-tuning without accessing the full model (e.g., closedsource LLMs).”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 1 Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
The paper relies on this without support.
Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.
“However, fine-tuning LLMs entails a high demand for computational resources and a substantial amount of domain data that may not be sharable due to privacy concerns.”
The paper relies on this without support.
Communication and computation resources of clients in FL are usually limited.
“As communication and computation resources of clients are usually limited in FL”
The paper relies on this without support.
Fine-tuning LLMs in FL is very costly.
“Since fine-tuning LLMs in FL is very costly”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A distinguishing characteristic of MoE models is their flexibility in managing large datasets, capable of amplifying model capacity by over a thousand times while only experiencing minor reductions in computational efficiency [289], [292].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [292]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
GPU memory components sum to 120 GB, not 112 GB
MinorThe stated 28 GB for the model plus another 92 GB for the optimizer and gradients totals 120 GB, not the reported 112 GB.
Reported 112 · recomputed 120
“full-parameter fine-tuning LLaMA-7B requires about 28GB of GPU memory for the model.In addition, the SGD optimizer and the gradients need another 92GB of GPU memory, leading to at least 112GB of GPU memory in total”
Efficiency experiments list a third GPU type
MinorThe efficiency section says all its experiments used two hardware types, A100 and V100, but Table 3 also reports computation times on a 3090.
“All the experiments are conducted on two types of hardware, both with 512GB of RAM: Nvidia A100 GPU (80GB) with Intel Xeon Platinum 8369B CPU and Nvidia V100 GPU (32GB) with Intel Xeon Platinum 8163 CPU.”
“| Comp. Time on 3090 (Sec.) | OOM | 0.18±0.05 | 0.18±0.04 | 0.16±0.05 |”
Blanket A100 hardware claim conflicts with reported V100 results
MinorAppendix D says all experiments used the same A100-based hardware configuration, whereas Table 3 reports results obtained on a V100.
“All the experiments are conducted on the machines with the same hardware configuration: Nvidia A100 GPU (80GB) with Intel Xeon Platinum 8369B CPU and 512GB of RAM.”
“| Comp. Time on V100 (Sec.) | OOM | 0.33±0.07 | 0.33±0.08 | 0.33±0.10 |”
Citations
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationTaking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,195
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“Fine-tuning can enrich LLMs with domain knowledge, enhance their specific ability, improve the fairness and reliability of the outputs, and prevent certain damage caused by hallucination (Ji et al., 2023).”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Self-citationFederated Optimization: Distributed Machine Learning for On-Device Intelligence
Konečný, J., McMahan, H. B., Ramage, D., & Richtárik, P. (2016) · arXiv · doi:10.48550/arxiv.1610.02527 · Cited by 1,563
Flagged first, then by citations
Research works printed but not linked: 24
- Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, et al. 2019. Towards federated learning at scale: System design. In Proc. of machine learning and systems (MLSys'19), Vol. 1. 374--388.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 1877--1901.
- Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM.
- Zhongxiang Dai, Bryan Kian Hsiang Low, and Patrick Jaillet. 2020. Federated Bayesian Optimization via Thompson Sampling. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 9687--9699.
- L. Peter Deutsch. 1996 b. GZIP file format specification version 4.3. RFC. Network Working Group.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of NAACL-HLT (NAACL-HLT'19). 4171--4186.
- Dawei Gao, Daoyuan Chen, Zitao Li, Yuexiang Xie, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. 2023. FS-Real: A Real-World Cross-Device Federated Learning Platform. PVLDB, Vol. 16 (2023), 4046--4049.
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. In Proc. of the International Conference on Machine Learning (ICML'21). 2790--2799.
- Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proc. of the International Conference on Learning Representations (ICLR'22).
- Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic Best Arm Identification and Hyperparameter Optimization. In Proc. of the Artificial intelligence and statistics (AISTATS'16). 240--248.
- Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient Low-Rank Hypercomplex Adapter Layers. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'21), Vol. 34. 1022--1035.
- Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. 2022. FedScale: Benchmarking Model and System Performance of Federated Learning at Scale. In Proc. of the International Conference on Machine Learning (ICML'22), Vol. 162. 11814--11827.
- Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021. Ditto: Fair and Robust Federated Learning Through Personalization. In Proc. of the International Conference on Machine Learning (ICML'21), Vol. 139. 6357--6368.
- Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT Understands, Too. arXiv preprint arXiv:2103.10385 (2021). · arXiv
- H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proc. of the Artificial intelligence and statistics (AISTATS'17). 1273--1282.
- Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018. Mixed Precision Training. In Proc. of the International Conference on Learning Representations (ICLR'18).
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'19), Vol. 32. 8024--8035.
- Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterFusion: Non-Destructive Task Composition for Transfer Learning. arXiv preprint arXiv:2005.00247 (2020). · arXiv
- Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. 2024. Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes. In International Conference on Machine Learning.
- Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyang Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. ZeRO-Offload: Democratizing Billion-Scale Model Training. In Proc. of the USENIX Annual Technical Conference (USENIX ATC'21). 551--564.
- Canh T Dinh, Nguyen Tran, and Josh Nguyen. 2020. Personalized Federated Learning with Moreau Envelopes. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 21394--21405.
- Zhen Wang, Weirui Kuang, Ce Zhang, Bolin Ding, and Yaliang Li. 2023. FedHPO-Bench: A Benchmark Suite for Federated Hyperparameter Optimization. In Proc. of the International Conference on Machine Learning (ICML'23). 35908--35948.
- Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2023. GLM-130B: An Open Bilingual Pre-trained Model. In Proc. of the International Conference on Learning Representations (ICLR'23).
- Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. 2023. Single-shot Hyper-parameter Optimization for Federated Learning. In Proc. of the International Conference on Learning Representations (ICLR'23).
Cited by
Sorted by citations
Items 101–141 of 141
Mbounkap, J., Djigo, H., Masheh, T., Nzooh, A., & Kar, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7685-1.ch013 · Cited by 1
Fan, C., Chen, Y., Jiang, Q., Yu, H. P., & Yang, Z. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3679423
Privacy-Preserving Model Compression: Adaptive Pruning for Federated Split Learning of LLMs
Attota, D. C., Le, L., & Xie, Y. (2025) · International Conference on Machine Learning and Applications · doi:10.1109/icmla66185.2025.00053
Self-citationA Review of Research on Fine-Tuning Techniques for Large Language Models
Han, Y., Li, X., Guo, T., Qin, Z., Lu, G., Li, R., & Jiang, R. (2025) · International Conference on Computational Modeling, Simulation and Data Analysis (CMSDA) · doi:10.1145/3796731.3796807
Robust Federated Fine-Tuning Over Heterogeneous and Unreliable Communication Networks
Wang, Y., Dai, Z., Zhou, J., Xiao, F., & Chang, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461620
Self-citationA Unified Four-Stage Dynamic Cycle for Robust Federated Fine-Tuning of Large Language Models
Tan, B., Ren, J., Li, Y., & Chaddad, A. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463011
Self-citationFederated Learning: A Comprehensive Review of Methods, Challenges, Applications and Trends
BashariRad, B., Choenni, S., & Omar, M. S. B. A. A. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1015368
Net4LMs: Wireless Network-Aided Language Models
Du, H., Chen, X., Liu, Y., & Huang, K. (2026) · Wireless networks · doi:10.1007/978-3-032-20194-2_4
Fine-Tuning for Code Intelligence: Evaluating LLMs on Custom Programming Benchmarks
Sahu, H., Barbhuyan, M. F. K., Kamavisdar, J., Mishra, A., Pandey, R., & Singh, S. P. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-17184-9_28
Kang, N., & Im, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6837687
Yang, Z. L., Qiu, X., & Cui, H. L. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539341
Harmon, I., Sánchez-Cano, I. R., Brailsford, J., & Fishe, J. N. (2026) · Prehospital Emergency Care · doi:10.1080/10903127.2026.2683523
Selective Layer Adaptive Attacks Against Federated Large Language Models
Olapojoye, R., Faiyaz, A., & Salman, T. (2026) · IEEE International Conference on Secure IoT, Assured and Trusted Computing (SATC) · doi:10.1109/satc69565.2026.11542419
A Survey on Advances of Foundation Models in Federated Learning
Zhu, S., Chen, J., & Ochiai, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_2
Wang, B., & Hua, X. (2026) · Cluster Computing · doi:10.1007/s10586-026-06141-6
Zhang, C., Wei, B., & Yang, L. (2026) · Journal of Transportation Engineering Part A Systems · doi:10.1061/jtepbs.teeng-9794
Ma, J., Dang, S., Fang, Z., Zhang, Z., Ge, Z., & Li, X. J. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571275
FedDynMask: Efficient Federated Fine-Tuning for Edge LLMs via Dynamic Sparse Masking
Wang, Y., Gao, Z., Zhang, Y., & Wang, R. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571368
Liu, Y., Han, W., Cai, C., Yuan, X., & Wang, C. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571688
Deng, G., Chen, S., Luo, Y., Liu, Y., Wang, S., Cai, Z., Liu, L., Jia, X., & Fu, S. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00205
Per-FedLoRA: Personalized Federated LoRA Fine-Tuning for Multi-Task Large Language Models
Chen, Z. L., Ji, Y., Peng, H. H., Wang, B., Shi, L., Zeng, Z., & Zhang, X. (2026) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc69287.2026.11580063
Himeur, Y., Alnaseri, O., Atalla, S., Mansoor, W., & Al-Ahmad, H. (2026) · IEEE International Conference on Smart Information Systems and Technologies (SIST) · doi:10.1109/sist61674.2026.11596283
SeqFedRPC : Sequential Federated Learning With Regularized Parameter Clustering
Mao, P., Du, T. Y., Wu, Z., Wang, D., & Xiao, H. (2026) · Transactions on Emerging Telecommunications Technologies · doi:10.1002/ett.70477
Tan, K. R., Andrea, F., Tzeliang, A. Y., & Kemala, A. P. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11620748
From Issues to Insights: RAG-based Explanation Generation from Software Engineering Artifacts
Pöttgen, D., Sadeghi, M., Unterbusch, M., & Vogelsang, A. (2026) · IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) · doi:10.1145/3786164.3788448
ADAS-LORA: Efficient Federated LLM Tuning in Edge Systems by Decoupling Compute and Network
Campos, P. E., Liebe, L., Nguyen, T.-T., Tau, N.-Q., Wu, Y., & Lee, D. (2026) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw72724.2026.00056
Jiang, Y., & Song, B. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3720660
HeimdaLLM: Efficient Cloud-assisted Federated Fine-tuning with Zeroth-Order Rectification for LLMs
He, S., Zhou, J., Li, L., Wu, Y., & Xiao, M. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817661
Self-citationModality-incomplete Federated Learning via personalized prompt generation and alignment
Wang, S., Qu, Z., Liu, Y., Kan, S., & Wang, J. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114603
FedPOD-LoRA: Federated Fine-Tuning Based on Proximal Optimization and Dual-LoRA CollaborationPreprint
Li, Y., Guo, M., Wang, Y., Wu, A., Xin, Y., Yuan, Y., & Ren, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7292747
Self-citationFedCod: An Efficient Coded Communication Protocol for Cross-Silo Federated Learning
Yan, P., Li, J., Wang, H., Hua, Y., Song, T., Peng, L., & Guan, H. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661089
Intelligent agents: architectures, learning paradigms, deployment models, and applications
Dritsas, E., & Τρίγκα, Μ. (2026) · Evolutionary Intelligence · doi:10.1007/s12065-026-01259-3
Privacy-Preserving Cross-Workflow Recommendation via Federated Learning on Semantic Knowledge Graphs
Zhang, J., Liu, C., Xie, X., Han, Z., Ramachandran, R., Lee, T. J., & Lee, S. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00027 · Cited by 1
ASLoRA: An Adaptive Scaling Method for Rotational LoRA in Federated Large Language Model Fine-Tuning
Han, X., Wu, S., Chen, C., Tian, H., Liu, G., & Dou, W. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00028
Kumar, B., Verma, A., & Buyya, R. (2026) · Wireless Networks · doi:10.1007/s11276-026-04224-1
Wu, S., Wang, Y., & Zhang, H. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-026-03286-z
Böcekçi, S. C., & Yıldız, K. (2026) · Cluster Computing · doi:10.1007/s10586-026-06630-8 · Cited by 1
Current Research and Open Gaps in Federated Large Language Models: A Systematic Review
Ribeiro, R. G., Guimarães, F. G., Lima, H. S., Martins, V. A. D. O., Bastos, A. L., Santana, L. D., Pereira, B. C. G., Ferreira, C. B., Souza, J. B. F. D., & Lima, M. C. C. D. (2026) · SBC Reviews on Computer Science · doi:10.5753/reviews.2026.6627
ReCliFF: Adaptive Orthogonal Decoupling for Federated Fine-Tuning of Medical MLLMs
Jiang, Y., Feng, C.-M., Sun, R., & Zhang, L. (2026) · doi:10.1109/cvprf71909.2026.00507
DGIML: A Decentralized Identity and Game-Theoretic Incentive Mechanism for LLM Collaboration
Lan, W., Ying, Z., Cai, J., & Xiong, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6840-5_25
Talukder, N. (2026) · International Journal of Modern Science and Research Technology · doi:10.67317/ijmsrt26oct084
Metadata
- Authors
- Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J.
- Affiliations
- Alibaba Group (China) (CN)
- Alibaba Group (United States) (US)
- ORCID
- Venue
- Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2024-08-24
- Pages
- 5260-5271
- OpenAlex
- W4401863415
- Cited by
- 142
- References
- 17
- Citations by year
- 2026: 59 · 2025: 63 · 2024: 16
- FWCI
- 40.72
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- federated learning, large language models, LLM fine-tuning, privacy-preserving machine learning, data privacy, communication efficiency, computational resource efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
