The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index; 1 of the 26 sources its claims cite were checked for support.
FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning
Open accessAbstract
Large language models (LLMs) have demonstrated great capabilities in various natural language understanding and generation tasks.These pre-trained LLMs can be further improved for specific downstream tasks by fine-tuning.However, the adoption of LLM in real-world applications can be hindered by privacy concerns and the resource-intensive nature of model training and fine-tuning.When multiple entities have similar interested tasks but cannot directly share their local data due to privacy regulations, federated learning (FL) is a mainstream solution to leverage the data of different entities.Besides avoiding direct data sharing, FL can also achieve rigorous data privacy protection, model intelligent property protection, and model customization via composition with different techniques.Despite the aforementioned advantages of FL, fine-tuning LLMs in FL settings still lacks adequate support from the existing frameworks and, therefore, faces challenges in optimizing the consumption of significant communication and computational resources, preparing various data for different tasks, and satisfying diverse information protection demands.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
FederatedScope-LLM (FS-LLM) is an open-source package built on FederatedScope for end-to-end federated fine-tuning of large language models, with benchmarking datasets, PEFT algorithms, and efficiency-oriented training operators. Experiments show federated PEFT, especially LoRA, can approach centralized fine-tuning while greatly reducing communication and computation costs, and also expose open challenges in privacy-preserving, personalized, and hyperparameter-optimized federated LLM fine-tuning.
Key findings
- FS-LLM provides three main modules: LLM-Benchmarks for federated fine-tuning datasets and evaluation, LLM-AlgZoo for PEFT and no-full-model algorithms, and LLM-Trainer for acceleration/resource-efficient operators and extensibility.
- In LLaMA-7B experiments, LoRA outperformed P-tuning and prompt tuning across all three tasks under federated settings: Fed-CodeAlpaca 13.29±0.10 vs 9.71±0.66 and 9.63±0.36; Fed-Dolly 46.57±0.24 vs 41.50±0.32 and 40.72±0.64; Fed-GSM8K-3 14.25±1.37 vs 11.75±0.39 and 9.86±0.59. Federated fine-tuning also substantially outperformed local fine-tuning and was competitive with global/centralized fine-tuning.
- PEFT algorithms had similar GPU memory usage (13,440–13,538 MB) but very different message sizes: full fine-tuning 12,852.01 MB, LoRA 21.40 MB, P-tuning 256.48 MB, prompt tuning 0.17 MB. At 100 Mbps, per-round transmission time ranged from about 0.01 seconds with prompt tuning to 40 seconds with P-tuning.
- FedOT without full model access remained competitive at 20% layer dropping (Fed-CodeAlpaca 7.14±2.75, Fed-Dolly 44.88±0.75, Fed-GSM8K-3 9.02±0.71), but at 50% dropping performance collapsed (0.16±0.15, 37.01±2.34, 2.98±0.98). pFedMe did not outperform FedAvg under any tested data distribution, and validation loss ranks were inconsistent with evaluation-score ranks in FedHPO.
Methods
- The package is built on FederatedScope and automates dataset preprocessing, federated fine-tuning execution/simulation, and performance evaluation. It includes curated federated datasets: Fed-CodeAlpaca (9 clients, ~8.0k samples), Fed-Dolly (8 clients, ~15.0k samples), and Fed-GSM8K-3 (3 clients, ~7.5k samples), paired with HumanEval, HELM, and GSM8K-test evaluations.
- Main experiments fine-tune LLaMA-7B with LoRA, P-tuning, and prompt tuning under global, federated, and local scenarios using FedAvg; OPT-2.7B is tested with LoRA. FedOT uses the first and last two LLaMA-7B layers as the adapter and an emulator compressed by dropping 20% or 50% of remaining layers. pFL compares pFedMe and FedAvg with LoRA under Dirichlet-split Fed-CodeAlpaca variants (α from 0.05 to 50.0).
- Experimental protocol: 500 communication rounds, 30 local update steps, batch size 1, three random seeds, average evaluation score with standard deviation. Evaluation score is Pass@1 for HumanEval, a mixture of 16 HELM subtask metrics, and accuracy for GSM8K-test. Hardware includes Nvidia A100 80GB and V100 32GB machines.
Limitations
- Stated by the paper: prefix-tuning was excluded from experiments because its implementation contained unresolved issues; OPT-2.7B failed some HELM subtasks due to exceeded input length; pFedMe underperformed FedAvg, partly because half-precision training restricted the usable hyperparameter space; FedOT has a trade-off between compression rate and model performance; validation loss may not reliably indicate generalization performance; cross-device federated LLM fine-tuning is left as future work.
- Evident from the text: the main benchmark tables report only LLaMA-7B and OPT-2.7B, with additional LLM results deferred to the full version; the paper also notes unresolved compatibility issues between efficient training operators and pFL algorithms, and between shared-base-LLM adapter designs and some pFL algorithms.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
15 Own interpretation · 15 Own result · 10 Not checked yet · 7 No basis given · 3 Own method · 1 Partly supported · accuracy not yet measured
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, from the table, we can observe that the computation time varies due to different GPUs, with almost a twofold difference.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, pFedMe surprisingly does not outperform FedAvg under any data distribution, which shows different results with previous language models in pFL-bench [8].”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, there are large differences in the message sizes, which in turn lead to large variations in the transmission time.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All algorithms with federated fine-tuning can significantly outperform those under the local scenario”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, we observe that the performance of fine-tuned LLMs in FL is highly dependent on the choice of hyperparameters.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > E Study of FedHPO for LLMs
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which shows the performance of fine-tuning with FedAvg gradually approaches that under the global scenario as data heterogeneity decreases.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, we reveal a significant discrepancy between the ranks of validation loss and the ranks of final generalization performance in evaluation tasks during fine-tuning.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“fine-tuning with different PEFT algorithms has negligible impact on the GPU memory consumption.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, OPT-2.7Bfails in the evaluation of some subtasks in HELM due to the exceeded input length, which limits the scope of the application of the previous-generation language models with fewer parameters.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“FedOTstill achieves competitive performance compared to some of those PEFT algorithms, even though the clients cannot access the full model.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 13
- Probably not a substantial claim. 9
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These algorithms perform fine-tuning by only training (additional) modules with limited parameters, known as adapters, but keep other parameters frozen.Compared to full-parameter finetuning in FL, clients only need to transmit the adapters in each communication round, which reduces the transmission time to tens of seconds or even seconds.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 4.1 Reducing Communication & Computation Costs with PEFT Algorithms
What the sources say
- Not checked yetcites [22]
- Partly supportedcites [34] sourceExperimental — accuracy not yet measured
The Introduction and §2 Related Work say adapter-tuning freezes pretrained parameters and trains small, additional adapter modules. The paper does not establish that federated-learning clients transmit only adapters each round or that transmission falls to tens of seconds or seconds. Its §5.3 speed measurements concern prefix-tuning, not federated communication. (LLM, conf=0.88)
- Not checked yetcites [38]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the results show that when the dropping rate increases to 50% from 20%, the model loses a large amount of knowledge from the pre-training stage, almost fails to retain the capacity of the CoT and code generation, and hardly acquire new knowledge from finetuning.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“There is a trade-off between the compression rate and the performance of LLMs: increasing the compression rate enhances the privacy of LLMs but degrades their performance.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though the model performance of OPT-2.7B is improved with federated fine-tuning compared to those under the local scenario, it is not significant.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for a client with 100Mbps bandwidth, the transmission time (upload and download) per round ranges from about 0.01 seconds (with prompt tuning) to 40 seconds (with P-tuning) when using different PEFT algorithms.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“one feasible solution is federated learning (FL) [2,28,36,41,50,66], which allows multiple entities to learn model collaboratively without sharing data.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 1 Introduction
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FedOT sacrifices communication efficiency for model performance, making it effective in scenarios where clients cannot access the full model.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To conduct the experiments uniformly and fairly, we fix the FL-specific hyperparameters and the hyperparameters that have a large impact on the computation cost for all experiments.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 6.1 Effectiveness of PEFT Algorithms in FS-LLM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“No existing FL package contains comprehensive and efficient implementations of LLM finetuning algorithms and a standardized benchmark for comparing the model performance, communication cost, and computation overhead when federated fine-tuning LLMs.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 1 Introduction
What the sources say
- Not checked yetcites [4]
- Not checked yetcites [54]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All the experiments are conducted on the machines with the same hardware configuration three times with different random seeds.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 6.1 Effectiveness of PEFT Algorithms in FS-LLM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Though these algorithms fine-tune a small number of parameters of LLMs, they can still be computationally expensive.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Appendix A The Scope of FS-LLM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Compared to full-parameter fine-tuning, these PEFT algorithms significantly reduce memory consumption, training time, and communication costs for fine-tuning LLMs.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Appendix A The Scope of FS-LLM
What the sources say
- Not checked yetcites LoRA [22]
- Not checked yetcites prefix-tuning [34]
- Not checked yetcites P-tuning [38]
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Communication and computation resources of clients in FL are usually limited.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We accommodate both closed-source and open-source LLMs through a unified interface, built on FS event-driven architecture that simplifies the extension and customization of federated fine-tuning algorithms.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 4.3 Extendability by Unified Interfaces for Federated Fine-tuning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Note that there is generally a distribution shift between the fine-tuning and evaluation datasets, making it much more challenging than those in other domains of FL [6,18,61].”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 3.2 Federated LLM Fine-tuning Evaluation
What the sources say
- Not checked yetcites [6]
- Not checked yetcites [18]
- Not checked yetcites [61]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure the consistency of the evaluation results, we containerize the runtime environment of these evaluation tasks to docker images for conveniently assessing the performance of the federated LLM fine-tuning.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 3.2 Federated LLM Fine-tuning Evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“we exclude prefix-tuning from our experiments, because its implementation in [40] contains unresolved issues when we were preparing this package”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 6.1 Effectiveness of PEFT Algorithms in FS-LLM
What the sources say
- Not checked yetcites [40]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This poses a great challenge for finding the optimal hyperparameters, as it requires a fine-grained and exhaustive search over the hyperparameter space.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, we introduce LLM-Benchmarks, the first convenient and fair module to evaluate federated LLM fine-tuning.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 3 LLM-Benchmarks: A Complete Pipeline for Benchmarking
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“It is unclear whether the existing algorithms for solving advanced FL problems, such as personalized FL (pFL) [8,58] and federated hyperparameter optimization (FedHPO) [62], are still effective with different federated fine-tuning algorithms for LLMs.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 1 Introduction
What the sources say
- Not checked yetcites [8]
- Not checked yetcites [58]
- Not checked yetcites [62]
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Fe-dOT safeguards both the intelligent property of the model providers and the data privacy of the clients, while leveraging the distributed data for adapting LLMs to specific domains.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 4.2 Fine-tuning without Accessing Full Model
What the sources say
- Not checked yetcites [64] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“For instance, pFL [58] and FedHPO [62] are two advanced FL tasks that can significantly improve model performance in FL.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 5.2 Training Paradigm for Extendability towards Advanced FL Tasks
What the sources say
- Not checked yetcites [58]
- Not checked yetcites [62]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With such an implementation, by adding a pFL hook-like function, we can achieve the cooperation of any PEFT algorithms with the implemented pFL algorithms, which provides strong extensibility for federated fine-tuning LLMs in personalized settings.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 5.2 Training Paradigm for Extendability towards Advanced FL Tasks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Because pre-trained LLMs are of great intelligent property value and may not belong to clients, it might be necessary to let clients conduct federated fine-tuning without accessing the full model (e.g., closedsource LLMs).”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper relies on this without support.
Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.
“However, fine-tuning LLMs entails a high demand for computational resources and a substantial amount of domain data that may not be sharable due to privacy concerns.”
The paper relies on this without support.
Communication and computation resources of clients in FL are usually limited.
“As communication and computation resources of clients are usually limited in FL”
The paper relies on this without support.
Fine-tuning LLMs in FL is very costly.
“Since fine-tuning LLMs in FL is very costly”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A distinguishing characteristic of MoE models is their flexibility in managing large datasets, capable of amplifying model capacity by over a thousand times while only experiencing minor reductions in computational efficiency [289], [292].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [292]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“full-parameter fine-tuning LLaMA-7B requires about 28GB of GPU memory for the model.In addition, the SGD optimizer and the gradients need another 92GB of GPU memory, leading to at least 112GB of GPU memory in total”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 4.1 Reducing Communication & Computation Costs with PEFT Algorithms
Reported 112 · recomputed 120
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > 6.2 Efficiency of PEFT Algorithms in FS-LLM
“All the experiments are conducted on two types of hardware, both with 512GB of RAM: Nvidia A100 GPU (80GB) with Intel Xeon Platinum 8369B CPU and Nvidia V100 GPU (32GB) with Intel Xeon Platinum 8163 CPU.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
“| Comp. Time on 3090 (Sec.) | OOM | 0.18±0.05 | 0.18±0.04 | 0.16±0.05 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > D Experiment Protocols and Results
“All the experiments are conducted on the machines with the same hardware configuration: Nvidia A100 GPU (80GB) with Intel Xeon Platinum 8369B CPU and 512GB of RAM.”
§ FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning > Results and Analysis.
“| Comp. Time on V100 (Sec.) | OOM | 0.33±0.07 | 0.33±0.08 | 0.33±0.10 |”
Citations
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationTaking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,195
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“Fine-tuning can enrich LLMs with domain knowledge, enhance their specific ability, improve the fairness and reliability of the outputs, and prevent certain damage caused by hallucination (Ji et al., 2023).”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Self-citationFederated Optimization: Distributed Machine Learning for On-Device Intelligence
Konečný, J., McMahan, H. B., Ramage, D., & Richtárik, P. (2016) · arXiv · doi:10.48550/arxiv.1610.02527 · Cited by 1,563
Research works printed but not linked: 24
- Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, et al. 2019. Towards federated learning at scale: System design. In Proc. of machine learning and systems (MLSys'19), Vol. 1. 374--388.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 1877--1901.
- Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM.
- Zhongxiang Dai, Bryan Kian Hsiang Low, and Patrick Jaillet. 2020. Federated Bayesian Optimization via Thompson Sampling. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 9687--9699.
- L. Peter Deutsch. 1996 b. GZIP file format specification version 4.3. RFC. Network Working Group.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of NAACL-HLT (NAACL-HLT'19). 4171--4186.
- Dawei Gao, Daoyuan Chen, Zitao Li, Yuexiang Xie, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. 2023. FS-Real: A Real-World Cross-Device Federated Learning Platform. PVLDB, Vol. 16 (2023), 4046--4049.
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. In Proc. of the International Conference on Machine Learning (ICML'21). 2790--2799.
- Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proc. of the International Conference on Learning Representations (ICLR'22).
- Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic Best Arm Identification and Hyperparameter Optimization. In Proc. of the Artificial intelligence and statistics (AISTATS'16). 240--248.
- Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient Low-Rank Hypercomplex Adapter Layers. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'21), Vol. 34. 1022--1035.
- Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. 2022. FedScale: Benchmarking Model and System Performance of Federated Learning at Scale. In Proc. of the International Conference on Machine Learning (ICML'22), Vol. 162. 11814--11827.
- Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021. Ditto: Fair and Robust Federated Learning Through Personalization. In Proc. of the International Conference on Machine Learning (ICML'21), Vol. 139. 6357--6368.
- Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT Understands, Too. arXiv preprint arXiv:2103.10385 (2021). · arXiv
- H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proc. of the Artificial intelligence and statistics (AISTATS'17). 1273--1282.
- Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018. Mixed Precision Training. In Proc. of the International Conference on Learning Representations (ICLR'18).
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'19), Vol. 32. 8024--8035.
- Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterFusion: Non-Destructive Task Composition for Transfer Learning. arXiv preprint arXiv:2005.00247 (2020). · arXiv
- Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. 2024. Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes. In International Conference on Machine Learning.
- Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyang Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. ZeRO-Offload: Democratizing Billion-Scale Model Training. In Proc. of the USENIX Annual Technical Conference (USENIX ATC'21). 551--564.
- Canh T Dinh, Nguyen Tran, and Josh Nguyen. 2020. Personalized Federated Learning with Moreau Envelopes. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 21394--21405.
- Zhen Wang, Weirui Kuang, Ce Zhang, Bolin Ding, and Yaliang Li. 2023. FedHPO-Bench: A Benchmark Suite for Federated Hyperparameter Optimization. In Proc. of the International Conference on Machine Learning (ICML'23). 35908--35948.
- Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2023. GLM-130B: An Open Bilingual Pre-trained Model. In Proc. of the International Conference on Learning Representations (ICLR'23).
- Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. 2023. Single-shot Hyper-parameter Optimization for Federated Learning. In Proc. of the International Conference on Learning Representations (ICLR'23).
Cited by
Sorted by citations
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Foundation and large language models: fundamentals, challenges, opportunities, and social impacts
Myers, D., Mohawesh, R., Chellaboina, V. I., Sathvik, A. L., Venkatesh, P., Ho, Y.-H., Henshaw, H., Al-Hawawreh, M. S., Berdik, D., & Jararweh, Y. (2023) · Cluster Computing · doi:10.1007/s10586-023-04203-7 · Cited by 217
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
“Its two primary approaches are end-to-end fine-tuning, which adjusts all weights of the encoder and classifier [260], [261], and feature-extraction fine-tuning, where the encoder weights are frozen to extract features for a downstream classifier [262], [263], [264].”
BackgroundShukla, S., Rajkumar, S., Sinha, A., Esha, M., Elango, K., & Sampath, V. (2025) · Scientific Reports · doi:10.1038/s41598-025-95858-2 · Cited by 120
On protecting the data privacy of Large Language Models (LLMs) and LLM agents: A literature review
Yan, B., Li, K., Xu, M., Dong, Y., Zhang, Y., Ren, Z., & Cheng, X. (2025) · High-Confidence Computing · doi:10.1016/j.hcc.2025.100300 · Cited by 114
Metadata
- Authors
- Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J.
- Affiliations
- Alibaba Group (China) (CN)
- Alibaba Group (United States) (US)
- ORCID
- Venue
- Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2024-08-24
- Pages
- 5260-5271
- OpenAlex
- W4401863415
- Cited by
- 142
- References
- 17
- Citations by year
- 2026: 59 · 2025: 63 · 2024: 16
- FWCI
- 40.72
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- federated learning, large language models, LLM fine-tuning, privacy-preserving machine learning, data privacy, communication efficiency, computational resource efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
