CoChat
Sign Up
Cited by 142Privacy-Preserving Technologies in DataTopic ModelingDomain Adaptation and Few-Shot Learning

FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning

Open access
Kuang, Weirui, Qian, Bingchen, Li, Zitao, Chen, Daoyuan, Gao, Dawei, Pan, Xuchen, Xie, Yuexiang, Li, Yaliang +2 more·2024·Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
DOI 10.1145/3637528.3671573
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) have demonstrated great capabilities in various natural language understanding and generation tasks.These pre-trained LLMs can be further improved for specific downstream tasks by fine-tuning.However, the adoption of LLM in real-world applications can be hindered by privacy concerns and the resource-intensive nature of model training and fine-tuning.When multiple entities have similar interested tasks but cannot directly share their local data due to privacy regulations, federated learning (FL) is a mainstream solution to leverage the data of different entities.Besides avoiding direct data sharing, FL can also achieve rigorous data privacy protection, model intelligent property protection, and model customization via composition with different techniques.Despite the aforementioned advantages of FL, fine-tuning LLMs in FL settings still lacks adequate support from the existing frameworks and, therefore, faces challenges in optimizing the consumption of significant communication and computational resources, preparing various data for different tasks, and satisfying diverse information protection demands.

AI snapshot

Written from the paper’s full text.

TL;DR

FederatedScope-LLM (FS-LLM) is an open-source package built on FederatedScope for end-to-end federated fine-tuning of large language models, with benchmarking datasets, PEFT algorithms, and efficiency-oriented training operators. Experiments show federated PEFT, especially LoRA, can approach centralized fine-tuning while greatly reducing communication and computation costs, and also expose open challenges in privacy-preserving, personalized, and hyperparameter-optimized federated LLM fine-tuning.

Key findings

  • FS-LLM provides three main modules: LLM-Benchmarks for federated fine-tuning datasets and evaluation, LLM-AlgZoo for PEFT and no-full-model algorithms, and LLM-Trainer for acceleration/resource-efficient operators and extensibility.
  • In LLaMA-7B experiments, LoRA outperformed P-tuning and prompt tuning across all three tasks under federated settings: Fed-CodeAlpaca 13.29±0.10 vs 9.71±0.66 and 9.63±0.36; Fed-Dolly 46.57±0.24 vs 41.50±0.32 and 40.72±0.64; Fed-GSM8K-3 14.25±1.37 vs 11.75±0.39 and 9.86±0.59. Federated fine-tuning also substantially outperformed local fine-tuning and was competitive with global/centralized fine-tuning.
  • PEFT algorithms had similar GPU memory usage (13,440–13,538 MB) but very different message sizes: full fine-tuning 12,852.01 MB, LoRA 21.40 MB, P-tuning 256.48 MB, prompt tuning 0.17 MB. At 100 Mbps, per-round transmission time ranged from about 0.01 seconds with prompt tuning to 40 seconds with P-tuning.
  • FedOT without full model access remained competitive at 20% layer dropping (Fed-CodeAlpaca 7.14±2.75, Fed-Dolly 44.88±0.75, Fed-GSM8K-3 9.02±0.71), but at 50% dropping performance collapsed (0.16±0.15, 37.01±2.34, 2.98±0.98). pFedMe did not outperform FedAvg under any tested data distribution, and validation loss ranks were inconsistent with evaluation-score ranks in FedHPO.

Methods

  • The package is built on FederatedScope and automates dataset preprocessing, federated fine-tuning execution/simulation, and performance evaluation. It includes curated federated datasets: Fed-CodeAlpaca (9 clients, ~8.0k samples), Fed-Dolly (8 clients, ~15.0k samples), and Fed-GSM8K-3 (3 clients, ~7.5k samples), paired with HumanEval, HELM, and GSM8K-test evaluations.
  • Main experiments fine-tune LLaMA-7B with LoRA, P-tuning, and prompt tuning under global, federated, and local scenarios using FedAvg; OPT-2.7B is tested with LoRA. FedOT uses the first and last two LLaMA-7B layers as the adapter and an emulator compressed by dropping 20% or 50% of remaining layers. pFL compares pFedMe and FedAvg with LoRA under Dirichlet-split Fed-CodeAlpaca variants (α from 0.05 to 50.0).
  • Experimental protocol: 500 communication rounds, 30 local update steps, batch size 1, three random seeds, average evaluation score with standard deviation. Evaluation score is Pass@1 for HumanEval, a mixture of 16 HELM subtask metrics, and accuracy for GSM8K-test. Hardware includes Nvidia A100 80GB and V100 32GB machines.

Limitations

  • Stated by the paper: prefix-tuning was excluded from experiments because its implementation contained unresolved issues; OPT-2.7B failed some HELM subtasks due to exceeded input length; pFedMe underperformed FedAvg, partly because half-precision training restricted the usable hyperparameter space; FedOT has a trade-off between compression rate and model performance; validation loss may not reliably indicate generalization performance; cross-device federated LLM fine-tuning is left as future work.
  • Evident from the text: the main benchmark tables report only LLaMA-7B and OPT-2.7B, with additional LLM results deferred to the full version; the paper also notes unresolved compatibility issues between efficient training operators and pFL algorithms, and between shared-base-LLM adapter designs and some pFL algorithms.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3637528.3671573

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 26 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Different GPUs yield almost twofold differences in computation time.

    “Moreover, from the table, we can observe that the computation time varies due to different GPUs, with almost a twofold difference.”
    Quote in the paperSubstantial claim
  • Findingown data

    pFedMe does not achieve better performance than FedAvg under any of the tested data distributions.

    “However, pFedMe surprisingly does not outperform FedAvg under any data distribution, which shows different results with previous language models in pFL-bench [8].”
    Quote in the paperSubstantial claim
  • Findingown data

    There are large differences in message sizes among PEFT algorithms, leading to large variations in transmission time.

    “However, there are large differences in the message sizes, which in turn lead to large variations in the transmission time.”
    Quote in the paperSubstantial claim
  • Findingown data

    Federated fine-tuning with PEFT algorithms significantly outperforms local fine-tuning.

    “All algorithms with federated fine-tuning can significantly outperform those under the local scenario”
    Quote in the paperSubstantial claim
  • Findingnone given

    The performance of fine-tuned LLMs in FL depends strongly on the chosen hyperparameters.

    “However, we observe that the performance of fine-tuned LLMs in FL is highly dependent on the choice of hyperparameters.”
    Quote in the paperSubstantial claim
  • May combine two claims. 13
  • Classified as probably not a substantial claim. 9
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    When the dropping rate increases from 20% to 50%, the model loses much pre-training knowledge, almost loses its chain-of-thought and code-generation abilities, and barely gains new knowledge from fine-tuning.

    “the results show that when the dropping rate increases to 50% from 20%, the model loses a large amount of knowledge from the pre-training stage, almost fails to retain the capacity of the CoT and code generation, and hardly acquire new knowledge from finetuning.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Higher compression rates improve privacy but lower LLM performance, revealing a trade-off.

    “There is a trade-off between the compression rate and the performance of LLMs: increasing the compression rate enhances the privacy of LLMs but degrades their performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Federated fine-tuning of OPT-2.7B improves performance compared to local scenario, but the improvement is not significant.

    “Though the model performance of OPT-2.7B is improved with federated fine-tuning compared to those under the local scenario, it is not significant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With 100Mbps bandwidth, one round's transmission takes roughly 0.01 seconds for prompt tuning and 40 seconds for P-tuning.

    “For example, for a client with 100Mbps bandwidth, the transmission time (upload and download) per round ranges from about 0.01 seconds (with prompt tuning) to 40 seconds (with P-tuning) when using different PEFT algorithms.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Federated learning allows multiple entities to collaboratively train a model without sharing their data.

    “one feasible solution is federated learning (FL) [2,28,36,41,50,66], which allows multiple entities to learn model collaboratively without sharing data.”
    Quote in the paperSubstantial claim0 of 6 sources checked
    • cites [2]

      Not checked in the free run.

    • cites [28]

      Not checked in the free run.

    • cites [36] source

      Not checked in the free run.

    • cites [41]

      Not checked in the free run.

    • cites [50] source

      Not checked in the free run.

    • cites [66]

      Not checked in the free run.

    • Rests on an assumption: Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.

  • May combine two claims.

    Interpretationreasoning

    FedOT trades communication efficiency for model performance, making it effective in settings where clients cannot access the full model.

    “FedOT sacrifices communication efficiency for model performance, making it effective in scenarios where clients cannot access the full model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    PEFT algorithms reduce communication and computation costs in federated fine-tuning by training only adapters and transmitting only adapters each round.

    “These algorithms perform fine-tuning by only training (additional) modules with limited parameters, known as adapters, but keep other parameters frozen.Compared to full-parameter finetuning in FL, clients only need to transmit the adapters in each communication round, which reduces the transmission time to tens of seconds or even seconds.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [22]

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

    • cites [38]

      Not checked in the free run.

    • Rests on an assumption: Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    For all experiments, the authors held constant both FL-specific hyperparameters and those strongly affecting computation cost.

    “To conduct the experiments uniformly and fairly, we fix the FL-specific hyperparameters and the hyperparameters that have a large impact on the computation cost for all experiments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Existing FL frameworks do not offer a full set of efficient LLM fine-tuning algorithms or a standard benchmark that compares performance, communication, and computation for federated LLM fine-tuning.

    “No existing FL package contains comprehensive and efficient implementations of LLM finetuning algorithms and a standardized benchmark for comparing the model performance, communication cost, and computation overhead when federated fine-tuning LLMs.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [4]

      Not checked in the free run.

    • cites [54]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Each experiment ran three times with different random seeds on identical hardware.

    “All the experiments are conducted on the machines with the same hardware configuration three times with different random seeds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Even though PEFT algorithms fine-tune only a small number of LLM parameters, they can still be computationally expensive.

    “Though these algorithms fine-tune a small number of parameters of LLMs, they can still be computationally expensive.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    Compared with full-parameter fine-tuning, PEFT algorithms substantially reduce memory consumption, training time, and communication costs for fine-tuning LLMs.

    “Compared to full-parameter fine-tuning, these PEFT algorithms significantly reduce memory consumption, training time, and communication costs for fine-tuning LLMs.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites LoRA [22]

      Not checked in the free run.

    • cites prefix-tuning [34]

      Not checked in the free run.

    • cites P-tuning [38]

      Not checked in the free run.

    • Rests on an assumption: Communication and computation resources of clients in FL are usually limited.

  • May combine two claims.

    Interpretationreasoning

    The unified interfaces support both closed-source and open-source LLMs and simplify extension and customization of federated fine-tuning algorithms.

    “We accommodate both closed-source and open-source LLMs through a unified interface, built on FS event-driven architecture that simplifies the extension and customization of federated fine-tuning algorithms.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Fine-tuning and evaluation datasets usually differ in distribution, which makes evaluation harder than in other FL areas.

    “Note that there is generally a distribution shift between the fine-tuning and evaluation datasets, making it much more challenging than those in other domains of FL [6,18,61].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [6]

      Not checked in the free run.

    • cites [18]

      Not checked in the free run.

    • cites [61]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The runtime environment of evaluation tasks was containerized into Docker images to ensure consistency of evaluation results.

    “To ensure the consistency of the evaluation results, we containerize the runtime environment of these evaluation tasks to docker images for conveniently assessing the performance of the federated LLM fine-tuning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Prefix-tuning was excluded from experiments because its implementation contained unresolved issues.

    “we exclude prefix-tuning from our experiments, because its implementation in [40] contains unresolved issues when we were preparing this package”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40]

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Because evaluation scores change sharply and non-smoothly with hyperparameters, finding optimal hyperparameters is challenging and requires a fine-grained exhaustive search.

    “This poses a great challenge for finding the optimal hyperparameters, as it requires a fine-grained and exhaustive search over the hyperparameter space.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    LLM-Benchmarks is the first module for fair evaluation of federated LLM fine-tuning.

    “Therefore, we introduce LLM-Benchmarks, the first convenient and fair module to evaluate federated LLM fine-tuning.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Whether personalized FL and federated hyperparameter optimization methods still work when combined with various LLM federated fine-tuning algorithms is not known.

    “It is unclear whether the existing algorithms for solving advanced FL problems, such as personalized FL (pFL) [8,58] and federated hyperparameter optimization (FedHPO) [62], are still effective with different federated fine-tuning algorithms for LLMs.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [8]

      Not checked in the free run.

    • cites [58]

      Not checked in the free run.

    • cites [62]

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    FedOT protects model providers' intellectual property and clients' data privacy while using distributed data to adapt LLMs to specific domains.

    “Fe-dOT safeguards both the intelligent property of the model providers and the data privacy of the clients, while leveraging the distributed data for adapting LLMs to specific domains.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [64] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Personalized FL and FedHPO can significantly improve model performance in FL.

    “For instance, pFL [58] and FedHPO [62] are two advanced FL tasks that can significantly improve model performance in FL.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [58]

      Not checked in the free run.

    • cites [62]

      Not checked in the free run.

  • The quote only partly states this claim.

    Interpretationreasoning

    Optimizing pFL algorithms by only using global and local adapters achieves cooperation of any PEFT algorithms with implemented pFL algorithms.

    “With such an implementation, by adding a pFL hook-like function, we can achieve the cooperation of any PEFT algorithms with the implemented pFL algorithms, which provides strong extensibility for federated fine-tuning LLMs in personalized settings.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Clients may need to fine-tune federatedly without full model access, for instance when models are closed-source.

    “Because pre-trained LLMs are of great intelligent property value and may not belong to clients, it might be necessary to let clients conduct federated fine-tuning without accessing the full model (e.g., closedsource LLMs).”
    Quote in the paperSubstantial claim
23 of 23

  • The paper relies on this without support.

    Fine-tuning LLMs requires high computational resources and substantial domain data that may not be sharable due to privacy concerns.

    “However, fine-tuning LLMs entails a high demand for computational resources and a substantial amount of domain data that may not be sharable due to privacy concerns.”

  • The paper relies on this without support.

    Communication and computation resources of clients in FL are usually limited.

    “As communication and computation resources of clients are usually limited in FL”

  • The paper relies on this without support.

    Fine-tuning LLMs in FL is very costly.

    “Since fine-tuning LLMs in FL is very costly”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A distinguishing characteristic of MoE models is flexibility in managing large datasets, capable of amplifying model capacity by over a thousand times while only experiencing minor reductions in computational efficiency.

    “A distinguishing characteristic of MoE models is their flexibility in managing large datasets, capable of amplifying model capacity by over a thousand times while only experiencing minor reductions in computational efficiency [289], [292].”

    From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [292]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • GPU memory components sum to 120 GB, not 112 GB

    Minor

    The stated 28 GB for the model plus another 92 GB for the optimizer and gradients totals 120 GB, not the reported 112 GB.

    Reported 112 · recomputed 120

  • Efficiency experiments list a third GPU type

    Minor

    The efficiency section says all its experiments used two hardware types, A100 and V100, but Table 3 also reports computation times on a 3090.

  • Blanket A100 hardware claim conflicts with reported V100 results

    Minor

    Appendix D says all experiments used the same A100-based hardware configuration, whereas Table 3 reports results obtained on a V100.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, et al. 2019. Towards federated learning at scale: System design. In Proc. of machine learning and systems (MLSys'19), Vol. 1. 374--388.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 1877--1901.
  3. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM.
  4. Zhongxiang Dai, Bryan Kian Hsiang Low, and Patrick Jaillet. 2020. Federated Bayesian Optimization via Thompson Sampling. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 9687--9699.
  5. L. Peter Deutsch. 1996 b. GZIP file format specification version 4.3. RFC. Network Working Group.
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of NAACL-HLT (NAACL-HLT'19). 4171--4186.
  7. Dawei Gao, Daoyuan Chen, Zitao Li, Yuexiang Xie, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. 2023. FS-Real: A Real-World Cross-Device Federated Learning Platform. PVLDB, Vol. 16 (2023), 4046--4049.
  8. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. In Proc. of the International Conference on Machine Learning (ICML'21). 2790--2799.
  9. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proc. of the International Conference on Learning Representations (ICLR'22).
  10. Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic Best Arm Identification and Hyperparameter Optimization. In Proc. of the Artificial intelligence and statistics (AISTATS'16). 240--248.
  11. Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient Low-Rank Hypercomplex Adapter Layers. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'21), Vol. 34. 1022--1035.
  12. Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. 2022. FedScale: Benchmarking Model and System Performance of Federated Learning at Scale. In Proc. of the International Conference on Machine Learning (ICML'22), Vol. 162. 11814--11827.
  13. Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021. Ditto: Fair and Robust Federated Learning Through Personalization. In Proc. of the International Conference on Machine Learning (ICML'21), Vol. 139. 6357--6368.
  14. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT Understands, Too. arXiv preprint arXiv:2103.10385 (2021). · arXiv
  15. H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proc. of the Artificial intelligence and statistics (AISTATS'17). 1273--1282.
  16. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018. Mixed Precision Training. In Proc. of the International Conference on Learning Representations (ICLR'18).
  17. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'19), Vol. 32. 8024--8035.
  18. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterFusion: Non-Destructive Task Composition for Transfer Learning. arXiv preprint arXiv:2005.00247 (2020). · arXiv
  19. Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. 2024. Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes. In International Conference on Machine Learning.
  20. Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyang Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. ZeRO-Offload: Democratizing Billion-Scale Model Training. In Proc. of the USENIX Annual Technical Conference (USENIX ATC'21). 551--564.
  21. Canh T Dinh, Nguyen Tran, and Josh Nguyen. 2020. Personalized Federated Learning with Moreau Envelopes. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 21394--21405.
  22. Zhen Wang, Weirui Kuang, Ce Zhang, Bolin Ding, and Yaliang Li. 2023. FedHPO-Bench: A Benchmark Suite for Federated Hyperparameter Optimization. In Proc. of the International Conference on Machine Learning (ICML'23). 35908--35948.
  23. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2023. GLM-130B: An Open Bilingual Pre-trained Model. In Proc. of the International Conference on Learning Representations (ICLR'23).
  24. Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. 2023. Single-shot Hyper-parameter Optimization for Federated Learning. In Proc. of the International Conference on Learning Representations (ICLR'23).

Cited by

Sorted by citations

Related

Authors
Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J.
Affiliations
  • Alibaba Group (China) (CN)
  • Alibaba Group (United States) (US)
ORCID
Venue
Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Publisher
ACM
Published
2024-08-24
Pages
5260-5271
DOI
10.1145/3637528.3671573
OpenAlex
W4401863415
Cited by
142
References
17
Citations by year
2026: 59 · 2025: 63 · 2024: 16 · 2023: 4
FWCI
40.72
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
federated learning, large language models, LLM fine-tuning, privacy-preserving machine learning, data privacy, communication efficiency, computational resource efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review