CoChat
Sign Up
PreprintCited by 1Peer-to-Peer Network TechnologiesCloud Computing and Resource ManagementBlockchain Technology Applications and Security

PlanetServe: A Decentralized, Scalable, and Privacy-Preserving Overlay for Democratizing Large Language Model Serving

Open access
Fang, Fei, Hua, Yifan, Wang, Shengze, Zhou, Ruilin, Liu, Yi, Qian, Chen, Zhang, Xiaoxue·2025·arXiv
DOI 10.48550/arxiv.2504.20101arXiv 2504.20101
Open this paper in CoChat — free accountOpen PDF

Abstract

While significant progress has been made in research and development on open-source and cost-efficient large-language models (LLMs), serving scalability remains a critical challenge, particularly for small organizations and individuals seeking to deploy and test their LLM innovations. Inspired by peer-to-peer networks that leverage decentralized overlay nodes to increase throughput and availability, we propose GenTorrent, an LLM serving overlay that harnesses computing resources from decentralized contributors. We identify four key research problems inherent to enabling such a decentralized infrastructure: 1) overlay network organization; 2) LLM communication privacy; 3) overlay forwarding for resource efficiency; and 4) verification of serving quality. This work presents the first systematic study of these fundamental problems in the context of decentralized LLM serving. Evaluation results from a prototype implemented on a set of decentralized nodes demonstrate that GenTorrent achieves a latency reduction of over 50% compared to the baseline design without overlay forwarding. Furthermore, the security features introduce minimal overhead to serving latency and throughput. We believe this work pioneers a new direction for democratizing and scaling future AI serving capabilities.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.20101

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing frameworks rely on a small central organization to impose constraints on GPU providers.

    “Existing frameworks (Fang et al., 2025) attempt to rely on a small central organization to impose substantial constraints on GPU providers”

    From WWW.Serve: Interconnecting Global LLM Services through Decentralization (2026) · cites this paper as Fang et al., 2025

    Section 3.3 says prior frameworks such as SGLang and Preble assume a centralized scheduler or request router, while PlanetServe has no central coordinator. It does not say that a small central organization imposes substantial constraints on GPU providers. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Existing decentralized frameworks predominantly emphasize the rights and protections of users and the cooperative aspect among GPU providers while overlooking the inherent competitive dynamics.

    “Fundamentally, they predominantly emphasize the rights and protections of users and the cooperative aspect among GPU providers while overlooking the inherent competitive dynamics”

    From WWW.Serve: Interconnecting Global LLM Services through Decentralization (2026) · cites this paper as Fang et al., 2025

    Section 8 (Related Works; page not given) describes Petals as collaborative and identifies challenges other systems do not address, but it does not characterize existing frameworks as predominantly focused on user protections and provider cooperation. Section 2.2 discusses PlanetServe’s own incentives for cooperation. Searches for competition and prior frameworks found no passage addressing the claim that those frameworks overlook competitive dynamics. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

  • Existing decentralized frameworks impose constraints on GPU providers such as excessive platform-level oversight and fixed software/hardware configurations.

    “requiring them to accept excessive platform-level oversight (Fang et al., 2025; Wu et al., 2025) and to execute all assigned requests with fixed software stacks (Mei et al., 2025a; Borzunov et al., 2023; Mei et al., 2025b) on fixed hardware configurations.”

    From WWW.Serve: Interconnecting Global LLM Services through Decentralization (2026) · cites this paper as Fang et al., 2025

    The Fang et al. citation is attached to the claim about excessive platform-level oversight, not the later claims about fixed software or hardware. The Introduction describes PlanetServe as independent of centralized cloud control, while §3.4 describes its own decentralized verification committee. Neither passage says that existing decentralized frameworks require GPU providers to accept excessive platform-level oversight; searches for provider restrictions and platform control found no such passage. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 47

  1. [7][7] Kevin Bauer, Joshua Juen, Nikita Borisov, Dirk Grun- wald, Douglas Sicker, and Damon McCoy. On the opti- mal path length for tor. In Proc. of HotPETs, 2010.
  2. [8][8] Burton H. Bloom. Space/time trade-offs in hash coding with allowable errors. Commun. ACM, 1970.
  3. [10][10] Ethan Buchman, Jae Kwon, and Zarko Milosevic. The latest gossip on bft consensus. arXiv preprint arXiv:1807.04938, 2018. · arXiv
  4. [11][11] Ethan Buchman, Jae Kwon, and Zarko Milosevic. The latest gossip on BFT consensus. arXiv preprint arXiv:1807.04938, 2019. · arXiv
  5. [12][12] Miguel Castro and Barbara Liskov. Practical byzantine fault tolerance. In Proc. of USENIX OSDI, 1999.
  6. [14][14] Yihua Cheng, Kuntai Du, Jiayi Yao, and Junchen Jiang. Do large language models need a content delivery net- work? arXiv preprint arXiv:2409.13761, 2024. · arXiv
  7. [15][15] Brent Chun et al. PlanetLab: An Overlay Testbed for Broad-Coverage Services. ACM SIGCOMM Computer Communication Review, 2003.
  8. [16][16] DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948, 2025. · arXiv
  9. [17][17] Roger Dingledine, Nick Mathewson, and Paul Syverson. Tor: The second-generation onion router. In Proc. of USENIX Security 2004, 2004.
  10. [20][20] Angela Fan, Mike Lewis, and Yann Dauphin. Hi- erarchical neural story generation. arXiv preprint arXiv:1805.04833, 2018. · arXiv
  11. [21][21] Bin Fan, Dave G. Andersen, Michael Kaminsky, and Michael D. Mitzenmacher. Cuckoo filter: Practically better than bloom. In Proc. of CoNEXT, 2014.
  12. [22][22] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. Serverlessllm: low-latency serverless inference for large language models. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Imple- mentation, 2024.
  13. [24][24] Jinsong Han and Yunhao Liu. Rumor Riding: Anonymizing Unstructured Peer-to-Peer Systems. In Proc. of IEEE ICNP, 2006.
  14. [25][25] Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Stein- hardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
  15. [26][26] Hsu-Chun Hsiao et al. Lap: Lightweight anonymity and privacy. In Proc. of IEEE S&P, 2012.
  16. [27][27] Kunal Jain, Anjaly Parayil, Ankur Mallick, Esha Choukse, Xiaoting Qin, Jue Zhang, Íñigo Goiri, Rujia Wang, Chetan Bansal, Victor Rühle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan. Intelligent router for llm workloads: Improving performance through workload-aware scheduling. CoRR, abs/2408.13510, 2024. 13 · arXiv
  17. [28][28] Sachin Katti, Jeff Cohen, and Dina Katabi. Information slicing: Anonymity using unreliable overlays. In Proc. of NSDI 2007, 2007.
  18. [29][29] Hugo Krawczyk. Secret sharing made short. In Proc. of CRYPTO, 1993.
  19. [30][30] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023.
  20. [31][31] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. Infinigen: efficient generative inference of large language models with dynamic kv cache management. In Proceedings of the 18th USENIX Conference on Op- erating Systems Design and Implementation, 2024.
  21. [32][32] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of ICML, 2023.
  22. [33][33] Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language mod- els understand long contexts? arXiv preprint arXiv:2311.04939, 2023. · arXiv
  23. [34][34] Yuhan Liu, Yuyang Huang, Jiayi Yao, Zhuohan Gu, Kun- tai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, et al. Droidspeak: Kv cache shar- ing for cross-llm communication and multi-llm serving. arXiv preprint arXiv:2411.02820, 2024. · arXiv
  24. [35][35] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, 2024.
  25. [38][38] Donald R. Morrison. Patricia—practical algorithm to retrieve information coded in alphanumeric. Journal of the ACM, 1968.
  26. [40][40] Claus Pahl, Antonio Brogi, Jacopo Soldani, and Pooyan Jamshidi. Cloud Container Technologies: A State-of- the-Art Review. IEEE Transactions on Cloud Comput- ing, 2017.
  27. [41][41] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024.
  28. [42][42] Chen Qian, Junjie Shi, Zihao Yu, Ye Yu, and Sheng Zhong. Garlic cast: Lightweight and decentralized anonymous content sharing. In 2016 IEEE 22nd Interna- tional Conference on Parallel and Distributed Systems (ICPADS), pages 216–223, 2016.
  29. [43][43] M. O. Rabin. Efficient dispersal of information for security, load balancing, and fault tolerance. the Journal of ACM, 1989.
  30. [44][44] M. G. Reed, P. F. Syverson, and D. M. Goldschlag. Anonymous connections and onion routing. IEEE Jour- nal on Selected Areas in Communications, 1998.
  31. [45][45] Ronald L Rivest. All-or-nothing encryption and the package transform. In Fast Software Encryption: 4th International Workshop, FSE'97 Haifa, Israel, January 20–22 1997 Proceedings 4, pages 210–218. Springer, 1997.
  32. [46][46] Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–9. IEEE, 2023.
  33. [47][47] A. Shamir. How to share a secret. the Communications of the ACM, 1979.
  34. [48][48] Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. Fairness in serving large language mod- els. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, 2024.
  35. [49][49] Clay Shields and Brian Neil Levine. A protocol for anonymous communication over the internet. In Proc. of CCS, 2000.
  36. [50][50] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Princi- ples, 2024.
  37. [51][51] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dong- ming Li, and Yiying Zhang. Efficient distributed prompt scheduling for llm serving. In Proc. of ICLR, 2025. 14
  38. [52][52] Douglas R. Stinson. Something about all or noth- ing (transforms). Designs, Codes and Cryptography, 22(2):133–138, 2001.
  39. [53][53] Hugo Touvron et al. LLaMA: Open and Effi- cient Foundation Language Models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
  40. [55][55] David Isaac Wolinsky, Ewa Syta, and Bryan Ford. Hang with your buddies to resist intersection attacks. In Proc. of CCS, 2013.
  41. [56][56] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serv- ing long-context large language models with elastic se- quence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, 2024.
  42. [57][57] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023. · arXiv
  43. [59][59] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative mod- els. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, 2022. USENIX Association.
  44. [60][60] Ruisi Zhang, Yifei Zhao, Neusha Javidnia, Mengxin Zheng, and Farinaz Koushanfar. Attestllm: Efficient attestation framework for billion-scale on-device llms. arXiv preprint arXiv:2509.06326, 2025. · arXiv
  45. [61][61] Lianmin Zheng et al. Sglang: Efficient execution of structured language model programs. In Proc. of NeurIPS, 2024.
  46. [62][62] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- serve: disaggregating prefill and decoding for goodput- optimized large language model serving. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, 2024.
  47. [63][63] Li Zhuang, Feng Zhou, Ben Y Zhao, and Antony Row- stron. Cashmere: Resilient anonymous routing. In Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation-Volume 2, pages 301–314, 2005. Artifact Appendix We release an open-source prototype of PlanetServe, together with the demos and evaluation scripts. The source code is publicly available at: https://github.com/fffeifang/ PlanetServe.git. The repository includes a top-level README that provides an overview of the PlanetServe system, repository structure, and evaluation roadmap. Each major component is further documented with a dedicated README describing its pur- pose, usage, and experimental setup of the corresponding component.

Cited by

Sorted by citations

Authors
Fang, F., Hua, Y., Wang, S., Zhou, R., Liu, Y., Qian, C., & Zhang, X.
Affiliations
  • University of California, Santa Cruz (US)
  • University of Nevada, Reno (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.20101
arXiv
2504.20101
OpenAlex
W4416679339
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2420632, 2322919, 2426031, 2426940)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Peer-to-Peer Network Technologies
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
  • Physical Sciences › Computer Science › Information Systems › Blockchain Technology Applications and Security
Keywords
large language model serving, peer-to-peer networks, privacy-preserving communication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review