CoChat
Sign Up
PreprintCited by 1Topic ModelingNatural Language Processing TechniquesArtificial Intelligence Applications

xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning

Open access
Qian, Cheng, Liu, Zuxin, Kokane, Shirley, Prabhakar, Akshara, Qiu, Jielin, Chen, Haolin, Liu, Zhiwei, Ji, Heng +5 more·2025·arXiv
DOI 10.48550/arxiv.2510.08439arXiv 2510.08439
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern LLM deployments confront a widening cost-performance spectrum: premium models deliver strong reasoning but are expensive, while lightweight models are economical yet brittle on complex tasks. Static escalation rules and keyword heuristics under-utilize this spectrum and fail to adapt across task types. We present xRouter, a tool-calling-based routing system in which a learned router can either answer directly or invoke one or more external models. The router is trained end-to-end with reinforcement learning using an explicit, cost-aware reward that encodes cost-performance trade-offs, eliminating the need for hand-engineered routing rules. Our implementation encompasses the full reinforcement learning framework, including reward and cost accounting, as well as the deployment and evaluation pipelines. Across diverse benchmarks, xRouter achieves strong cost-performance trade-offs (e.g., substantial cost reductions at comparable task completion rates), and provides empirical insights into what reliably helps learned routing and what does not, ranging from model trainability to the difficulty of eliciting sophisticated orchestration behaviors in small open models. We hope these findings and our open implementation will serve as a practical substrate for advancing learned, cost-aware LLM orchestration.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.08439

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works

    “[1 ] and [20] analyze scaling laws and their cost implications, while [14] proposes optimal compute allocation strategies that jointly consider training and inference efficiency.”

  • A Unified Approach to Routing and Cascading for LLMs

    Dekoninck, J., Baader, M., & Vechev, M. (2024) · arXiv · doi:10.48550/arxiv.2410.10347 · cited 1×

    “The proliferation of large language models (LLMs) has turned single-model inference into a multi-model selection problem [11, 43, 53, 7, 49, 10, 26].”

Flagged first, then by citations

Research works printed but not linked: 54

  1. Tom Brown et al. "Language models are few-shot learners". In: Advances in neural information processing systems 33 (2020), pp. 1877-1901.
  2. Thomas Carta et al. Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning. 2024. arXiv: 2302 . 02662 [cs.LG]. url: https : //arxiv.org/abs/2302.02662.
  3. Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. 2023. arXiv: 2305 . 05176 [cs.LG]. url: https://arxiv.org/abs/2305.05176. · arXiv
  4. Xiusi Chen et al. "Rm-r1: Reward modeling as reasoning". In: arXiv preprint arXiv:2505.02387 (2025). · arXiv
  5. Zhoujun Cheng et al. "Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective". In: arXiv preprint arXiv:2506.14965 (2025). · arXiv
  6. Paul F Christiano et al. "Deep reinforcement learning from human preferences". In: Advances in neural information processing systems 30 (2017).
  7. Yu-Neng Chuang et al. "Learning to Route LLMs with Confidence Tokens". In: Forty- second International Conference on Machine Learning. 2025. url: https://openreview. net/forum?id=U08mUogGDM.
  8. Karl Cobbe et al. "Training verifiers to solve math word problems". In: arXiv preprint arXiv:2110.14168 (2021). · arXiv
  9. Daniel Crankshaw et al. "Clipper: A Low-Latency Online Prediction Serving System". In: 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI '17). Boston, MA: USENIX Association, Mar. 2017, pp. 613-627. isbn: 978-1-931971-37-9. url: https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/ crankshaw. References
  10. Dujian Ding et al. "Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing". In: The Twelfth International Conference on Learning Representations. 2024. url: https: //openreview.net/forum?id=02f3mUtqnM.
  11. Song Han, Huizi Mao, and William J. Dally. "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding". In: International Conference on Learning Representations (ICLR). 2016. arXiv: 1510.00149. url: https: //arxiv.org/abs/1510.00149. · arXiv
  12. Qi He et al. "Veri-R1: Toward Precise and Faithful Claim Verification via Online Rein- forcement Learning". In: arXiv preprint arXiv:2510.01932 (2025). · arXiv
  13. Jordan Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. arXiv: 2203.15556 [cs.CL]. url: https://arxiv.org/abs/2203.15556. · arXiv
  14. Chengsong Huang et al. "LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition". In: First Conference on Language Modeling. 2024. url: https : //openreview.net/forum?id=TrloAXEJ2B.
  15. Robert A Jacobs et al. "Adaptive mixtures of local experts". In: Neural computation 3.1 (1991), pp. 79-87.
  16. Albert Q. Jiang et al. Mixtral of Experts. 2024. arXiv: 2401.04088 [cs.LG]. url: https: //arxiv.org/abs/2401.04088. · arXiv
  17. Michael I Jordan and Robert A Jacobs. "Hierarchical mixtures of experts and the EM algorithm". In: Neural computation 6.2 (1994), pp. 181-214.
  18. Yiping Kang et al. "Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge". In: Proceedings of the 22nd International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '17). Xi'an, China: ACM, 2017, pp. 615-629. doi: 10.1145/3037697.3037698. · DOI · Paper page
  19. Sayash Kapoor et al. "Ai agents that matter". In: arXiv preprint arXiv:2407.01502 (2024). · arXiv
  20. Hung Le et al. "CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning". In: Advances in Neural Information Processing Systems.
  21. Ed. by Alice H. Oh et al. 2022. url: https://openreview.net/forum?id=WaGvb7OzySA.
  22. Tian Li et al. "Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads". In: Proceedings of the VLDB Endowment 11.5 (2018), pp. 607-620. doi: 10.14778/3231751.3231757. url: https://www.vldb.org/pvldb/vol11/p607-li.pdf. · DOI
  23. Keming Lu et al. "Routing to the expert: Efficient reward-guided ensemble of large language models". In: arXiv preprint arXiv:2311.08692 (2023). · arXiv
  24. Reiichiro Nakano et al. "Webgpt: Browser-assisted question-answering with human feed- back". In: arXiv preprint arXiv:2112.09332 (2021). References · arXiv
  25. Isaac Ong et al. "RouteLLM: Learning to Route LLMs from Preference Data". In: The Thirteenth International Conference on Learning Representations. 2025. url: https : //openreview.net/forum?id=8sSqNntaMr.
  26. Long Ouyang et al. Training language models to follow instructions with human feedback. 2022. arXiv: 2203.02155 [cs.CL]. url: https://arxiv.org/abs/2203.02155. · arXiv
  27. Bhargavi Paranjape et al. ART: Automatic multi-step reasoning and tool-use for large language models. 2023. arXiv: 2303.09014 [cs.CL]. url: https://arxiv.org/abs/2303. 09014. · arXiv
  28. David Patterson et al. "Carbon emissions and large neural network training". In: arXiv preprint arXiv:2104.10350 (2021). · arXiv
  29. Akshara Prabhakar et al. LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks. 2024. arXiv: 2410.13025 [cs.CL]. url: https://arxiv.org/abs/2410.13025. · arXiv
  30. Cheng Qian et al. "Creator: Tool creation for disentangling abstract and concrete reasoning of large language models". In: Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, pp. 6922-6939.
  31. Cheng Qian et al. "Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model". In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024, pp. 831-854.
  32. Cheng Qian et al. "SMART: Self-Aware Agent for Tool Overuse Mitigation". In: arXiv preprint arXiv:2502.11435 (2025). · arXiv
  33. Cheng Qian et al. "Toolrl: Reward is all tool learning needs". In: arXiv preprint arXiv:2504.13958 (2025). · arXiv
  34. Cheng Qian et al. "UserRL: Training Interactive User-Centric Agent via Reinforcement Learning". In: arXiv preprint arXiv:2509.19736 (2025). · arXiv
  35. Yujia Qin et al. "Tool learning with foundation models". In: arXiv preprint arXiv.2304.08354 10 (2023). · arXiv
  36. Marija Šakota, Maxime Peyrard, and Robert West. "Fly-swat or cannon? cost-effective language model choice via meta-modeling". In: Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 2024, pp. 606-615.
  37. Timo Schick et al. "Toolformer: Language models can teach themselves to use tools". In: Advances in Neural Information Processing Systems 36 (2023), pp. 68539-68551.
  38. Noah Shinn et al. "Reflexion: Language agents with verbal reinforcement learning". In: Thirty-seventh Conference on Neural Information Processing Systems. 2023.
  39. Tal Shnitzer et al. Large Language Model Routing with Benchmark Datasets. 2023. arXiv: 2309.15789 [cs.CL]. url: https://arxiv.org/abs/2309.15789. · arXiv
  40. Wei Song et al. "IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory". In: arXiv preprint arXiv:2506.01048 (2025). · arXiv
  41. Nisan Stiennon et al. "Learning to summarize with human feedback". In: Advances in neural information processing systems 33 (2020), pp. 3008-3021.
  42. Dimitris Stripelis et al. "Tensoropera router: A multi-model router for efficient llm infer- ence". In: arXiv preprint arXiv:2408.12320 (2024). References · arXiv
  43. Emma Strubell, Ananya Ganesh, and Andrew McCallum. "Energy and policy considerations for modern deep learning research". In: Proceedings of the AAAI conference on artificial intelligence. Vol. 34. 09. 2020, pp. 13693-13696.
  44. Hongru Wang et al. "Acting Less is Reasoning More! Teaching Model to Act Efficiently". In: arXiv preprint arXiv:2504.14870 (2025). · arXiv
  45. Junlin Wang et al. "Reasoning in Token Economies: Budget-Aware Evaluation of LLM Rea- soning Strategies". In: arXiv preprint arXiv:2406.06461 (2024). url: https://isthatyou. github.io/. · arXiv
  46. Xuezhi Wang et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. 2023. arXiv: 2203.11171 [cs.CL]. url: https://arxiv.org/abs/2203.11171. · arXiv
  47. Yiding Wang et al. "Tabi: An efficient multi-level inference system for large language models". In: Proceedings of the Eighteenth European Conference on Computer Systems. 2023, pp. 233-248.
  48. Herbert Woisetschläger et al. "Dynamically Learned Test-Time Model Routing in Language Model Zoos with Service Level Guarantees". In: arXiv preprint arXiv:2505.19947 (2025). · arXiv
  49. Shunyu Yao et al. "Tree of Thoughts: Deliberate Problem Solving with Large Language Models". In: Thirty-seventh Conference on Neural Information Processing Systems. 2023. url: https://openreview.net/forum?id=5Xc1ecxO1h.
  50. Weiran Yao et al. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. 2024. arXiv: 2308.02151 [cs.CL]. url: https://arxiv.org/abs/2308. 02151. · arXiv
  51. Qiying Yu et al. "Dapo: An open-source llm reinforcement learning system at scale". In: arXiv preprint arXiv:2503.14476 (2025). · arXiv
  52. Haozhen Zhang, Tao Feng, and Jiaxuan You. "Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning". In: arXiv preprint arXiv:2506.09033 (2025). · arXiv
  53. Qingru Zhang et al. "AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning". In: International Conference on Learning Representations (ICLR). 2023. arXiv: 2303.10512. url: https://arxiv.org/abs/2303.10512. · arXiv
  54. Lianmin Zheng et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena". In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2023. url: https://openreview.net/forum?id=uccHPGDlao.

Cited by

Sorted by citations

Authors
Qian, C., Liu, Z., Kokane, S., Prabhakar, A., Qiu, J., Chen, H., Liu, Z., Ji, H., Yao, W., Heinecke, S., Savarese, S., Xiong, C., & Wang, H.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Salesforce (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2510.08439
arXiv
2510.08439
OpenAlex
W4415313833
Cited by
0
References
2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
LLM orchestration, cost-aware routing, reinforcement learning, tool calling, cost-performance trade-off, model selection, cost accounting, reward function design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review