CoChat

Efficient Serving of LLM Applications with Probabilistic Demand Modeling

Liu, Yifei, Gan, Zuo, Gan, Zhenghao, Wang, Weiye, Chen, Chen, Shan, Yizhou, Chen, Xusheng, Han, Zhenhua, Zhu, Yifei, Sun, Shixuan, Guo, Minyi · 2025 · arXiv

Preprint — published as 10.1145/3803390

Preprint

The DOI matches its DataCite record; the references it cites are not checked yet.

Abstract

Applications based on Large Language Models (LLMs) contains a series of tasks to address real-world problems with boosted capability, which have dynamic demand volumes on diverse backends. Existing serving systems treat the resource demands of LLM applications as a blackbox, compromising end-to-end efficiency due to improper queuing order and backend warm up latency. We find that the resource demands of LLM applications can be modeled in a general and accurate manner with Probabilistic Demand Graph (PDGraph). We then propose Hermes, which leverages PDGraph for efficient serving of LLM applications. Confronting probabilistic demand description, Hermes applies the Gittins policy to determine the scheduling order that can minimize the average application completion time. It also uses the PDGraph model to help prewarm cold backends at proper moments. Experiments with diverse LLM applications confirm that Hermes can effectively improve the application serving efficiency, reducing the average completion time by over 70% and the P95 completion time by over 80%.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.14851

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Metadata

Authors
Liu, Y., Gan, Z., Gan, Z., Wang, W., Chen, C., Shan, Y., Chen, X., Han, Z., Zhu, Y., Sun, S., & Guo, M.
Venue
arXiv
Publisher
arXiv
Published
2025
Type
Preprint

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Efficient Serving of LLM Applications with Probabilistic Demand Modeling. https://app.cochat.ai/papers/10.48550/arxiv.2506.14851
The paper
Liu, Y., Gan, Z., Gan, Z., Wang, W., Chen, C., Shan, Y., Chen, X., Han, Z., Zhu, Y., Sun, S., & Guo, M. (2025). Efficient Serving of LLM Applications with Probabilistic Demand Modeling. arXiv. https://doi.org/10.48550/arxiv.2506.14851

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.