The DOI matches its Crossref record; 56 of the 60 references it cites were found in a registry or an index.
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
Open accessAbstract
The rapid evolution and widespread adoption of generative large language models (LLMs) have made them a pivotal workload in various applications. Today, LLM inference clusters receive a large number of queries with strict Service Level Objectives (SLOs). To achieve the desired performance, these models execute on power-hungry GPUs causing the inference clusters to consume large amount of energy and, consequently, result in excessive carbon emissions. Fortunately, we find that there is a great opportunity to exploit the heterogeneity in inference compute properties and fluctuations in inference workloads, to significantly improve energy-efficiency. However, such a diverse and dynamic environment creates a large search-space where different system configurations (e.g., number of instances, model parallelism, and GPU frequency) translate into different energy-performance trade-offs. To address these challenges, we propose DynamoLLM, the first energy-management framework for LLM inference environments. DynamoLLM automatically and dynamically reconfigures the inference cluster to optimize for energy and cost of LLM serving under the service's performance SLOs. We show that at a service-level, DynamoLLM conserves 53% energy and 38% operational carbon emissions, and reduces 61% cost to the customer, while meeting the latency SLOs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“For the Falcon-180B model, which is comparable in size to GPT-3-175B, the server-level energy consumption reaches approximately 0.016kWh per medium-sized request.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“its applicability is currently limited to one of the models we evaluate-specifically Llama-3-70B-and to shorter output tasks.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a conversation task, which typically includes a CPU-intensive prompt phase that processes user input (also known as prompt) and a memory-intensive token phase that produces outputs.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“when considering service-level objectives (SLOs) for LLM response times in enterprise-grade Nvidia DGX H100 systems for conversation tasks, the server-level inference energy consumption for a much smaller model (for example, Llama-3-70B) is already approximately 0.010kWh per medium-sized request when using a state-of-the-art LLM inference solution and accounting for non-GPU server overhead.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we consider a medium-sized request, each with approximately 800 words of input and 150–300 words of output.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, the Llama-2-70B inference for a medium-length request on H100 GPUs consumes 9.4 Wh energy without batching [33], while the inference energy consumption is still over 4.0 Wh when batching is applied under various system loads using 'state-of-the-practice' optimizations (the last column in Table II) [45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“whereas [45] shows the server-level energy consumption is about 10 Wh (after multiplying the value in the last column of Table III by 1.6 to account for the non-GPU server energy).”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Its measurement also considers 'state-of-the-practice' optimization techniques commonly used in real systems, including batching.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[45] uses reasonable 'state-ofthe-practice' optimizations including batching.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The measurement in [45] only includes the GPU energy consumption for a small set of open LLMs.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Energy estimates assuming a fully utilized system without accounting for SLOs may not reflect the industry practice, since a fully-utilized system can lead to significant SLO violations, which are not tolerable in real-world LLM deployment, especially for commercial LLM applications such as real-time conversations that have strict SLO targets to deliver good quality of experiences [33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While [45] provides more realistic estimates for shorter tasks by incorporating batching and prompt processing”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Predicting LLM generation length is crucial for optimizing computational resources.”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“Subsequent research has explored two main directions of prediction models: encoder-only models for classification (DynamoLLM [42], S3 [25], TerriInfer [20], SSJF [37], and µ 3 [36]) and decoder-only models for generative prediction like Perception-only (PO) [57].”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“We deliberately make no percentage energy-savings claim versus [49] absent a measured power trace; quantifying end-to-end energy on hardware is future work.”
From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“[45] uses H100 GPUs (which may be more energy-consuming than A100 GPUs for LLM inference as shown by [33]) and considers both prompt processing and token generation energy consumption.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Using [45] where available and [38] otherwise, our approach balances methodological rigor with practical applicability, allowing consistent estimation across diverse tasks and model architectures.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Importantly, it considers the prompt processing phase and representative SLO targets, which are both crucial for real-world LLM deployment.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Nonetheless, [38] only considers the token generation phase, while neglecting the prompt processing phase (i.e., processing user prompts to generate the first output token) which is also energy-intensive [32, 33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing routing frameworks typically model this as a locally optimal decision-making problem, selecting the presumed best-fit LLM for each query individually, which overlooks global budget constraints, resulting in ineffective resource allocation.”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
Experimental — accuracy not yet measuredIn the citing paper
“we note that the server-level inference energy consumption estimated by [38] (as of November 20, 2024) is generally lower than that measured by [45] for the same model size, assuming the LLM inference system is optimized using state-of-the-practice techniques in [45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
In the citing paper
“since the energy measurement results in [45] do not include generating such long outputs using Llama-3-70B.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
In the citing paper
“On the other hand, [45] measures the actual GPU energy consumption for LLMs on enterprise-grade Nvidia DGX H100 servers.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
In the citing paper
“In practice, however, the actual LLM inference energy consumption depends on a variety of factors, including the hardware, service level objectives (SLOs), and system optimization [33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 5×
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2024) · 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca59077.2024.00019 · Cited by 298 · cited 15×
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialSelf-citationChen, S., Delimitrou, C., & Martínez, J. F. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304005 · Cited by 269
Sinan: ML-based and QoS-aware resource management for cloud microservices
Zhang, Y., Hua, W., Zhou, Z., Suh, G. E., & Delimitrou, C. (2021) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3445814.3446693 · Cited by 230
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., & Gadepally, V. (2023) · 2023 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec58863.2023.10363447 · Cited by 225
Patel, T., & Tiwari, D. (2020) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00025 · Cited by 155
Towards energy proportionality for large-scale latency-critical workloads
Lo, D., Cheng, L., Govindaraju, R., Barroso, L. A., & Kozyrakis, C. (2014) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2014.6853237 · Cited by 150
Flagged first, then by citations
4 of the 60 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Jin, S^3: Increasing GPU utilization during generative inference for higher throughput
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Scao, 2022, BLOOM: A 176b-parameter open-access multilingual language model, arXiv preprint arXiv:2211.05100 · arXiv
- Touvron, 2023, Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 · arXiv
- Yu, Orca: A Distributed Serving System for Transformer-Based Generative Models
- Zhou, PetS: A unified framework for parameter-efficient transformers serving
Cited by
Sorted by citations
Items 101–113 of 113
Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Scheduling
Da, W., & Kalyvianaki, E. (2026) · ACM International Systems and Storage Conference (SYSTOR) · doi:10.1145/3793230.3837768 · Cited by 1
Dong, D., Wu, C., Chen, Z., Yan, L., Niu, J., Yu, J., Ma, Z., & Zhai, J. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3732699
PreServe: Intelligent Management for LMaaS Systems via Hierarchical Prediction
Jiang, Z., Huang, Y., Yu, G., Huang, J., Gu, J., & Lyu, M. R.-T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773173
Adaptive GPU Sharing for Real-time LLM Serving with Best-effort Workloads
Li, S., Hanawa, T., & Miki, Y. (2026) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3816891.3834895
A Cross-Layer Survey of Low-Power Machine Learning From Algorithm to Hardware
Tan, Z.-X., Karkehabadi, A., Miao, N., Shao, W., Homayoun, H., & Sasan, A. (2026) · IEEE Access · doi:10.1109/access.2026.3734236
Learning to Compress: Energy-Aware Large Language Model Serving via Adaptive KV Caching
Cheng, Y., Shao, Z., Yu, N., Fu, Y., & Dong, Y. (2026) · IEEE Power Engineering Society General Meeting · doi:10.1109/pesgm58988.2026.11693290
Carbon- and Cost-Aware Scheduling of Cloud AI Inference Workloads Across Geo-Distributed Regions
Satla, S., Rao, M. N., Katari, T., Priyanka, R. M., Pranathi, K., & Vidya, R. (2026) · International Conference on Secure Information Systems and Technologies (ICSIST) · doi:10.1109/icsist70215.2026.11706162
E-LQO: A Comprehensive Energy Evaluation Framework for Learned Query Optimization
Liang, Z., Xu, Q., Chen, X., Chen, J., Xia, Y., & Zheng, K. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3837107
Vahidi, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7584825
Survival-Conditioned Machine Learning for LLM Gateway Routing and Admission
Nhidza, W., & Yahya, A. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.23397210
Genetic fragmentation gradient descent: failure-constrained scheduling for GPU-sharing clusters
Choi, S., & Sim, J. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08928-1
Gupta, G. K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7675463
Ma, J., Zhou, Y., Morstyn, T., & Wang, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7674326
Metadata
- Authors
- Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., & Choukse, E.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Microsoft (United States) (US)
- Venue
- 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA)
- Publisher
- IEEE
- Published
- 2025-03-01
- Pages
- 1348-1362
- ISSN
- 1530-0897 (ISSN-L)
- OpenAlex
- W4409248734
- Cited by
- 112
- References
- 60
- Citations by year
- 2026: 75 · 2025: 35
- FWCI
- 93.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Symposium on High-Performance Computer Architecture (HPCA) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- Semiconductor Research Corporation
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- large language model inference, energy efficiency, cluster design, performance optimization, distributed inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
