CoChat
Sign Up
Cited by 113Energy Efficiency in ComputingCloud Computing and Resource ManagementParallel Computing and Optimization Techniques

DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

Open access
Stojkovic, Jovan, Zhang, Chaojie, Goiri, Íñigo, Torrellas, Josep, Choukse, Esha·2025·2025 IEEE International Symposium on High Performance Computer Architecture (HPCA)
DOI 10.1109/hpca61900.2025.00102
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid evolution and widespread adoption of generative large language models (LLMs) have made them a pivotal workload in various applications. Today, LLM inference clusters receive a large number of queries with strict Service Level Objectives (SLOs). To achieve the desired performance, these models execute on power-hungry GPUs causing the inference clusters to consume large amount of energy and, consequently, result in excessive carbon emissions. Fortunately, we find that there is a great opportunity to exploit the heterogeneity in inference compute properties and fluctuations in inference workloads, to significantly improve energy-efficiency. However, such a diverse and dynamic environment creates a large search-space where different system configurations (e.g., number of instances, model parallelism, and GPU frequency) translate into different energy-performance trade-offs. To address these challenges, we propose DynamoLLM, the first energy-management framework for LLM inference environments. DynamoLLM automatically and dynamically reconfigures the inference cluster to optimize for energy and cost of LLM serving under the service's performance SLOs. We show that at a service-level, DynamoLLM conserves 53% energy and 38% operational carbon emissions, and reduces 61% cost to the customer, while meeting the latency SLOs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/hpca61900.2025.00102

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For the Falcon-180B model, the server-level energy consumption reaches approximately 0.016 kWh per medium-sized request.

    “For the Falcon-180B model, which is comparable in size to GPT-3-175B, the server-level energy consumption reaches approximately 0.016kWh per medium-sized request.30”

    From Making AI Less 'Thirsty' (2025) · cites this paper as [30]

    Table III in Section III (p. 3) reports 7.94–10.34 Wh for Falcon-180B medium-sized requests at medium system load, depending on GPU frequency. That is approximately 0.008–0.010 kWh, not 0.016 kWh per request. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • A conversation task typically includes a CPU-intensive prompt phase that processes user input and a memory-intensive token phase that produces outputs.

    “a conversation task, which typically includes a CPU-intensive prompt phase that processes user input (also known as prompt) and a memory-intensive token phase that produces outputs.30”

    From Making AI Less 'Thirsty' (2025) · cites this paper as [30]

    Section II, Background (p. 2) describes a compute-intensive prefill phase that processes input tokens and a memory-intensive decode phase that generates output tokens. It does not characterize the first phase as CPU-intensive; that hardware-specific part of the claim is unsupported. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • When considering service-level objectives for LLM response times in enterprise-grade Nvidia DGX H100 systems for conversation tasks, the server-level inference energy consumption for Llama-3-70B is already approximately 0.010 kWh per medium-sized request when using a state-of-the-art LLM inference solution and accounting for non-GPU server overhead.

    “when considering service-level objectives (SLOs) for LLM response times in enterprise-grade Nvidia DGX H100 systems for conversation tasks, the server-level inference energy consumption for a much smaller model (for example, Llama-3-70B) is already approximately 0.010kWh per medium-sized request when using a state-of-the-art LLM inference solution and accounting for non-GPU server overhead.30”

    From Making AI Less 'Thirsty' (2025) · cites this paper as [30]

    Section III reports measurements on an NVIDIA DGX H100 using vLLM and gives SLO-constrained energy figures for medium-sized Llama3-70B requests. Table III reports 4.28–6.45 Wh per request at medium load, not approximately 10 Wh (0.010 kWh). The cited passages do not establish that adding non-GPU server overhead raises those figures to 10 Wh, or give that figure specifically for Conversation requests. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • A medium-sized request has approximately 800 words of input and 150-300 words of output.

    “we consider a medium-sized request, each with approximately 800 words of input and 150–300 words of output.30”

    From Making AI Less 'Thirsty' (2025) · cites this paper as [30]

    Section III, Table IV (p. 3) classifies requests by input and output length, so it supports the idea of a medium-sized request. But its thresholds are in tokens—fewer than 1,024 input tokens and fewer than 350 output tokens—not approximately 800 input words and 150–300 output words. The paper does not establish that word-count conversion. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Predicting LLM generation length is crucial for optimizing computational resources.

    “Predicting LLM generation length is crucial for optimizing computational resources.”

    From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

4 of the 60 references couldn’t be matched to a known paper.

Research works printed but not linked: 10

  1. [6][6] S. Chen, C. Delimitrou, and J. F. Martı́nez, "PARTIES: QoS-Aware Resource Partitioning for Multiple Interactive Services," in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '19), 2019.
  2. [19][19] Y. Jin, C.-F. Wu, D. Brooks, and G.-Y. Wei, " S3: Increasing GPU utilization during generative inference for higher throughput," in Pro- ceedings of the Advances in Neural Information Processing Systems (NeurIPS '23), 2023.
  3. [21][21] H. Kasture, D. B. Bartolini, N. Beckmann, and D. Sanchez, "Rubik: Fast analytical power management for latency-critical systems," in Proceedings of the 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO '15), 2015.
  4. [37][37] A. Mirhosseini and T. Wenisch, "µSteal: A Theory-Backed Framework for Preemptive Work and Resource Stealing in Mixed-Criticality Mi- croservices," in Proceedings of the ACM International Conference on Supercomputing (ICS '21), 2021.
  5. [40][40] S. M. Nabavinejad, S. Reda, and M. Ebrahimi, "BatchSizer: Power- Performance Trade-off for DNN Inference," in Proceedings of the 26th Asia and South Pacific Design Automation Conference (ASP-DAC '21), 2021.
  6. [56][56] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  7. [59][59] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé et al., "BLOOM: A 176b-parameter open-access multilingual language model," arXiv preprint arXiv:2211.05100, 2022. · arXiv
  8. [67][67] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
  9. [73][73] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, "Orca: A Distributed Serving System for Transformer-Based Generative Models," in Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI '22), 2022.
  10. [81][81] Z. Zhou, X. Wei, J. Zhang, and G. Sun, "PetS: A unified framework for Parameter-Efficient transformers serving," in Proceedings of the USENIX Annual Technical Conference (USENIX ATC '22), 2022.

Cited by

Sorted by citations

Related

Authors
Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., & Choukse, E.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Microsoft (United States) (US)
ORCID
Venue
2025 IEEE International Symposium on High Performance Computer Architecture (HPCA)
Publisher
IEEE
Published
2025-03-01
Pages
1348-1362
ISSN
1530-0897 (ISSN-L)
DOI
10.1109/hpca61900.2025.00102
OpenAlex
W4409248734
Cited by
112
References
60
Citations by year
2026: 75 · 2025: 35 · 2024: 2
FWCI
93.99
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Symposium on High-Performance Computer Architecture (HPCA) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
Semiconductor Research Corporation
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Keywords
large language model inference, energy efficiency, cluster design, performance optimization, distributed inference

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review