The DOI matches its Crossref record; 56 of the 60 references it cites were found in a registry or an index.
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
Open accessAbstract
The rapid evolution and widespread adoption of generative large language models (LLMs) have made them a pivotal workload in various applications. Today, LLM inference clusters receive a large number of queries with strict Service Level Objectives (SLOs). To achieve the desired performance, these models execute on power-hungry GPUs causing the inference clusters to consume large amount of energy and, consequently, result in excessive carbon emissions. Fortunately, we find that there is a great opportunity to exploit the heterogeneity in inference compute properties and fluctuations in inference workloads, to significantly improve energy-efficiency. However, such a diverse and dynamic environment creates a large search-space where different system configurations (e.g., number of instances, model parallelism, and GPU frequency) translate into different energy-performance trade-offs. To address these challenges, we propose DynamoLLM, the first energy-management framework for LLM inference environments. DynamoLLM automatically and dynamically reconfigures the inference cluster to optimize for energy and cost of LLM serving under the service's performance SLOs. We show that at a service-level, DynamoLLM conserves 53% energy and 38% operational carbon emissions, and reduces 61% cost to the customer, while meeting the latency SLOs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“For the Falcon-180B model, which is comparable in size to GPT-3-175B, the server-level energy consumption reaches approximately 0.016kWh per medium-sized request.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“its applicability is currently limited to one of the models we evaluate-specifically Llama-3-70B-and to shorter output tasks.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a conversation task, which typically includes a CPU-intensive prompt phase that processes user input (also known as prompt) and a memory-intensive token phase that produces outputs.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“when considering service-level objectives (SLOs) for LLM response times in enterprise-grade Nvidia DGX H100 systems for conversation tasks, the server-level inference energy consumption for a much smaller model (for example, Llama-3-70B) is already approximately 0.010kWh per medium-sized request when using a state-of-the-art LLM inference solution and accounting for non-GPU server overhead.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we consider a medium-sized request, each with approximately 800 words of input and 150–300 words of output.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, the Llama-2-70B inference for a medium-length request on H100 GPUs consumes 9.4 Wh energy without batching [33], while the inference energy consumption is still over 4.0 Wh when batching is applied under various system loads using 'state-of-the-practice' optimizations (the last column in Table II) [45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“whereas [45] shows the server-level energy consumption is about 10 Wh (after multiplying the value in the last column of Table III by 1.6 to account for the non-GPU server energy).”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Its measurement also considers 'state-of-the-practice' optimization techniques commonly used in real systems, including batching.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[45] uses reasonable 'state-ofthe-practice' optimizations including batching.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The measurement in [45] only includes the GPU energy consumption for a small set of open LLMs.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Energy estimates assuming a fully utilized system without accounting for SLOs may not reflect the industry practice, since a fully-utilized system can lead to significant SLO violations, which are not tolerable in real-world LLM deployment, especially for commercial LLM applications such as real-time conversations that have strict SLO targets to deliver good quality of experiences [33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While [45] provides more realistic estimates for shorter tasks by incorporating batching and prompt processing”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Predicting LLM generation length is crucial for optimizing computational resources.”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“Subsequent research has explored two main directions of prediction models: encoder-only models for classification (DynamoLLM [42], S3 [25], TerriInfer [20], SSJF [37], and µ 3 [36]) and decoder-only models for generative prediction like Perception-only (PO) [57].”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“We deliberately make no percentage energy-savings claim versus [49] absent a measured power trace; quantifying end-to-end energy on hardware is future work.”
From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“[45] uses H100 GPUs (which may be more energy-consuming than A100 GPUs for LLM inference as shown by [33]) and considers both prompt processing and token generation energy consumption.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Using [45] where available and [38] otherwise, our approach balances methodological rigor with practical applicability, allowing consistent estimation across diverse tasks and model architectures.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“we note that the server-level inference energy consumption estimated by [38] (as of November 20, 2024) is generally lower than that measured by [45] for the same model size, assuming the LLM inference system is optimized using state-of-the-practice techniques in [45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Importantly, it considers the prompt processing phase and representative SLO targets, which are both crucial for real-world LLM deployment.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Nonetheless, [38] only considers the token generation phase, while neglecting the prompt processing phase (i.e., processing user prompts to generate the first output token) which is also energy-intensive [32, 33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing routing frameworks typically model this as a locally optimal decision-making problem, selecting the presumed best-fit LLM for each query individually, which overlooks global budget constraints, resulting in ineffective resource allocation.”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
Experimental — accuracy not yet measuredIn the citing paper
“since the energy measurement results in [45] do not include generating such long outputs using Llama-3-70B.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
In the citing paper
“On the other hand, [45] measures the actual GPU energy consumption for LLMs on enterprise-grade Nvidia DGX H100 servers.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
In the citing paper
“In practice, however, the actual LLM inference energy consumption depends on a variety of factors, including the hardware, service level objectives (SLOs), and system optimization [33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 5×
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2024) · 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca59077.2024.00019 · Cited by 298 · cited 15×
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialSelf-citationChen, S., Delimitrou, C., & Martínez, J. F. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304005 · Cited by 269
Sinan: ML-based and QoS-aware resource management for cloud microservices
Zhang, Y., Hua, W., Zhou, Z., Suh, G. E., & Delimitrou, C. (2021) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3445814.3446693 · Cited by 230
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., & Gadepally, V. (2023) · 2023 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec58863.2023.10363447 · Cited by 225
Patel, T., & Tiwari, D. (2020) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00025 · Cited by 155
Towards energy proportionality for large-scale latency-critical workloads
Lo, D., Cheng, L., Govindaraju, R., Barroso, L. A., & Kozyrakis, C. (2014) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2014.6853237 · Cited by 150
Research works printed but not linked: 6
- Jin, S^3: Increasing GPU utilization during generative inference for higher throughput
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Scao, 2022, BLOOM: A 176b-parameter open-access multilingual language model, arXiv preprint arXiv:2211.05100 · arXiv
- Touvron, 2023, Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 · arXiv
- Yu, Orca: A Distributed Serving System for Transformer-Based Generative Models
- Zhou, PetS: A unified framework for parameter-efficient transformers serving
Cited by
Sorted by citations
Li, P., Yang, J., Islam, M. A., & Ren, S. (2025) · Communications of the ACM · doi:10.1145/3724499 · Cited by 176
Industrial applications of large language models
Raza, M., Jahangir, Z., Riaz, M. B., Saeed, M. J., & Sattar, M. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-98483-1 · Cited by 234
TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
Stojkovic, J., Zhang, C., Goiri, Í., Choukse, E., Qiu, H., Fonseca, R., Torrellas, J., & Bianchini, R. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716025 · Cited by 44
Self-citationProviding load flexibility by reshaping power profiles of large language model workloads
Wang, Y., Guo, Q., & Chen, M. (2025) · Advances in Applied Energy · doi:10.1016/j.adapen.2025.100232 · Cited by 30
Camlet, A., Kusiak, A., Ossowska, A., & Świetlik, D. (2025) · Diagnostics · doi:10.3390/diagnostics15151851 · Cited by 16
LLMs on a Budget: System-Level Approaches to Power-Efficient and Scalable Fine-Tuning
Gogineni, K., Suvizi, A., & Venkataramani, G. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3580498 · Cited by 14
Carbon-Aware Quality Adaptation for Energy-Intensive Services
Wiesner, P., Grinwald, D., Weiß, P., Wilhelm, P., Khalili, R., & Kao, O. (2025) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3679240.3734614 · Cited by 12
A Water Efficiency Dataset for African Data Centers
Shumba, N., Tshekiso, O., Li, P., Fanti, G., & Ren, S. (2025) · Proceedings of the ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies · doi:10.1145/3715335.3735483 · Cited by 10
LithOS: An Operating System for Efficient Machine Learning on GPUs
Coppock, P. H., Zhang, B., Solomon, E. H., Kypriotis, V., Yang, L., Sharma, B., Schatzberg, D., Mowry, T. C., & Skarlatos, D. (2025) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3731569.3764818 · Cited by 24
Impact of decoding strategies on GPU energy usage in large language model text generation
Nik, A., Riegler, M. A., & Halvorsen, P. (2025) · Scientific Reports · doi:10.1038/s41598-025-31896-0 · Cited by 14
Optimizing LLM Inference Clusters for Enhanced Performance and Energy EfficiencyPreprint
Hisaharo, S., Nishimura, Y., & Takahashi, A. (2024) · doi:10.36227/techrxiv.172348951.12175366/v1 · Cited by 9
ReGate: Enabling Power Gating in Neural Processing Units
Xue, Y., & Huang, J. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756038 · Cited by 15
Distributed VLMs: Efficient Vision-Language Processing through Cloud-Edge Collaboration
Li, Y., Gumaste, D., Türkcan, M. K., Ghaderi, J., Zussman, G., & Kostić, Z. (2025) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops65533.2025.00078 · Cited by 8
Liu, M., Gu, J., & Zhao, T. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00045 · Cited by 5
Performance Characterization of Expert Router for Scalable LLM Inference
Pichlmeier, J., Ross, P., & Luckow, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826121 · Cited by 11
Disaggregated Speculative Decoding for Carbon-Efficient LLM Serving
Shi, T., Wu, Y., Liu, S., & Ding, Y. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3630094 · Cited by 4
MoEntwine: Unleashing the Potential of Wafer-Scale Chips for Large-Scale Expert Parallel Inference
Tang, X., Hou, J., Jiang, D., Wei, T., Liu, J., Deng, J., Wang, H., Yang, Q., Shang, H., Li, C., et al. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408594 · Cited by 4
Slower is Greener: Acceptance of Eco-feedback Interventions on Carbon Heavy Internet Services
Kim, H., Young, S., Chen, X., Gupta, U., & Hester, J. D. (2025) · ACM Journal on Computing and Sustainable Societies · doi:10.1145/3723038 · Cited by 4
Wafer-Scale Systems: A Carbon Perspective
Golden, A., Elgamal, M., Mahmoud, A., Hills, G., Wu, C.-J., Wei, G.-Y., & Brooks, D. M. (2025) · ACM SIGEnergy Energy Informatics Review · doi:10.1145/3757892.3757909 · Cited by 3
QoServe: Breaking the Silos of LLM Inference Serving
Goel, K., Mohan, J., Kwatra, N., Anupindi, R. S., & Ramjee, R. (2026) · Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3779212.3790206 · Cited by 3
SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference
Xia, T., Mao, Z., Kerney, J., Jackson, E. J., Li, Z., Xing, J., Shenker, S. J., & Stoica, I. G. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3769353 · Cited by 3
Provisioning high precision edge inference with runtime model reconfiguration
Sun, H., Qian, Z., Zhu, A., Zhang, S., Lu, S., Chu, X., & Luo, G. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111585 · Cited by 3
eInfer: Unlocking Fine-Grained Tracing for Distributed LLM Inference with eBPF
Chu, K., Su, J., Zhang, Y., Zhao, C., Yang, Y., Zheng, Y., Lin, S., Zhao, S., & Zhang, W. (2025) · ACM Workshop on eBPF and Kernel Extensions (eBPF) · doi:10.1145/3748355.3748372 · Cited by 3
Adaptive scheduling of online inference pipelines at the edge: A post-hoc request-oriented approach
Sun, H., Qian, Z., Zhu, A., Li, M., Zhang, S., Lu, S., Meng, L., Wang, D., & Li, X. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103597 · Cited by 5
Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments
Iliakopoulou, N., Stojkovic, J., Alverti, C., Xu, T., Franke, H., & Torrellas, J. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756083 · Cited by 15
Self-citationWAGES: Workload-Aware GPU Sharing System for Energy-Efficient Serverless LLM Serving
Wang, T., Rattihalli, G., Dhakal, A., Tang, X., & Milojicic, D. S. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767396 · Cited by 3
Engine-Agnostic Model Hot-Swapping for Cost-Effective LLM Inference
Stoyanov, R., Spišaková, V., Reber, A., Armour, W., Copik, M., & Bruno, R. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767354 · Cited by 2
Guo, T., Zhang, X.-W., Du, J.-S., Chen, Z.-G., Xiao, N., & Lu, Y.-T. (2025) · International Conference on Software Composition · doi:10.1145/3712285.3759823 · Cited by 15
HealthServe: A Public Health-Aware Inference Framework
Zhou, S., & Roy, R. B. (2026) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3805638 · Cited by 2
The Utilization Fallacy and the Real Drivers of Carbon-Efficient Inference Serving
Sinha, P., Liakopoulos, D., Li, R., & Yadwadkar, N. J. (2025) · ACM SIGEnergy Energy Informatics Review · doi:10.1145/3757892.3757903 · Cited by 1
Design and Operation of Elastic GPU-Pooling on Campus
Guo, K., Chen, J., Wang, Y., Anton, S., Dmitry, T., Sheng, J., Wei, J., Lin, J., Qi, Z., & Guan, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99854-6_27 · Cited by 1
Cloud abstractions for AI workloads
Canini, M., Benson, T., Bianchini, R., Goiri, Í., Kostić, D., Pietzuch, P., & Peter, S. (2025) · ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) · doi:10.1145/3725783.3764395 · Cited by 1
Self-citationLaser: Unlocking Layer-Level Scheduling for Efficient Multi-SLO LLM Serving
Liao, J., Dong, Q., Liang, Y., Zhou, Z., & Chen, X. (2026) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3774934.3786413 · Cited by 5
Towards High-Goodput LLM Serving with Prefill-decode Multiplexing
Chen, Y., Cui, W., Zhao, H., Xu, Z. S., Fan, X., Chen, X., Zhou, Y., Sun, S., He, B., & Chen, Q. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790236 · Cited by 23
CEDAR: Carbon Efficient Dynamic Allocation and Routing for Agentic LLM Inference
MORE, A., Anwar, T., & Yadav, P. (2026) · doi:10.1145/3802973.3804457 · Cited by 1
EnergAIzer: Fast and Accurate GPU Power Estimation Framework for AI Workloads
Lee, K., Song, Z., Lee, E. K., Zhang, X., Eilam, T., & Chandrakasan, A. P. (2026) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass69572.2026.00049 · Cited by 1
Zhang, B., Hou, J., & Zhang, J. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3703757 · Cited by 1
Trajano, A. F. R., Costa, C., Nobre, F. V. J., & Gomes, R. L. (2026) · doi:10.5753/sbrc.2026.19707 · Cited by 1
How Much Energy Is Wasted in LLM operations? Evidence from Kernel-Level DVFS
Spaan, J., Chen, K.-H., & Vărbănescu, A. L. (2026) · ACM International Conference on Computing Frontiers (CF) · doi:10.1145/3801487.3801822 · Cited by 1
BROS: Efficient LLM Serving on Hybrid Real-time and Best-effort Requests
Wan, B., Zhao, J., Jiang, C., Guo, C., & Wu, C. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571463 · Cited by 1
Guo, K., Chen, J., Chen, C., Wang, Y., Du, P., Qi, Z., & Guan, H. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3713365 · Cited by 1
Baek, S., Jeong, B., Byun, H., & Park, S. (2026) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid68966.2026.00014 · Cited by 1
Artificial intelligence data centers could reach one percent of global electricity demand by 2030
Chen, D., Zhou, Z., Cai, Y., Qin, J., Katchova, A. L., & Chen, L. (2026) · Communications Sustainability · doi:10.1038/s44458-026-00152-5 · Cited by 1
PRESAGE: Uncertainty-Aware Predictive Elastic Scheduling for Cloud Analytics and GenAI Inference
Wang, Y., Zou, X., Zhou, L., Wei, K., & Rhee, M. (2026) · IEEE Access · doi:10.1109/access.2026.3734498 · Cited by 3
ECCOS: Efficient Capability and Cost Coordinated Scheduling for Multi-LLM ServingPreprint
Mei, K., Xu, W., Lin, S., & Zhang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5159339 · Cited by 1
BlendServe: Optimizing Offline Inference with Resource-Aware Batching
Zhao, Y., Yang, S., Zhu, K., Zheng, L., Kasikci, B., Qiao, Y., Zhou, Y., Xing, J., & Stoica, I. (2026) · Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3779212.3790133
Aruna, K. B., Kaliraj, V., Sudha, I., & Sureka, V. (2026) · EAI Endorsed Transactions on Internet of Things · doi:10.4108/eetiot.13349
Dynamic Prediction and Control in Data Centers Via Knowledge Graph Physics-Informed Neural NetworksPreprint
Kachun, L. O. I., Song, C., Chao, H. J., Chen, L., & Guo, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5257808
Draco: Dynamic Resource Allocation for Concurrent ML Applications
Radig, T., & Karl, H. (2025) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid64434.2025.00040
DRAFAS: Dynamic Resource Allocation for AI-Native Services
Tu, N. V., Nam, S., Tan, L., & Hong, J. W. (2025) · Journal of Network and Systems Management · doi:10.1007/s10922-025-09962-9
Enabling Cost-Efficient LLM Inference on Mid-Tier GPUs With NMP DIMMs
Kim, H., La, A., & Izraelevitz, J. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3646622
Multiplexed Heterogeneous LLM Serving via Stage-Aligned Parallelism
Luo, T., Ng, K. K. W., Khor, Z. P., Sankhe, S., Loo, B. T., & Liu, V. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772207
AUE: A Normalized Energy Efficiency Metric for AI Servers Under LLM Workloads
Zhang, Y., Zhang, D., Liu, X., Wang, Q., Wang, B., Zhang, S., & Tian, Y. (2025) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads67057.2025.11323149
HARA: Optimization of Recommendation Systems Based on Hybrid Asynchronous Architecture
Zhu, J., Zhao, N., & Lan, Y. (2025) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic68075.2025.11324878
Proactive Concurrency Tuning for Inference Serving in Serverless Platform
Shwe, T., Muhammad-Bello, B. L., & Aritsugi, M. (2025) · doi:10.1109/cbdcom68404.2025.00011
PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models
Cho, E., Bang, J., Hwang, R., & Rhu, M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408493 · Cited by 2
Li, Z., Li, M., Huang, Z., Yang, J., Zhao, W., Lin, Y., Wang, C., Tian, H., & Xiong, X. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420504
Cache Your Prompt When It's Green — Carbon-Aware Caching for Large Language Model Serving
Tian, Y., Sun, D., Ding, Y., & Liu, S. (2026) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3788087 · Cited by 1
A Lightweight Framework for Energy-Aware Prediction and Scheduling in Heterogeneous HPC Clusters
Shan, H., Bai, X., & Cheng, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-8402-4_10
TOKCOINFER: Token-Level Multi-Model Collaboration for Energy-Efficient LLM Inference
Zhang, Y., Zhang, D., Wang, B., & Zhang, S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464816
OASIS: Optimal Allocation Strategy for Inference Services in Cloud Environments
Mittal, V., Baydoun, M., Mishra, A., Prakash, P., Rattihalli, G., Dhakal, A., Frachtenberg, E., Hajj, I. E., Faloutsos, M., & Milojicic, D. S. (2026) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3777911.3800633
Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications
Li, J., Zhu, Y., Chen, B., Lee, E. K., & Nahrstedt, K. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807662 · Cited by 3
Sampling Where It Matters: Predicting LLM Serving Performance
Aydar, E., Pinto, C., Venugopal, S., & Chatzopoulos, D. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807633
Dealing with the Elephant in the KV Cache: Video Frame Sampling for Multimodal LLM Inference
Papaioannou, K., & Doudali, T. D. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807613
Duc, M. V., Duong, N. D., Huy, N. D., Tuyen, N. D., Hung, N. T., & Thanh, N. H. (2026) · International Conference on Information Networking · doi:10.1109/icoin68469.2026.11480581
CALM: Cost-efficient, model Agnostic, Low-latency Modular LLM Deployment framework
Bansal, A., Gill, D. K., Joshi, R., Pham, H., Zamparo, L., & Liu, W. (2026) · International Conference on Advances in Artificial Intelligence and Machine Learning (AAIML) · doi:10.1109/aaiml67890.2026.11498121
Hu, Q., & Li, J. (2026) · Electronics · doi:10.3390/electronics15102139
Power Ranger: A Comprehensive Framework for GPU Power Characterization
Seigler, A., Jiang, Z., & John, L. K. (2026) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass69572.2026.00073
HealthServe: A Public Health-Aware Inference Framework
Zhou, S., & Roy, R. B. (2026) · ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS) · doi:10.1145/3801489.3806868
Measuring Tradeoffs in Multi-Agent Models: a Case Study in Shipping Cost Calculation
Ryser, A., Rosso, P., Audiffren, J., Ruffieux, S., Goetschmann, P., & Cudré-Mauroux, P. (2026) · Swiss Conference on Data Science · doi:10.1109/sds70563.2026.00040
Energy-Efficient Small Language Model Inference for Mobile Agents via DVFS
Chen, J., Han, L., Cao, J., & Li, Z. (2026) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3812836.3814753
Yu, J., Taneja, A., Lin, J., & Zhang, M. (2026) · ISC High Performance (ISC) · doi:10.23919/isc.2026.11520495
LHRS-LTW: A Load-Aware Hybrid-Cloud Resource Scheduling Framework for Large-Scale Training Workloads
Wei, A., Yang, J., Pang, P., Sun, S., Ruan, X., Chen, Y., & Guo, M. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3704297
To Defer or To Shift? The Role of AI Data Center Flexibility on Grid Interconnection
Chen, Y., & Zheng, X. (2026) · ACM Sustainability Week · doi:10.1145/3765611.3815593
Power Compression: Peak Power Aware Workflow Optimization for Agentic AI
Han, Y., & Ren, S. (2026) · ACM Sustainability Week · doi:10.1145/3765611.3815385
Abera, N. B., Musı́lek, P., & Chen, Y. (2026) · ACM Sustainability Week · doi:10.1145/3765611.3815065
LSTCB: Long-Short-Timescale Cooperative Batching for Energy-Efficient Edge Inference
Zheng, Z., Zhang, S., Lu, N., Wang, Z., & Luo, H. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571396
3D-Learning: Diffusion-Augmented Distributionally Robust Decision-Focused Learning
Wen, J., Fan, L., & Yang, J. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571436
Beyond Throughput: Performance and Energy Insights of LLM Inference Across AI Accelerators
Brunetta, G., Sastry, V. K., Wu, X., Taylor, V., Papka, M. E., & Lan, Z. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00083
PEACE: Preemptive and Efficient Cluster Scheduling for LLM Inference with Mixed Prompts
Zhang, Z., & Shen, H. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00105
FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
Gao, S., Yin, J., Wang, F., & Dong, W. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800525 · Cited by 6
Phase-aware Peak Power Reduction for Minimizing the Capital Expense of LLM Inference
Wu, S., Ma, Y., & Wang, X. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3815788
Overcoming Congestion in Edge MoE: A Bi-Level Game-Theoretic Orchestration Framework
Chen, Y. (2026) · International Conference on Signal Processing, Communication and Control Systems (SPCCS) · doi:10.1109/spccs69331.2026.11588633
Deploying LLMs on CPU-only Environments with llama.cpp Library Set: MedLocalGPT Project Case
Malakhov, K. S. (2026) · International Workshop of IT-professionals on Artificial Intelligence (ProfIT AI) · doi:10.55056/ceur-ws.org/vol-4164/paper11.pdf
MAP: GPU Resource Planning for Multi-Agent Systems With Stochastic and Verbose Outputs
Zhang, Q., Li, M., Li, Q., Hua, X., Chen, F., Zeng, D., & Li, P. L. (2026) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2026.3712315
Fine-Grained Energy Accounting in Production LLM Serving
Yuan, X., Liao, H., Zhang, K., Guo, D., & Tang, G. (2026) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3820441.3820447
HealthServe: A Public Health-Aware Inference Framework
Zhou, S., & Roy, R. B. (2026) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/3820171.3806868
PowerWeave: Unlocking Energy-Efficient ML on GPUs with OS-Level Spatial Power Management
Kypriotis, V., Dubberstein, E., Coppock, P. H., Solomon, E. H., Zamir, R., Srimani, T., & Skarlatos, D. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00168
CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM
Liu, Q., Chen, L., Wang, H., Yang, Y., Du, D., Mao, Z., Jing, N., Xia, Y., & Chen, H. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00056 · Cited by 3
ConServe: Contiguity-Preserving Memory Management for Multi-Turn LLM Serving
Li, B. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00075
Power Sloshing in Compound Servers for Large-Scale AI Inference Workloads
Cho, A., Stojkovic, J., Piga, L., Dhanotia, A., Sajal, S. M., Zuo, G., Malladi, K. T., Akers, D., Subramanian, K., Kanaujia, S., et al. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00167
Self-citationFleet-Level Optimization for Heterogeneous Device Provisioning in ML Model Deployments
Jiang, H. H., & Joe‐Wong, C. (2026) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw72724.2026.00038
PowerGrad: Hierarchical Power Management for Power-Limited ML Inference Clusters
Nam, H., Pothukuchi, R. P., Buyuktosunoglu, A., Amarnath, A., Bose, P. K., & Torrellas, J. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00166
Self-citationRearchitecting the Datacenter Lifecycle for AI
Stojkovic, J., Zhang, C., Goiri, Í., & Bianchini, R. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00029 · Cited by 1
Self-citationJiang, F., Zhang, J., & Li, X. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-0288-1_26
Performance Characterization and Optimization of LLM Inference on Tenstorrent AI Accelerators
Lim, J., Shin, D., Kim, U., Choi, J., Kim, K. M., Shin, S., Park, S., Koo, G., & Suh, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35251-4_32
Energy-Efficient LLM Inference with SLA-Constrained Adaptive GPU Power Capping
Trajano, A. F. R., Costa, C., Nobre, F. V. J., & Gomes, R. L. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00029
TierServe: Revenue-Maximizing LLM Inference Scheduling Across Multiple Subscription Tiers
Liu, C., Liu, J., Hou, X., & Guo, M. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-4805-6_10
GreenAlign: Aligning Mixed LLM Workloads with Intermittent Renewable Supply
Liu, C., Liu, J., Hou, X., Li, C., & Guo, M. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00021
Sharma, B., Tiwari, A., Kaur, R., Park, K., & Pinsky, E. (2026) · Green · doi:10.3390/green1020008
Metadata
- Authors
- Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., & Choukse, E.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Microsoft (United States) (US)
- Venue
- 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA)
- Publisher
- IEEE
- Published
- 2025-03-01
- Pages
- 1348-1362
- ISSN
- 1530-0897 (ISSN-L)
- OpenAlex
- W4409248734
- Cited by
- 112
- References
- 60
- Citations by year
- 2026: 75 · 2025: 35
- FWCI
- 93.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Symposium on High-Performance Computer Architecture (HPCA) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- Semiconductor Research Corporation
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- large language model inference, energy efficiency, cluster design, performance optimization, distributed inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
