The DOI matches its Crossref record; 56 of the 60 references it cites were found in a registry or an index.
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
Open accessAbstract
The rapid evolution and widespread adoption of generative large language models (LLMs) have made them a pivotal workload in various applications. Today, LLM inference clusters receive a large number of queries with strict Service Level Objectives (SLOs). To achieve the desired performance, these models execute on power-hungry GPUs causing the inference clusters to consume large amount of energy and, consequently, result in excessive carbon emissions. Fortunately, we find that there is a great opportunity to exploit the heterogeneity in inference compute properties and fluctuations in inference workloads, to significantly improve energy-efficiency. However, such a diverse and dynamic environment creates a large search-space where different system configurations (e.g., number of instances, model parallelism, and GPU frequency) translate into different energy-performance trade-offs. To address these challenges, we propose DynamoLLM, the first energy-management framework for LLM inference environments. DynamoLLM automatically and dynamically reconfigures the inference cluster to optimize for energy and cost of LLM serving under the service's performance SLOs. We show that at a service-level, DynamoLLM conserves 53% energy and 38% operational carbon emissions, and reduces 61% cost to the customer, while meeting the latency SLOs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“For the Falcon-180B model, which is comparable in size to GPT-3-175B, the server-level energy consumption reaches approximately 0.016kWh per medium-sized request.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“its applicability is currently limited to one of the models we evaluate-specifically Llama-3-70B-and to shorter output tasks.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a conversation task, which typically includes a CPU-intensive prompt phase that processes user input (also known as prompt) and a memory-intensive token phase that produces outputs.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“when considering service-level objectives (SLOs) for LLM response times in enterprise-grade Nvidia DGX H100 systems for conversation tasks, the server-level inference energy consumption for a much smaller model (for example, Llama-3-70B) is already approximately 0.010kWh per medium-sized request when using a state-of-the-art LLM inference solution and accounting for non-GPU server overhead.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we consider a medium-sized request, each with approximately 800 words of input and 150–300 words of output.30”
From Making AI Less 'Thirsty' (2025) · cites this paper as [30]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, the Llama-2-70B inference for a medium-length request on H100 GPUs consumes 9.4 Wh energy without batching [33], while the inference energy consumption is still over 4.0 Wh when batching is applied under various system loads using 'state-of-the-practice' optimizations (the last column in Table II) [45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“whereas [45] shows the server-level energy consumption is about 10 Wh (after multiplying the value in the last column of Table III by 1.6 to account for the non-GPU server energy).”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Its measurement also considers 'state-of-the-practice' optimization techniques commonly used in real systems, including batching.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[45] uses reasonable 'state-ofthe-practice' optimizations including batching.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The measurement in [45] only includes the GPU energy consumption for a small set of open LLMs.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Energy estimates assuming a fully utilized system without accounting for SLOs may not reflect the industry practice, since a fully-utilized system can lead to significant SLO violations, which are not tolerable in real-world LLM deployment, especially for commercial LLM applications such as real-time conversations that have strict SLO targets to deliver good quality of experiences [33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While [45] provides more realistic estimates for shorter tasks by incorporating batching and prompt processing”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Predicting LLM generation length is crucial for optimizing computational resources.”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“Subsequent research has explored two main directions of prediction models: encoder-only models for classification (DynamoLLM [42], S3 [25], TerriInfer [20], SSJF [37], and µ 3 [36]) and decoder-only models for generative prediction like Perception-only (PO) [57].”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“We deliberately make no percentage energy-savings claim versus [49] absent a measured power trace; quantifying end-to-end energy on hardware is future work.”
From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“[45] uses H100 GPUs (which may be more energy-consuming than A100 GPUs for LLM inference as shown by [33]) and considers both prompt processing and token generation energy consumption.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Using [45] where available and [38] otherwise, our approach balances methodological rigor with practical applicability, allowing consistent estimation across diverse tasks and model architectures.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Importantly, it considers the prompt processing phase and representative SLO targets, which are both crucial for real-world LLM deployment.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“Nonetheless, [38] only considers the token generation phase, while neglecting the prompt processing phase (i.e., processing user prompts to generate the first output token) which is also energy-intensive [32, 33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing routing frameworks typically model this as a locally optimal decision-making problem, selecting the presumed best-fit LLM for each query individually, which overlooks global budget constraints, resulting in ineffective resource allocation.”
From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [42]
Experimental — accuracy not yet measuredIn the citing paper
“we note that the server-level inference energy consumption estimated by [38] (as of November 20, 2024) is generally lower than that measured by [45] for the same model size, assuming the LLM inference system is optimized using state-of-the-practice techniques in [45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
In the citing paper
“since the energy measurement results in [45] do not include generating such long outputs using Llama-3-70B.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
In the citing paper
“On the other hand, [45] measures the actual GPU energy consumption for LLMs on enterprise-grade Nvidia DGX H100 servers.”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
In the citing paper
“In practice, however, the actual LLM inference energy consumption depends on a variety of factors, including the hardware, service level objectives (SLOs), and system optimization [33, 45].”
From A Water Efficiency Dataset for African Data Centers (2025) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 5×
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2024) · 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca59077.2024.00019 · Cited by 298 · cited 15×
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialSelf-citationChen, S., Delimitrou, C., & Martínez, J. F. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304005 · Cited by 269
Sinan: ML-based and QoS-aware resource management for cloud microservices
Zhang, Y., Hua, W., Zhou, Z., Suh, G. E., & Delimitrou, C. (2021) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3445814.3446693 · Cited by 230
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., & Gadepally, V. (2023) · 2023 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec58863.2023.10363447 · Cited by 225
Patel, T., & Tiwari, D. (2020) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00025 · Cited by 155
Towards energy proportionality for large-scale latency-critical workloads
Lo, D., Cheng, L., Govindaraju, R., Barroso, L. A., & Kozyrakis, C. (2014) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2014.6853237 · Cited by 150
EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference
Tambe, T., Hooper, C. R. C., Pentecost, L., Jia, T., Yang, E.-Y., Donato, M., Sanh, V., Whatmough, P. N., Rush, A. M., Brooks, D. M., et al. (2021) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3466752.3480095 · Cited by 145
Kasture, H., Bartolini, D. B., Beckmann, N., & Sánchez, D. (2015) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/2830772.2830797 · Cited by 144
Adrenaline: Pinpointing and reining in tail queries with quick voltage boosting
Hsu, C.-H., Zhang, Y., Laurenzano, M. A., Meisner, D., Wenisch, T. F., Mars, J., Tang, L., & Dreslinski, R. (2015) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2015.7056039 · Cited by 117
LLM in a flash: Efficient Large Language Model Inference with Limited Memory
Alizadeh, K., Mirzadeh, S. I., Belenko, D., Khatamifard, S., Cho, M., Del Mundo, C. C., Rastegari, M., & Farajtabar, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.678 · Cited by 113
SpotServe: Serving Generative Large Language Models on Preemptible Instances
Miao, X., Shi, C., Duan, J., Xi, X., Lin, D., Cui, B., & Jia, Z. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640411 · Cited by 105
INFaaS: Automated Model-less Inference Serving
Romero, F., Li, Q., Yadwadkar, N. J., & Kozyrakis, C. (2021) · USENIX Annual Technical Conference · Cited by 102
Characterizing Power Management Opportunities for LLMs in the Cloud
Patel, P., Choukse, E., Zhang, C., Goiri, Í., Warrier, B., Mahalingam, N., & Bianchini, R. (2024) · Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 · doi:10.1145/3620666.3651329 · Cited by 100
Self-citationAntMan: Dynamic Scaling on GPU Clusters for Deep Learning.
Xiao, W., Ren, S., Li, Y., Zhang, Y., Hou, P., Li, Z., Feng, Y., Lin, W., & Jia, Y. (2020) · Operating Systems Design and Implementation · Cited by 100
AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning
Kim, Y. G., & Wu, C.-J. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00090 · Cited by 88
The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study
Tang, Z., Wang, Y., Wang, Q., & Chu, X. (2019) · Proceedings of the Tenth ACM International Conference on Future Energy Systems · doi:10.1145/3307772.3328315 · Cited by 88
Coordinated Batching and DVFS for DNN Inference on GPU Accelerators
Nabavinejad, S. M., Reda, S., & Ebrahimi, M. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3144614 · Cited by 84
Twig: Multi-Agent Task Management for Colocated Latency-Critical Cloud Services
Nishtala, R., Petrucci, V., Carpenter, P., & Själander, M. (2020) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00023 · Cited by 76
AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning ServingPreprint
Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.11665 · Cited by 72
Power capping of CPU-GPU heterogeneous systems through coordinating DVFS and task mapping
Komoda, T., Hayashi, S., Nakada, T., Miwa, S., & Nakamura, H. (2013) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd.2013.6657064 · Cited by 69
Delimitrou, C., & Kozyrakis, C. (2018) · Communications of the ACM · doi:10.1145/3232559 · Cited by 68
ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference
Oh, H., Kim, K. J., Kim, J., Kim, S., Lee, J., Chang, D.-S., & Seo, J. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640383 · Cited by 60
ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching
Zhao, Y., Wu, D., & Wang, J. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00077 · Cited by 58
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
Zheng, Z., Ren, X., Xue, F., Luo, Y., Jiang, X., & You, Y. (2023) · neural information processing systems · doi:10.52202/075280-2859 · Cited by 57
Exploiting heterogeneity for tail latency and energy efficiency
Haque, M. E., He, Y., Elnikety, S., Nguyen, T. D., Bianchini, R., & McKinley, K. S. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3123956 · Cited by 55
Dynamic GPU Energy Optimization for Machine Learning Training Workloads
Wang, F., Zhang, W., Lai, S., Hao, M., & Wang, Z. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3137867 · Cited by 50
Flex: High-Availability Datacenters With Zero Reserved Power
Zhang, C., Kumbhare, A. G., Manousakis, I., Zhang, D., Misra, P. A., Assis, R., Woolcock, K., Mahalingam, N., Warrier, B., Gauthier, D., et al. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00033 · Cited by 45
Self-citationLLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language ModelsPreprint
Faiz, A., Kaneda, S., Wang, R., Osi, R., Sharma, P., Chen, F., & Jiang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.14393 · Cited by 45
Zeus: Understanding and Optimizing GPU Energy Consumption of DNN TrainingPreprint
You, J., Chung, J.-W., & Chowdhury, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.06102 · Cited by 43
SLO-Aware GPU DVFS for Energy-Efficient LLM Inference Serving
Kakolyris, A. K., Masouros, D., Xydis, S., & Soudris, D. (2024) · IEEE Computer Architecture Letters · doi:10.1109/lca.2024.3406038 · Cited by 42
Hsu, C.-H., Deng, Q., Mars, J., & Tang, L. (2018) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3173162.3173190 · Cited by 42
Paella: Low-latency Model Serving with Software-defined GPU Scheduling
Ng, K. K. W., Demoulin, H. M., & Liu, V. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613163 · Cited by 38
EcoFaaS: Rethinking the Design of Serverless Environments for Energy Efficiency
Stojkovic, J., Iliakopoulou, N., Xu, T., Franke, H., & Torrellas, J. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00042 · Cited by 34
Self-citationReTail: Opting for Learning Simplicity to Enable QoS-Aware Power Management in the Cloud
Chen, S., Jin, A., Delimitrou, C., & Martínez, J. F. (2022) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00020 · Cited by 34
Halpern, M., Boroujerdian, B., Mummert, T., Duesterwald, E., & Reddi, V. J. (2019) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass.2019.00012 · Cited by 34
Prediction-Based Power Oversubscription in Cloud PlatformsPreprint
Kumbhare, A. G., Azimi, R., Manousakis, I., Bonde, A., Frujeri, F. V., Mahalingam, N., Misra, P. A., Javadi, S. A., Schroeder, B., da Fontoura, M. F. M. C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15388 · Cited by 34
ALERT: Accurate Learning for Energy and TimelinessPreprint
Wan, C., Santriaji, M. H., Rogers, E. A., Hoffmann, H., Maire, M., & Lu, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00119 · Cited by 29
Gemini: Learning to Manage CPU Power for Latency-Critical Search Engines
Zhou, L., Bhuyan, L. N., & Ramakrishnan, K. K. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00059 · Cited by 28
Thunderbolt: Throughput-Optimized, Quality-of-Service-Aware Power Capping at Scale
Li, S., Wang, X., Zhang, X., Kontorinis, V., Kodakara, S. V., Lo, D., & Ranganathan, P. (2020) · Operating Systems Design and Implementation · Cited by 24
CuttleSys: Data-Driven Resource Management for Interactive Services on Reconfigurable Multicores
Kulkarni, N., Gonzalez-Pumariega, G., Khurana, A., Shoemaker, C. A., Delimitrou, C., & Albonesi, D. H. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00060 · Cited by 22
Indicator-Directed Dynamic Power Management for Iterative Workloads on GPU-Accelerated Systems
Zou, P., Lp, A., BARKER, K. J., & Ge, R. (2020) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid49817.2020.00-37 · Cited by 20
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., & Ramjee, R. (2023) · arXiv · doi:10.48550/arxiv.2308.16369 · Cited by 20
Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM InferencePreprint
Stojkovic, J., Choukse, E., Zhang, C., Goiri, Í., & Torrellas, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.20306 · Cited by 19
Self-citationZhang, Y., Wang, Q., Lin, Z., Xu, P., & Wang, B. (2024) · European Conference on Computer Systems · doi:10.1145/3627703.3629584 · Cited by 18
Yu, J., Kim, J.-S., & Seo, E. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10070943 · Cited by 17
Nabavinejad, S. M., Reda, S., & Ebrahimi, M. (2021) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3394885.3431535 · Cited by 14
Mirhosseini, A., & Wenisch, T. F. (2021) · International Conference on Supercomputing · doi:10.1145/3447818.3463529 · Cited by 12
SmartOClock: Workload- and Risk-Aware Overclocking in the Cloud
Stojkovic, J., Misra, P. A., Goiri, Í., Whitlock, S., Choukse, E., Das, M., Bansal, C., Lee, J., Sun, Z., Qiu, H., et al. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00040 · Cited by 12
Self-citationHybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V., Lakshmanan, L. V. S., & Awadallah, A. H. (2024) · arXiv · doi:10.48550/arxiv.2404.14618 · Cited by 12
WattWiser: Power & Resource-Efficient Scheduling for Multi-Model Multi-GPU Inference Servers
Jahanshahi, A., Rezvani, M., & Wong, D. (2023) · International Green and Sustainable Computing Conference · doi:10.1145/3634769.3634807 · Cited by 8
Efficient Interactive LLM Serving with Proxy Model-based Sequence Length PredictionPreprint
Qiu, H., Mao, W., Patke, A., Cui, S., Jha, S., Wang, C., Franke, H., Kalbarczyk, Z., Başar, T., & Iyer, R. K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.08509 · Cited by 6
Stanford Question Answering Dataset (SQuAD 2.0)
Rajpurkar, P. (2024) · TIB Data Manager · doi:10.57702/29ycgxdq · Cited by 2
Flagged first, then by citations
4 of the 60 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Jin, S^3: Increasing GPU utilization during generative inference for higher throughput
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Scao, 2022, BLOOM: A 176b-parameter open-access multilingual language model, arXiv preprint arXiv:2211.05100 · arXiv
- Touvron, 2023, Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 · arXiv
- Yu, Orca: A Distributed Serving System for Transformer-Based Generative Models
- Zhou, PetS: A unified framework for parameter-efficient transformers serving
Cited by
Sorted by citations
Li, P., Yang, J., Islam, M. A., & Ren, S. (2025) · Communications of the ACM · doi:10.1145/3724499 · Cited by 176
Industrial applications of large language models
Raza, M., Jahangir, Z., Riaz, M. B., Saeed, M. J., & Sattar, M. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-98483-1 · Cited by 234
TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
Stojkovic, J., Zhang, C., Goiri, Í., Choukse, E., Qiu, H., Fonseca, R., Torrellas, J., & Bianchini, R. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716025 · Cited by 44
Self-citationProviding load flexibility by reshaping power profiles of large language model workloads
Wang, Y., Guo, Q., & Chen, M. (2025) · Advances in Applied Energy · doi:10.1016/j.adapen.2025.100232 · Cited by 30
Camlet, A., Kusiak, A., Ossowska, A., & Świetlik, D. (2025) · Diagnostics · doi:10.3390/diagnostics15151851 · Cited by 16
Metadata
- Authors
- Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., & Choukse, E.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Microsoft (United States) (US)
- Venue
- 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA)
- Publisher
- IEEE
- Published
- 2025-03-01
- Pages
- 1348-1362
- ISSN
- 1530-0897 (ISSN-L)
- OpenAlex
- W4409248734
- Cited by
- 112
- References
- 60
- Citations by year
- 2026: 75 · 2025: 35
- FWCI
- 93.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Symposium on High-Performance Computer Architecture (HPCA) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- Semiconductor Research Corporation
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- large language model inference, energy efficiency, cluster design, performance optimization, distributed inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
