CoChat
Sign Up
PreprintCited by 11Energy Efficiency in ComputingCloud Computing and Resource ManagementGreen IT and Sustainability

The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective

Open access
Kim, Jiin, Shin, Byeongjun, Chung, Jinha, Rhu, Minsoo·2025·arXiv
DOI 10.48550/arxiv.2506.04301arXiv 2506.04301
Open this paper in CoChat — free accountOpen PDF

Abstract

Large-language-model (LLM)-based AI agents have recently showcased impressive versatility by employing dynamic reasoning, an adaptive, multi-step process that coordinates with external tools. This shift from static, single-turn inference to agentic, multi-turn workflows broadens task generalization and behavioral flexibility, but it also introduces serious concerns about system-level cost, efficiency, and sustainability. This paper presents the first comprehensive system-level analysis of AI agents, quantifying their resource usage, latency behavior, energy consumption, and datacenter-wide power consumption demands across diverse agent designs and test-time scaling strategies. We further characterize how AI agent design choices, such as few-shot prompting, reflection depth, and parallel reasoning, impact accuracy-cost tradeoffs. Our findings reveal that while agents improve accuracy with increased compute, they suffer from rapidly diminishing returns, widening latency variance, and unsustainable infrastructure costs. Through detailed evaluation of representative agents, we highlight the profound computational demands introduced by AI agent workflows, uncovering a looming sustainability crisis. These results call for a paradigm shift in agent design toward compute-efficient reasoning, balancing performance with deployability under real-world constraints.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.04301

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The computational cost of dynamic reasoning in LLM-based agents grows non-linearly with the complexity of adaptation mechanisms, imposing practical constraints on achievable generalization within realistic resource limitations.

    “Additionally, the computational cost associated with dynamic reasoning in LLM-based agents grows non-linearly with the complexity of adaptation mechanisms, imposing practical constraints on achievable generalization within realistic resource limitations (Kim et al., 2025).”

    From A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence (2025) · cites this paper as Kim et al., 2025

    The Introduction (p. 1) says dynamic reasoning can raise per-request costs by orders of magnitude, and the Test-Time Scaling analysis (pp. 9–10) reports substantial overhead and diminishing accuracy returns as reasoning paths or reflection steps increase. The paper supports practical resource constraints on deploying agents, but it does not establish a general non-linear relationship between cost and the complexity of adaptation mechanisms or directly measure a limit on achievable generalization. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 64

  1. [1][1] M. Adnan, A. Arunkumar, G. Jain, P. J. Nair, I. Soloveychik, and P. Kamath, 'Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,' in Proceedings of Machine Learning and Systems , 2024. -
  2. [2][2] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, 'Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve,' in Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI) , 2024. -
  3. [4][4] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai, 'GQA: Training generalized multi-query transformer models from multi-head checkpoints,' in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. -
  4. [6][6] P. Belcak, G. Heinrich, S. Diao, Y. Fu, X. Dong, S. Muralidharan, Y. C. Lin, and P. Molchanov, 'Small language models are the future of agentic ai,' arXiv preprint arXiv:2506.02153 , 2025. - · arXiv
  5. [7][7] M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler, 'Graph of thoughts: solving elaborate problems with large language models,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2024. -
  6. [8][8] T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, 'Medusa: Simple llm inference acceleration framework with multiple decoding heads,' in Proceedings of the 41st International Conference on Machine Learning , ser. ICML'24, 2024. -
  7. [9][9] G. I. Chaudhry, E. Choukse, ´ I˜ nigo Goiri, R. Fonseca, A. Belay, and R. Bianchini, 'Towards Resource-Efficient Compound AI Systems,' in arxiv.org , 2025. -
  8. [10][10] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, 'Evaluating Large Language Models Trained on Code,' in arxiv.org , 2021. -
  9. [11][11] R. Coulom, 'Efficient selectivity and backup operators in monte-carlo tree search,' in International conference on computers and games . Springer, 2006. -
  10. [12][12] A. de Vries, 'The growing energy footprint of artificial intelligence,' Joule , 2023. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S2542435123003653 -
  11. [13][13] DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang, 'Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,' 2025. [Online]. Available: https://arxiv.org/abs/2501.12948 - · arXiv
  12. [16][16] D. Ding, A. Mallick, S. Zhang, C. Wang, D. Madrigal, M. D. C. H. Garcia, M. Xia, L. V. Lakshmanan, Q. Wu, and V. R¨ uhle, 'Best-route: Adaptive llm routing with test-time optimal compute,' arXiv preprint arXiv:2506.22716 , 2025. - · arXiv
  13. [17][17] W. Fedus, B. Zoph, and N. Shazeer, 'Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,' Journal of Machine Learning Research , vol. 23, no. 120, pp. 1-39, 2022. -
  14. [19][19] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, 'Gptq: Accurate post-training quantization for generative pre-trained transformers,' arXiv preprint arXiv:2210.17323 , 2022. - · arXiv
  15. [20][20] B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo, 'Cost-Efficient large language model serving for multi-turn conversations with CachedAttention,' in 2024 USENIX Annual Technical Conference (USENIX ATC 24) . Santa Clara, CA: USENIX Association, Jul. 2024, pp. 111-126. [Online]. Available: https://www.usenix.org/conference/atc24/presentation/gao-bin-cost -
  16. [22][22] V. U. Gsteiger, P. H. Long, Y. Sun, P. Javanrood, and M. Shahrad, 'Caribou: Fine-grained geospatial shifting of serverless applications for sustainability,' in Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles , 2024, pp. 403-420. -
  17. [23][23] X. Gu, T. Pang, C. Du, Q. Liu, F. Zhang, C. Du, Y. Wang, and M. Lin, 'When attention sink emerges in language models: An empirical view,' arXiv preprint arXiv:2410.10781 , 2024. - · arXiv
  18. [24][24] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al. , 'Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,' arXiv preprint arXiv:2501.12948 , 2025. - · arXiv
  19. [25][25] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, 'Measuring Mathematical Problem Solving With the MATH Dataset,' in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS) , 2021. -
  20. [26][26] G. Hinton, O. Vinyals, and J. Dean, 'Distilling the knowledge in a neural network,' arXiv preprint arXiv:1503.02531 , 2015. - · arXiv
  21. [29][29] J. Jeong and J. Ahn, 'Accelerating llm serving for multi-turn dialogues with efficient resource management,' in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , 2025. -
  22. [31][31] S. Kim, S. Moon, R. Tabrizi, N. Lee, M. W. Mahoney, K. Keutzer, and A. Gholami, 'An LLM Compiler for Parallel Function Calling,' in Proceedings of the International Conference on Machine Learning (ICML) , 2024. -
  23. [32][32] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, 'Efficient Memory Management for Large Language Model Serving with PagedAttention,' in Proceedings of the ACM Symposium on Operating System Principles (SOSP) , 2023. -
  24. [33][33] W. Lee, J. Lee, J. Seo, and J. Sim, 'InfiniGen: Efficient generative inference of large language models with dynamic KV cache management,' in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , 2024. -
  25. [34][34] Y. Leviathan, M. Kalman, and Y. Matias, 'Fast inference from transformers via speculative decoding,' in Proceedings of the 40th International Conference on Machine Learning , 2023. -
  26. [35][35] B. Li, S. Samsi, V. Gadepally, and D. Tiwari, 'Clover: Toward sustainable ai with carbon-aware machine learning inference service,' in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2023, pp. 1-15. -
  27. [36][36] G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, 'CAMEL: Communicative agents for 'mind' exploration of large language model society,' in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS) , 2023. -
  28. [37][37] W. Liang, L. Yu, L. Luo, S. Iyer, N. Dong, C. Zhou, G. Ghosh, M. Lewis, W.-t. Yih, L. Zettlemoyer et al. , 'Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models,' arXiv preprint arXiv:2411.04996 , 2024. - · arXiv
  29. [38][38] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, 'Awq: Activation-aware weight quantization for on-device llm compression and acceleration,' Proceedings of machine learning and systems , vol. 6, pp. 87-100, 2024. -
  30. [39][39] A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo et al. , 'Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,' arXiv preprint arXiv:2405.04434 , 2024. - · arXiv
  31. [40][40] A. Liu, J. Liu, Z. Pan, Y. He, G. Haffari, and B. Zhuang, 'Minicache: Kv cache compression in depth dimension for large language models,' in Advances in Neural Information Processing Systems . Curran Associates, Inc., 2024. -
  32. [41][41] Y. Liu, H. Li, Y. Cheng, S. Ray, Y. Huang, Q. Zhang, K. Du, J. Yao, S. Lu, G. Ananthanarayanan, M. Maire, H. Hoffmann, A. Holtzman, and J. Jiang, 'Cachegen: Kv cache compression and streaming for fast large language model serving,' in Proceedings of the ACM SIGCOMM 2024 Conference , ser. ACM SIGCOMM '24, 2024. -
  33. [42][42] Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. R´ e, and B. Chen, 'Deja vu: contextual sparsity for efficient llms at inference time,' in Proceedings of the 40th International Conference on Machine Learning , ser. ICML'23, 2023. -
  34. [48][48] N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Cand` es, and T. Hashimoto, 's1: Simple test-time scaling,' 2025. [Online]. Available: https: //arxiv.org/abs/2501.19393 -
  35. [49][49] S. Muralidharan, S. Turuvekere Sreenivas, R. Joshi, M. Chochowski, M. Patwary, M. Shoeybi, B. Catanzaro, J. Kautz, and P. Molchanov, 'Compact language models via pruning and knowledge distillation,' Advances in Neural Information Processing Systems , vol. 37, pp. 41 076-41 102, 2024. -
  36. [59][59] P. Panda, R. Magazine, C. Devaguptapu, S. Takemori, and V. Sharma, 'Adaptive llm routing under budget constraints,' arXiv preprint arXiv:2508.21141 , 2025. - · arXiv
  37. [61][61] P. Patel, E. Choukse, C. Zhang, A. Shah, ´ I. Goiri, S. Maleki, and R. Bianchini, 'Splitwise: Efficient generative llm inference using phase 62. splitting,' in Proceedings of the International Symposium on Computer Architecture (ISCA) , 2024. -
  38. [63][63] O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis, 'Measuring and narrowing the compositionality gap in language models,' in The 2023 Conference on Empirical Methods in Natural Language Processing , 2023. -
  39. [64][64] Y. Qu, M. Y. Yang, A. Setlur, L. Tunstall, E. E. Beeching, R. Salakhutdinov, and A. Kumar, 'Optimizing test-time compute via meta reinforcement fine-tuning,' arXiv preprint arXiv:2503.07572 , 2025. - · arXiv
  40. [65][65] A. Radovanovi´ c, R. Koningstein, I. Schneider, B. Chen, A. Duarte, B. Roy, D. Xiao, M. Haridasan, P. Hung, N. Care et al. , 'Carbon-aware computing for datacenters,' IEEE Transactions on Power Systems , vol. 38, no. 2, pp. 1270-1280, 2022. -
  41. [67][67] K. Santhanam, D. Raghavan, M. S. Rahman, T. Venkatesh, N. Kunjal, P. Thaker, P. Levis, and M. Zaharia, 'ALTO: An Efficient Network Orchestrator for Compound AI Systems,' in Proceedings of the 4th Workshop on Machine Learning and Systems , 2024. -
  42. [72][72] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, 'Reflexion: Language Agents with Verbal Reinforcement Learning,' in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS) , 2023. -
  43. [75][75] C. Snell, J. Lee, K. Xu, and A. Kumar, 'Scaling llm test-time compute optimally can be more effective than scaling model parameters,' 2024. [Online]. Available: https://arxiv.org/abs/2408.03314 - · arXiv
  44. [76][76] --, 'Scaling llm test-time compute optimally can be more effective than scaling model parameters,' arXiv preprint arXiv:2408.03314 , 2024. - · arXiv
  45. [77][77] S. T. Sreenivas, S. Muralidharan, R. Joshi, M. Chochowski, A. S. Mahabaleshwarkar, G. Shen, J. Zeng, Z. Chen, Y. Suhara, S. Diao et al. , 'Llm pruning and distillation in practice: The minitron approach,' arXiv preprint arXiv:2408.11796 , 2024. - · arXiv
  46. [78][78] X. Tan, Y. Jiang, Y. Yang, and H. Xu, 'Towards end-to-end optimization of llm-based applications with ayo,' in Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , 2025. -
  47. [79][79] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom, 'Llama 2: Open foundation and fine-tuned chat models,' in arxiv.org , 2023. -
  48. [83][83] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, 'Self-Consistency Improves Chain of Thought Reasoning in Language Models,' in Proceedings of the International Conference on Learning Representations (ICLR) , 2023. -
  49. [84][84] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, 'Chain-of-thought prompting elicits reasoning in large language models,' in Proceedings of the 36th International Conference on Neural Information Processing Systems , ser. NIPS '22. Red Hook, NY, USA: Curran Associates Inc., 2022. -
  50. [87][87] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang, 'Autogen: Enabling next-gen LLM applications via multiagent conversations,' in First Conference on Language Modeling (CoLM) , 2024. -
  51. [89][89] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, 'Smoothquant: Accurate and efficient post-training quantization for large language models,' in International conference on machine learning . PMLR, 2023, pp. 38 087-38 099. -
  52. [90][90] Z. Xie, H. Kang, Y. Sheng, T. Krishna, K. Fatahalian, and C. Kozyrakis, 'AI Metropolis: Scaling Large Language Model-Based Multi-Agent Simulation with Out-of-Order Execution,' in arxiv.org , 2024. -
  53. [91][91] W. Yang, S. Ma, Y. Lin, and F. Wei, 'Towards thinking-optimal scaling of test-time compute for llm reasoning,' arXiv preprint arXiv:2502.18080 , 2025. - · arXiv
  54. [92][92] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, 'HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering,' in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018. -
  55. [93][93] J. Yao, H. Li, Y. Liu, S. Ray, Y. Cheng, Q. Zhang, K. Du, S. Lu, and J. Jiang, 'Cacheblend: Fast large language model serving for rag with cached knowledge fusion,' in Proceedings of the Twentieth European Conference on Computer Systems , ser. EuroSys '25, 2025. -
  56. [94][94] S. Yao, H. Chen, J. Yang, and K. Narasimhan, 'WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents,' in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS) , 2022. -
  57. [95][95] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, 'Tree of Thoughts: Deliberate Problem Solving with Large Language Models,' in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS) , 2023. -
  58. [96][96] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, 'React: Synergizing Reasoning and Acting in Language Models,' in Proceedings of the International Conference on Learning Representations (ICLR) , 2023. -
  59. [98][98] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, 'Orca: A Distributed Serving System for Transformer-Based Generative Models,' in Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI) , 2022. -
  60. [99][99] J. Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y. Wei, L. Wang, Z. Xiao et al. , 'Native sparse attention: Hardware-aligned and natively trainable sparse attention,' arXiv preprint arXiv:2502.11089 , 2025. - · arXiv
  61. [100][100] L. Zheng, L. Yin, Z. Xie, C. L. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez et al. , 'Sglang: Efficient execution of structured language model programs,' Advances in neural information processing systems , vol. 37, pp. 62 557-62 583, 2024. -
  62. [101][101] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, 'DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,' in Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI) , 2024. -
  63. [102][102] A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y.-X. Wang, 'Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models,' in Proceedings of the International Conference on Machine Learning (ICML) , 2024. -
  64. [104][104] D. Zhou, N. Sch¨ arli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, and E. Chi, 'Least-to-Most Prompting Enables Complex Reasoning in Large Language Models,' in Proceedings of the International Conference on Learning Representations (ICLR) , 2023.

Cited by

Sorted by citations

Authors
Kim, J., Shin, B., Chung, J., & Rhu, M.
Affiliations
Korea Advanced Institute of Science and Technology (KR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
1-16
ISSN
1530-0897 (ISSN-L)
DOI
10.48550/arxiv.2506.04301
arXiv
2506.04301
OpenAlex
W7133534962
Cited by
11
References
0
Citations by year
2026: 10 · 2025: 1
FWCI
85.25
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Locations
  • IEEE International Symposium on High-Performance Computer Architecture (HPCA) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • KAIST Institutional Repository (KAIST) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Ministry of Science and ICT, South Korea (RS-202400438851)
  • Samsung (SRFC-IT2402-03)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
Keywords
AI agents, dynamic reasoning, large language models, test-time scaling, energy consumption, latency, few-shot prompting, parallel reasoning, accuracy-cost trade-off, AI sustainability, computational efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review