CoChat
Sign Up
Cited by 7Reinforcement Learning in RoboticsAI-based Problem Solving and PlanningNatural Language Processing Techniques

The DOI matches its Crossref record.

Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models

Open access
Zhai, Yuanzhao, Yang, Tingkai, Xu, Kele, Feng, Dawei, Yang, Cheng, Ding, Bo, Wang, Huaimin·2025·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v39i25.34924
Open this paper in CoChat — free accountOpen PDF

Abstract

Agents significantly enhance the capabilities of standalone Large Language Models (LLMs) by perceiving environments, making decisions, and executing actions. However, LLM agents still face challenges in tasks that require multiple decision-making steps. Estimating the value of actions in specific tasks is difficult when intermediate actions are neither appropriately rewarded nor penalized. In this paper, we propose leveraging a task-relevant Q-value model to guide action selection. Specifically, we first collect decision-making trajectories annotated with step-level Q values via Monte Carlo Tree Search (MCTS) and construct preference data. We then use another LLM to fit these preferences through step-level Direct Policy Optimization (DPO), which serves as the Q-value model. During inference, at each decision-making step, LLM agents select the action with the highest Q value before interacting with the environment. We apply our method to various open-source and API-based LLM agents, demonstrating that Q-value models significantly improve their performance. Notably, the performance of the agent built with Phi-3-mini-4k-instruct improved by 103% on WebShop and 75% on HotPotQA when enhanced with Q-value models, even surpassing GPT-4o-mini. Additionally, Q-value models offer several advantages, such as generalization to different LLM agents and seamless integration with existing prompting strategies.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v39i25.34924

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 38

  1. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  2. Browne, C. B.; Powley, E.; Whitehouse, D.; Lucas, S. M.; Cowling, P. I.; Rohlfshagen, P.; Tavener, S.; Perez, D.; Samothrakis, S.; and Colton, S. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computa- tional Intelligence and AI in games, 4(1): 1-43.
  3. Chen, B.; Shu, C.; Shareghi, E.; Collier, N.; Narasimhan, K.; and Yao, S. 2023. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915. · arXiv
  4. Chen, G.; Liao, M.; Li, C.; and Fan, K. 2024a. AlphaMath Almost Zero: process Supervision without process. arXiv preprint arXiv:2405.03553. · arXiv
  5. Chen, Z.; Liu, K.; Wang, Q.; Zhang, W.; Liu, J.; Lin, D.; Chen, K.; and Zhao, F. 2024b. Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models. arXiv preprint arXiv:2403.12881. · arXiv
  6. Christianos, F.; Papoudakis, G.; Zimmer, M.; Coste, T.; Wu, Z.; Chen, J.; Khandelwal, K.; Doran, J.; Feng, X.; Liu, J.; et al. 2023. Pangu-agent: A fine-tunable generalist agent with structured reasoning. arXiv preprint arXiv:2312.14878. · arXiv
  7. Feng, X.; Wan, Z.; Wen, M.; Wen, Y.; Zhang, W.; and Wang, J. 2023. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179. · arXiv
  8. Hao, S.; Gu, Y.; Ma, H.; Hong, J.; Wang, Z.; Wang, D.; and Hu, Z. 2023. Reasoning with Language Model is Planning with World Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 8154- 8173.
  9. Kocsis, L.; and Szepesvári, C. 2006. Bandit based monte- carlo planning. In European conference on machine learning, 282-293. Springer.
  10. Koh, J. Y.; McAleer, S.; Fried, D.; and Salakhutdinov, R. 2024. Tree Search for Language Model Agents. arXiv preprint arXiv:2407.01476. · arXiv
  11. Konda, V.; and Tsitsiklis, J. 1999. Actor-critic algorithms. Advances in neural information processing systems, 12.
  12. Lambert, N.; Pyatkin, V.; Morrison, J.; Miranda, L.; Lin, B. Y.; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y.; et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787. · arXiv
  13. Luo, L.; Liu, Y.; Liu, R.; Phatale, S.; Lara, H.; Li, Y.; Shu, L.; Zhu, Y.; Meng, L.; Sun, J.; et al. 2024. Improve Mathemati- cal Reasoning in Language Models by Automated Process Supervision. arXiv preprint arXiv:2406.06592. · arXiv
  14. Ma, C.; Zhang, J.; Zhu, Z.; Yang, C.; Yang, Y.; Jin, Y.; Lan, Z.; Kong, L.; and He, J. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. arXiv preprint arXiv:2401.13178. · arXiv
  15. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2023. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Sys- tems, 36.
  16. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533.
  17. Pang, J.-C.; Wang, P.; Li, K.; Chen, X.-H.; Xu, J.; Zhang, Z.; and Yu, Y. 2024. Language Model Self-improvement by Reinforcement Learning Contemplation. In The Twelfth International Conference on Learning Representations.
  18. Rafailov, R.; Hejna, J.; Park, R.; and Finn, C. 2024. From r to Q*: Your Language Model is Secretly a Q-Function. arXiv preprint arXiv:2404.12358. · arXiv
  19. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems.
  20. Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Pro- cessing Systems, 36.
  21. Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587): 484-489.
  22. Song, Y.; Yin, D.; Yue, X.; Huang, J.; Li, S.; and Lin, B. Y. 2024. Trial and Error: Exploration-Based Trajectory Opti- mization for LLM Agents. arXiv preprint arXiv:2403.02502. · arXiv
  23. Wang, L.; Ma, C.; Feng, X.; Zhang, Z.; Yang, H.; Zhang, J.; Chen, Z.; Tang, J.; Chen, X.; Lin, Y.; et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 186345.
  24. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. · arXiv
  25. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824-24837.
  26. Xi, Z.; Chen, W.; Hong, B.; Jin, S.; Zheng, R.; He, W.; Ding, Y.; Liu, S.; Guo, X.; Wang, J.; et al. 2024a. Training Large Language Models for Reasoning through Reverse Curricu- lum Reinforcement Learning. International Conference on Machine Learning.
  27. Xi, Z.; Ding, Y.; Chen, W.; Hong, B.; Guo, H.; Wang, J.; Yang, D.; Liao, C.; Guo, X.; He, W.; et al. 2024b. Agent- Gym: Evolving Large Language Model-based Agents across Diverse Environments. arXiv preprint arXiv:2406.04151. · arXiv
  28. Xie, Y.; Kawaguchi, K.; Zhao, Y.; Zhao, X.; Kan, M.-Y.; He, J.; and Xie, Q. 2023. Decomposition enhances rea- soning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633, 2. · arXiv
  29. Yang, J. Q.; Salamatian, S.; Sun, Z.; Suresh, A. T.; and Beirami, A. 2024. Asymptotics of language model align- ment. arXiv preprint arXiv:2404.01730. · arXiv
  30. Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. arXiv preprint arXiv:1809.09600. · arXiv
  31. Yao, S.; Chen, H.; Yang, J.; and Narasimhan, K. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 20744-20757.
  32. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  33. Yuan, Z.; Yuan, H.; Li, C.; Dong, G.; Tan, C.; and Zhou, C. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825. · arXiv
  34. Zeng, A.; Liu, M.; Lu, R.; Wang, B.; Liu, X.; Dong, Y.; and Tang, J. 2023. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823. · arXiv
  35. Zhai, Y.; Bai, H.; Lin, Z.; Pan, J.; Tong, S.; Zhou, Y.; Suhr, A.; Xie, S.; LeCun, Y.; Ma, Y.; et al. 2024. Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning. arXiv preprint arXiv:2405.10292. · arXiv
  36. Zhong, H.; Feng, G.; Xiong, W.; Zhao, L.; He, D.; Bian, J.; and Wang, L. 2024. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922. · arXiv
  37. Zhou, A.; Yan, K.; Shlapentokh-Rothman, M.; Wang, H.; and Wang, Y.-X. 2024a. Language agent tree search uni- fies reasoning acting and planning in language models. In International conference on machine learning. PMLR.
  38. Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Bisk, Y.; Fried, D.; Alon, U.; et al. 2024b. We- barena: A realistic web environment for building autonomous agents. The Twelfth International Conference on Learning Representations.

Cited by

Sorted by citations

See all 7

Related

See all

Metadata

Authors
Zhai, Y., Yang, T., Xu, K., Feng, D., Yang, C., Ding, B., & Wang, H.
Affiliations
National University of Defense Technology (CN)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2025-04-11
Volume
39
Issue
25
Pages
27161-27169
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v39i25.34924
OpenAlex
W4409362104
Cited by
7
References
0
Citations by year
2026: 5 · 2025: 2
FWCI
6.65
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
LLM agents, multi-step decision-making, action selection, preference data, Monte Carlo tree search, preference-based reinforcement learning, WebShop, HotpotQA

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models. https://app.cochat.ai/papers/10.1609/aaai.v39i25.34924
The paper
Zhai, Y., Yang, T., Xu, K., Feng, D., Yang, C., Ding, B., & Wang, H. (2025). Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v39i25.34924