CoChat

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index; 19 of the 22 sources its claims cite were checked for support.

Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training

Open access
Zhai, Yuanzhao, Liu, Huanxi, Zhang, Zhuo, Lin, Tong, Xu, Kele, Yang, Cheng, Feng, Dawei, Ding, Bo, Wang, Huaimin · 2025 · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval

Abstract

Large Language Model (LLM) agents frequently produce sub-optimal actions when tackling complex, multi-step decision-making tasks. Employing self-critique to identify flaws and suggest enhancements is an effective strategy for refining actions. Although trajectory-level critique is commonly employed, it often fails to identify flawed steps accurately. In this paper, we introduce SLSC-MCTS, a method that integrates Monte Carlo Tree Search with Step-Level Self-Critique to enhance LLM agents during both testing and self-training phases. During decision tree expansion with SLSC-MCTS, the LLM agent initially generates an action, receives environmental feedback, and subsequently generates further actions via self-critique and refinement. Through multiple episodes of SLSC-MCTS, LLM agents can effectively utilize step-level critiques while disregarding ineffective ones based on node values, thereby incorporating the critiques more robustly. Additionally, our method further empowers LLM agents in a self-training manner, collecting training data from the constructed decision tree to iteratively fine-tune the LLM agents. The self-training data gathered via SLSC-MCTS is diverse and high-quality, which further enhances the reasoning, critiquing, and refining abilities of LLM agents. Experimental results demonstrate that SLSC-MCTS significantly improves LLM agents during testing, surpassing state-of-the-art baselines and achieving shorter task completion trajectories across information retrieval benchmarks such as WebShop and HotPotQA. After three iterations of self-training, LLM agents established by Llama-3.1-8B-Instruct show substantial improvement, even surpassing human experts in WebShop.

Overview

CoChat Summary

TL;DR

SLSC-MCTS combines Monte Carlo Tree Search with step-level self-critique and refinement, then uses the resulting decision trees for iterative self-training of LLM agents. It improves WebShop and HotPotQA performance over baselines, with self-trained Llama-3.1-8B-Instruct reaching 64% success on WebShop in the main table and surpassing human expert performance.

Key findings

  • On WebShop, SLSC-MCTS + self-training achieved average reward 0.87 and success rate 64%, versus MCTS + self-training at 0.79 and 46%, and human expert at 0.82 and 60%; the abstract reports 63.5% success versus human experts’ 59.6%.
  • On HotPotQA, SLSC-MCTS + self-training reached 78% exact match versus 70% for MCTS + self-training; test-time-only SLSC-MCTS reached 72% exact match versus 70% for MCTS, with trajectory length 2.43 versus 2.94.
  • Step-level self-critique without MCTS was counterproductive: WebShop success 3% and HotPotQA 12%, lower than ReAct (10%, 30%) and Reflexion (18%, 49%).
  • SLSC-MCTS collected more and more diverse self-training data than MCTS; by the 3rd iteration, total node number was 46.76 versus 24.40, retention rate 52% versus 44%, and Self-BLEU 0.25 versus 0.54.

Methods

  • SLSC-MCTS builds a decision tree through expansion, evaluation, backpropagation, and selection. During expansion, the first node uses a direct action; later sibling nodes use self-critique followed by conditional refinement, repeated up to width \(w\). Evaluation uses rollout, and selection uses UCT.
  • The self-training pipeline collects reasoning/refining trajectories and step-level critique data from the tree, filters trajectories by outcome reward \(r(u,\tau)>\beta\) and critiques by positive value difference \(\Delta V(s'_t)>0\), then fine-tunes with rejection sampling fine-tuning (RFT).
  • Experiments use WebShop and HotPotQA with Llama-3.1-8B-Instruct; WebShop uses 1800 training and 200 test samples, HotPotQA uses 1000 training and 100 test samples, with 100 validation samples each. Hyperparameters include expansion width 3, MCTS episodes \(M=30\), RFT threshold \(r(u,\tau)>0.7\), learning rate \(5e-6\), and 3 epochs; Phi-3.5-mini-instruct is also tested.

Limitations

  • The paper states that self-critique introduces more computational costs, even if this can be offset by higher search efficiency of SLSC-MCTS.
  • The paper states that leveraging step-level self-critique assumes the ability to revert immediately after taking an action; this may not be universally applicable in all multi-step decision-making tasks.
  • Evident from the text: evaluation is limited to two retrieval-based agent tasks (WebShop and HotPotQA), mainly with Llama-3.1-8B-Instruct plus one smaller-model check, and HotPotQA uses an oracle setup where the environment provides correctness feedback as soon as an answer is received.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3726302.3729965

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

8 Valid · 5 Not valid · 2 Not checked · 70 Own results and interpretations

41 more claims

Create a free account to see all 85 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 17
  • It may state two separate claims as one. 24
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Research works printed but not linked: 3

  1. Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, et al. 2024. Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning. International Conference on Machine Learning (2024).
  2. Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao. [n.,d.]. CriticEval: Evaluating Large-scale Language Model as Critic. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  3. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Pengfei Liu, et al. 2024. Generative Judge for Evaluating Alignment. In The Twelfth International Conference on Learning Representations.

Cited by

See all 3

Metadata

Authors
Zhai, Y., Liu, H., Zhang, Z., Lin, T., Xu, K., Yang, C., Feng, D., Ding, B., & Wang, H.
Affiliations
  • National University of Defense Technology (CN)
  • Harbin Institute of Technology (CN)
Venue
Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2025-07-13
Pages
1444-1454
OpenAlex
W4412394791
Cited by
3
References
10
Citations by year
2026: 2 · 2025: 1
FWCI
2.82
Type
conference paper
Language
en
Open access
Locations
International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
Funders
  • National Science and Technology Major Project (2023ZD0121101)
  • National Key Laboratory under grant (231-HF-D04-01)
  • the Open Fund of National Key Laboratory of Parallel and Distributed Computing (PDL) (NO.2024-KJWPDL-02)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
LLM agents, self-training, Monte Carlo tree search, multi-step decision-making, LLM fine-tuning, WebShop, HotpotQA, information retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.