The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index; 19 of the 22 sources its claims cite were checked for support.
Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training
Open accessAbstract
Large Language Model (LLM) agents frequently produce sub-optimal actions when tackling complex, multi-step decision-making tasks. Employing self-critique to identify flaws and suggest enhancements is an effective strategy for refining actions. Although trajectory-level critique is commonly employed, it often fails to identify flawed steps accurately. In this paper, we introduce SLSC-MCTS, a method that integrates Monte Carlo Tree Search with Step-Level Self-Critique to enhance LLM agents during both testing and self-training phases. During decision tree expansion with SLSC-MCTS, the LLM agent initially generates an action, receives environmental feedback, and subsequently generates further actions via self-critique and refinement. Through multiple episodes of SLSC-MCTS, LLM agents can effectively utilize step-level critiques while disregarding ineffective ones based on node values, thereby incorporating the critiques more robustly. Additionally, our method further empowers LLM agents in a self-training manner, collecting training data from the constructed decision tree to iteratively fine-tune the LLM agents. The self-training data gathered via SLSC-MCTS is diverse and high-quality, which further enhances the reasoning, critiquing, and refining abilities of LLM agents. Experimental results demonstrate that SLSC-MCTS significantly improves LLM agents during testing, surpassing state-of-the-art baselines and achieving shorter task completion trajectories across information retrieval benchmarks such as WebShop and HotPotQA. After three iterations of self-training, LLM agents established by Llama-3.1-8B-Instruct show substantial improvement, even surpassing human experts in WebShop.
Overview
CoChat Summary
TL;DR
SLSC-MCTS combines Monte Carlo Tree Search with step-level self-critique and refinement, then uses the resulting decision trees for iterative self-training of LLM agents. It improves WebShop and HotPotQA performance over baselines, with self-trained Llama-3.1-8B-Instruct reaching 64% success on WebShop in the main table and surpassing human expert performance.
Key findings
- On WebShop, SLSC-MCTS + self-training achieved average reward 0.87 and success rate 64%, versus MCTS + self-training at 0.79 and 46%, and human expert at 0.82 and 60%; the abstract reports 63.5% success versus human experts’ 59.6%.
- On HotPotQA, SLSC-MCTS + self-training reached 78% exact match versus 70% for MCTS + self-training; test-time-only SLSC-MCTS reached 72% exact match versus 70% for MCTS, with trajectory length 2.43 versus 2.94.
- Step-level self-critique without MCTS was counterproductive: WebShop success 3% and HotPotQA 12%, lower than ReAct (10%, 30%) and Reflexion (18%, 49%).
- SLSC-MCTS collected more and more diverse self-training data than MCTS; by the 3rd iteration, total node number was 46.76 versus 24.40, retention rate 52% versus 44%, and Self-BLEU 0.25 versus 0.54.
Methods
- SLSC-MCTS builds a decision tree through expansion, evaluation, backpropagation, and selection. During expansion, the first node uses a direct action; later sibling nodes use self-critique followed by conditional refinement, repeated up to width \(w\). Evaluation uses rollout, and selection uses UCT.
- The self-training pipeline collects reasoning/refining trajectories and step-level critique data from the tree, filters trajectories by outcome reward \(r(u,\tau)>\beta\) and critiques by positive value difference \(\Delta V(s'_t)>0\), then fine-tunes with rejection sampling fine-tuning (RFT).
- Experiments use WebShop and HotPotQA with Llama-3.1-8B-Instruct; WebShop uses 1800 training and 200 test samples, HotPotQA uses 1000 training and 100 test samples, with 100 validation samples each. Hyperparameters include expansion width 3, MCTS episodes \(M=30\), RFT threshold \(r(u,\tau)>0.7\), learning rate \(5e-6\), and 3 epochs; Phi-3.5-mini-instruct is also tested.
Limitations
- The paper states that self-critique introduces more computational costs, even if this can be offset by higher search efficiency of SLSC-MCTS.
- The paper states that leveraging step-level self-critique assumes the ability to revert immediately after taking an action; this may not be universally applicable in all multi-step decision-making tasks.
- Evident from the text: evaluation is limited to two retrieval-based agent tasks (WebShop and HotPotQA), mainly with Llama-3.1-8B-Instruct plus one smaller-model check, and HotPotQA uses an oracle setup where the environment provides correctness feedback as soon as an answer is received.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 5 Not valid · 2 Not checked · 70 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“To ensure the quality of the trajectories, we follow [48] to filter the trajectories based on task completion status and only keep those trajectories with an outcome reward 𝑟 ( 𝑢, 𝜏 ) > 𝛽 .”
What the sources say
- Disputedcites [48] [48] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents. arXiv preprint arXiv:2403.02502 (2024). - sourceExperimental — accuracy not yet measured
The Exploration Phase (page unavailable) says the method keeps failure–success trajectory pairs with different final rewards, including the lower-reward trajectory, and discards pairs when both trajectories succeed. That contradicts the attributed rule of keeping only trajectories with outcome reward above β; the passage uses β in its reward-model equation, not as a filtering threshold. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing step-level methods for LLM agents [8-10] primarily focus on selecting the optimal action at each step from multiple candidates.”
What the sources say
- Partly supportedcites [8] [8] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. Language agent tree search unifies reasoning acting and planning in language models. In International conference on machine learning . PMLR. - sourceExperimental — accuracy not yet measured
Section 4.1 says LATS samples multiple candidate actions at each step, and Section 4.2 describes evaluating and selecting promising nodes. But the paper frames LATS as searching for the best trajectory, balancing exploration and exploitation, rather than primarily selecting an optimal action independently at each step. The broader claim about what existing methods primarily do is therefore not fully supported by this source. (LLM, conf=0.82)
- Supportedcites [9] [9] Yuanzhao Zhai, Tingkai Yang, Kele Xu, Feng Dawei, Cheng Yang, Bo Ding, and Huaimin Wang. 2025. Enhancing decision-making for LLM agents via step-level q-value models. Proceedings of the AAAI conference on artificial intelligence (2025). - source
The Introduction describes this paper’s step-level method: the agent samples multiple candidate actions at each decision-making step and selects the one with the highest Q-value. This supports the claim as a description of a cited example of such a method; the paper does not establish how prevalent the approach is across all existing methods. (LLM, conf=0.90)
- Partly supportedcites [10] [10] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199 (2024). - sourceExperimental — accuracy not yet measured
Section 5.1.1 (page unavailable) describes Agent Q sampling multiple proposed actions at each step and selecting among them using feedback and MCTS. It does not establish that existing step-level methods *primarily* focus on this approach; that generalization goes beyond what the cited passage says. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“our preliminary experiments indicate that directly applying them to LLM agents can be counterproductive.”
What the sources say
- Not matchedcites [13] [13] OpenAI. 2024. Learning to Reason with Large Language Models. https://openai. com/index/learning-to-reason-with-llms/. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [14] [14] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948 (2025). - sourceExperimental — accuracy not yet measured
The Introduction and Section 2.3 describe reflection as an emergent, beneficial reasoning behavior in DeepSeek-R1-Zero. Section G.2 discusses limitations of step-level process rewards, but not applying self-critiques to LLM agents. Searches for agent transfer and counterproductive self-critiques found no passage addressing the claim. (LLM, conf=0.74)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Despite notable advancements, even advanced LLM agents encounter difficulties in managing complex multi-step decision-making tasks [4]”
What the sources say
- Supportedcites [4] [4] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2024. Webarena: A realistic web environment for building autonomous agents. The Twelfth International Conference on Learning Representations (2024). - source
The Abstract describes WebArena’s tasks as diverse and long-horizon and reports that the best GPT-4-based agent completed only 14.41%, compared with 78.24% for humans. The Introduction likewise says that powerful LLM agents have limited success on complex web tasks. This supports the citing sentence’s general point about difficulty with complex multi-step tasks. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Model collapse is one of the critical challenges when training models on self-generated data [35, 36], especially in multi-round iterations due to the limited quality and diversity of self-generated data.”
What the sources say
- Supportedcites [35] [35] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. The Curse of Recursion: Training on Generated Data Makes Models Forget. ArXiv abs/2305.17493 (2023). https://api.semanticscholar.org/ CorpusID:258987240 - source
Section 3 defines model collapse as a degenerative process in which generated data pollute the training sets of successive models. Section 3.1 attributes it to errors that compound across generations, including information lost during resampling; the Introduction describes models forgetting improbable events. Together, these support the claim about repeated training on self-generated data with diminished quality and diversity. (LLM, conf=0.91)
- Partly supportedcites [36] [36] Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. 2024. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413 (2024). - sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) describes model collapse as progressive degradation across iterations of training on model-generated outputs. The Results (page unavailable) report declining quality and diversity of generated faces, but present these as outcomes, not as the established cause of collapse. The paper also finds that accumulating synthetic data alongside real data avoids collapse, so the claim needs that training-data qualification. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
41 more claims
Create a free account to see all 85 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 24
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.3 Main Results · p. 7
“| SLSC-MCTS (Ours) | 0.74 | 29% | 4.15”
§ Table 4: Effect of using different critique models. · p. 9
“| SLSC | 0.74 | 0.31 | 4.15 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.3 Main Results · p. 6
“As shown, SLSC-MCTS outperforms all baselines in both datasets by introducing more test-time computation.”
§ 5.3 Main Results · p. 7
“| LATS [8] | 0.69 | 27% | 5.08 | 73% | 3.21 |”
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citationMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970 · cited 1×
“Recently, Monte Carlo Tree Search (MCTS) [24] and other tree-based search algorithms has been demonstrated effective for LLM to improve planning and reasoning [25–29].”
Bandit Based Monte-Carlo Planning
Kocsis, L., & Szepesvári, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11871842_29 · Cited by 2,855
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Self-citation
Research works printed but not linked: 3
- Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, et al. 2024. Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning. International Conference on Machine Learning (2024).
- Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao. [n.,d.]. CriticEval: Evaluating Large-scale Language Model as Critic. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Pengfei Liu, et al. 2024. Generative Judge for Evaluating Alignment. In The Twelfth International Conference on Learning Representations.
Cited by
Evaluating LLMs on Kazakhstan's mathematics exam for university admission
Kadyrov, S., Abdrasilov, B. S., Sabyrov, A., Baizhanov, N., Makhmutova, A., & Kyllonen, P. C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1642570 · Cited by 6
FragLDM: Fragment-Guided Latent Diffusion Model for 3D Molecular Generation
Wang, R., Sun, R., Song, H., et al. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464993
Self-citation
Metadata
- Authors
- Zhai, Y., Liu, H., Zhang, Z., Lin, T., Xu, K., Yang, C., Feng, D., Ding, B., & Wang, H.
- Affiliations
- National University of Defense Technology (CN)
- Harbin Institute of Technology (CN)
- ORCID
- Venue
- Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2025-07-13
- Pages
- 1444-1454
- OpenAlex
- W4412394791
- Cited by
- 3
- References
- 10
- Citations by year
- 2026: 2 · 2025: 1
- FWCI
- 2.82
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Funders
- National Science and Technology Major Project (2023ZD0121101)
- National Key Laboratory under grant (231-HF-D04-01)
- the Open Fund of National Key Laboratory of Parallel and Distributed Computing (PDL) (NO.2024-KJWPDL-02)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLM agents, self-training, Monte Carlo tree search, multi-step decision-making, LLM fine-tuning, WebShop, HotpotQA, information retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training. https://app.cochat.ai/papers/10.1145/3726302.3729965
- The paper
- Zhai, Y., Liu, H., Zhang, Z., Lin, T., Xu, K., Yang, C., Feng, D., Ding, B., & Wang, H. (2025). Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3726302.3729965
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.