The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index; 19 of the 22 sources its claims cite were checked for support.
Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training
Open accessAbstract
Large Language Model (LLM) agents frequently produce sub-optimal actions when tackling complex, multi-step decision-making tasks. Employing self-critique to identify flaws and suggest enhancements is an effective strategy for refining actions. Although trajectory-level critique is commonly employed, it often fails to identify flawed steps accurately. In this paper, we introduce SLSC-MCTS, a method that integrates Monte Carlo Tree Search with Step-Level Self-Critique to enhance LLM agents during both testing and self-training phases. During decision tree expansion with SLSC-MCTS, the LLM agent initially generates an action, receives environmental feedback, and subsequently generates further actions via self-critique and refinement. Through multiple episodes of SLSC-MCTS, LLM agents can effectively utilize step-level critiques while disregarding ineffective ones based on node values, thereby incorporating the critiques more robustly. Additionally, our method further empowers LLM agents in a self-training manner, collecting training data from the constructed decision tree to iteratively fine-tune the LLM agents. The self-training data gathered via SLSC-MCTS is diverse and high-quality, which further enhances the reasoning, critiquing, and refining abilities of LLM agents. Experimental results demonstrate that SLSC-MCTS significantly improves LLM agents during testing, surpassing state-of-the-art baselines and achieving shorter task completion trajectories across information retrieval benchmarks such as WebShop and HotPotQA. After three iterations of self-training, LLM agents established by Llama-3.1-8B-Instruct show substantial improvement, even surpassing human experts in WebShop.
Overview
CoChat Summary
TL;DR
SLSC-MCTS combines Monte Carlo Tree Search with step-level self-critique and refinement, then uses the resulting decision trees for iterative self-training of LLM agents. It improves WebShop and HotPotQA performance over baselines, with self-trained Llama-3.1-8B-Instruct reaching 64% success on WebShop in the main table and surpassing human expert performance.
Key findings
- On WebShop, SLSC-MCTS + self-training achieved average reward 0.87 and success rate 64%, versus MCTS + self-training at 0.79 and 46%, and human expert at 0.82 and 60%; the abstract reports 63.5% success versus human experts’ 59.6%.
- On HotPotQA, SLSC-MCTS + self-training reached 78% exact match versus 70% for MCTS + self-training; test-time-only SLSC-MCTS reached 72% exact match versus 70% for MCTS, with trajectory length 2.43 versus 2.94.
- Step-level self-critique without MCTS was counterproductive: WebShop success 3% and HotPotQA 12%, lower than ReAct (10%, 30%) and Reflexion (18%, 49%).
- SLSC-MCTS collected more and more diverse self-training data than MCTS; by the 3rd iteration, total node number was 46.76 versus 24.40, retention rate 52% versus 44%, and Self-BLEU 0.25 versus 0.54.
Methods
- SLSC-MCTS builds a decision tree through expansion, evaluation, backpropagation, and selection. During expansion, the first node uses a direct action; later sibling nodes use self-critique followed by conditional refinement, repeated up to width \(w\). Evaluation uses rollout, and selection uses UCT.
- The self-training pipeline collects reasoning/refining trajectories and step-level critique data from the tree, filters trajectories by outcome reward \(r(u,\tau)>\beta\) and critiques by positive value difference \(\Delta V(s'_t)>0\), then fine-tunes with rejection sampling fine-tuning (RFT).
- Experiments use WebShop and HotPotQA with Llama-3.1-8B-Instruct; WebShop uses 1800 training and 200 test samples, HotPotQA uses 1000 training and 100 test samples, with 100 validation samples each. Hyperparameters include expansion width 3, MCTS episodes \(M=30\), RFT threshold \(r(u,\tau)>0.7\), learning rate \(5e-6\), and 3 epochs; Phi-3.5-mini-instruct is also tested.
Limitations
- The paper states that self-critique introduces more computational costs, even if this can be offset by higher search efficiency of SLSC-MCTS.
- The paper states that leveraging step-level self-critique assumes the ability to revert immediately after taking an action; this may not be universally applicable in all multi-step decision-making tasks.
- Evident from the text: evaluation is limited to two retrieval-based agent tasks (WebShop and HotPotQA), mainly with Llama-3.1-8B-Instruct plus one smaller-model check, and HotPotQA uses an oracle setup where the environment provides correctness feedback as soon as an answer is received.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 5 Not valid · 2 Not checked · 70 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“To ensure the quality of the trajectories, we follow [48] to filter the trajectories based on task completion status and only keep those trajectories with an outcome reward 𝑟 ( 𝑢, 𝜏 ) > 𝛽 .”
What the sources say
- Disputedcites [48] [48] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents. arXiv preprint arXiv:2403.02502 (2024). - sourceExperimental — accuracy not yet measured
The Exploration Phase (page unavailable) says the method keeps failure–success trajectory pairs with different final rewards, including the lower-reward trajectory, and discards pairs when both trajectories succeed. That contradicts the attributed rule of keeping only trajectories with outcome reward above β; the passage uses β in its reward-model equation, not as a filtering threshold. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing step-level methods for LLM agents [8-10] primarily focus on selecting the optimal action at each step from multiple candidates.”
What the sources say
- Partly supportedcites [8] [8] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. Language agent tree search unifies reasoning acting and planning in language models. In International conference on machine learning . PMLR. - sourceExperimental — accuracy not yet measured
Section 4.1 says LATS samples multiple candidate actions at each step, and Section 4.2 describes evaluating and selecting promising nodes. But the paper frames LATS as searching for the best trajectory, balancing exploration and exploitation, rather than primarily selecting an optimal action independently at each step. The broader claim about what existing methods primarily do is therefore not fully supported by this source. (LLM, conf=0.82)
- Supportedcites [9] [9] Yuanzhao Zhai, Tingkai Yang, Kele Xu, Feng Dawei, Cheng Yang, Bo Ding, and Huaimin Wang. 2025. Enhancing decision-making for LLM agents via step-level q-value models. Proceedings of the AAAI conference on artificial intelligence (2025). - source
The Introduction describes this paper’s step-level method: the agent samples multiple candidate actions at each decision-making step and selects the one with the highest Q-value. This supports the claim as a description of a cited example of such a method; the paper does not establish how prevalent the approach is across all existing methods. (LLM, conf=0.90)
- Partly supportedcites [10] [10] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199 (2024). - sourceExperimental — accuracy not yet measured
Section 5.1.1 (page unavailable) describes Agent Q sampling multiple proposed actions at each step and selecting among them using feedback and MCTS. It does not establish that existing step-level methods *primarily* focus on this approach; that generalization goes beyond what the cited passage says. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“our preliminary experiments indicate that directly applying them to LLM agents can be counterproductive.”
What the sources say
- Not matchedcites [13] [13] OpenAI. 2024. Learning to Reason with Large Language Models. https://openai. com/index/learning-to-reason-with-llms/. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [14] [14] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948 (2025). - sourceExperimental — accuracy not yet measured
The Introduction and Section 2.3 describe reflection as an emergent, beneficial reasoning behavior in DeepSeek-R1-Zero. Section G.2 discusses limitations of step-level process rewards, but not applying self-critiques to LLM agents. Searches for agent transfer and counterproductive self-critiques found no passage addressing the claim. (LLM, conf=0.74)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Despite notable advancements, even advanced LLM agents encounter difficulties in managing complex multi-step decision-making tasks [4]”
What the sources say
- Supportedcites [4] [4] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2024. Webarena: A realistic web environment for building autonomous agents. The Twelfth International Conference on Learning Representations (2024). - source
The Abstract describes WebArena’s tasks as diverse and long-horizon and reports that the best GPT-4-based agent completed only 14.41%, compared with 78.24% for humans. The Introduction likewise says that powerful LLM agents have limited success on complex web tasks. This supports the citing sentence’s general point about difficulty with complex multi-step tasks. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Model collapse is one of the critical challenges when training models on self-generated data [35, 36], especially in multi-round iterations due to the limited quality and diversity of self-generated data.”
What the sources say
- Supportedcites [35] [35] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. The Curse of Recursion: Training on Generated Data Makes Models Forget. ArXiv abs/2305.17493 (2023). https://api.semanticscholar.org/ CorpusID:258987240 - source
Section 3 defines model collapse as a degenerative process in which generated data pollute the training sets of successive models. Section 3.1 attributes it to errors that compound across generations, including information lost during resampling; the Introduction describes models forgetting improbable events. Together, these support the claim about repeated training on self-generated data with diminished quality and diversity. (LLM, conf=0.91)
- Partly supportedcites [36] [36] Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. 2024. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413 (2024). - sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) describes model collapse as progressive degradation across iterations of training on model-generated outputs. The Results (page unavailable) report declining quality and diversity of generated faces, but present these as outcomes, not as the established cause of collapse. The paper also finds that accumulating synthetic data alongside real data avoids collapse, so the claim needs that training-data qualification. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
41 more claims
Create a free account to see all 85 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 24
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.3 Main Results · p. 7
“| SLSC-MCTS (Ours) | 0.74 | 29% | 4.15”
§ Table 4: Effect of using different critique models. · p. 9
“| SLSC | 0.74 | 0.31 | 4.15 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.3 Main Results · p. 6
“As shown, SLSC-MCTS outperforms all baselines in both datasets by introducing more test-time computation.”
§ 5.3 Main Results · p. 7
“| LATS [8] | 0.69 | 27% | 5.08 | 73% | 3.21 |”
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citationMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970 · cited 1×
“Recently, Monte Carlo Tree Search (MCTS) [24] and other tree-based search algorithms has been demonstrated effective for LLM to improve planning and reasoning [25–29].”
Bandit Based Monte-Carlo Planning
Kocsis, L., & Szepesvári, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11871842_29 · Cited by 2,855
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Self-citationProceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,635
Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search
Coulom, R. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-75538-8_7 · Cited by 1,240
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep ThinkingPreprint
Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Fan, Y., & Mao, Y. H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.04519 · Cited by 426
Monte Carlo Tree Search Boosts Reasoning via Iterative Preference LearningPreprint
Xie, Y., Goyal, A., Zheng, W., Kan, M., Lillicrap, T., Kawaguchi, K., & Shieh, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.00451 · Cited by 308
(2022) · doi:10.52202/068431-1126 · Cited by 182
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.14219 · Cited by 166 · cited 1×
“5-mini-instruct [ 52] (3.8B parameters).”
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z., & Hu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.14992 · Cited by 166
The Curse of Recursion: Training on Generated Data Makes Models Forget
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023) · arXiv · doi:10.48550/arxiv.2305.17493 · Cited by 151
AgentGym: Evolving Large Language Model-based Agents across Diverse EnvironmentsPreprint
Xi, Z., Ding, Y.-W., Chen, W.-X., Hong, B., Guo, H., Wang, J.-Z., Yang, D., Liao, C.-Y., Guo, X., He, W., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2406.04151 · Cited by 120
Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language ModelsPreprint
Chen, Z.-H., Liu, K.-K., Wang, Q., Zhang, W.-W., Liu, J.-N., Lin, D.-H., Chen, K., & Zhao, F. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2403.12881 · Cited by 116
Large Language Models are Better Reasoners with Self-Verification
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., & Zhao, J. (2022) · arXiv · doi:10.48550/arxiv.2212.09561 · Cited by 110
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022) · arXiv · doi:10.48550/arxiv.2207.01206 · Cited by 107
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1508 · Cited by 107
Large Language Models and Future of Information Retrieval: Opportunities and Challenges
Zhai, C. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657848 · Cited by 100
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., & Manning, C. D. (2018) · arXiv · doi:10.48550/arxiv.1809.09600 · Cited by 97
ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search
Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., & Tang, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2066 · Cited by 93
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K. (2023) · arXiv · doi:10.48550/arxiv.2305.20050 · Cited by 65 · cited 1×
“Compared to outcome reward models (ORMs) that are trained based on the final results of responses, process reward models (PRMs) [42, 43] are often deemed more effective, as they offer step-wise reward signals for process supervision.”
AgentTuning: Enabling Generalized Agent Abilities for LLMs
Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y., & Tang, J. (2023) · arXiv · doi:10.48550/arxiv.2310.12823 · Cited by 40
Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models
Zhai, Y.-Z., Yang, T., Xu, K.-L., Feng, D.-W., Yang, C., Ding, B., & Wang, H.-M. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2409.09345 · Cited by 35
Self-citationReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search
Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., & Tang, J. (2024) · arXiv · doi:10.48550/arxiv.2406.03816 · Cited by 32
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27 · cited 1×
“Snell et al. [20] categorizes the test-time computation for LLM reasoning into sequential revisions and parallel sampling.”
Self-citationSelf-Verification Improves Few-Shot Clinical Information Extraction
Gero, Z., Singh, C., Cheng, H., Naumann, T., Galley, M., Gao, J., & Poon, H. (2023) · arXiv · doi:10.48550/arxiv.2306.00024 · Cited by 21
Progress or Regress? Self-Improvement Reversal in Post-trainingPreprint
Wu, T., Li, X., & Liu, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.05013 · Cited by 19
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
Zhou, A., Yan, K., Shlapentokh-Rothman, M., Wang, H., & Wang, Y.-X. (2023) · arXiv · doi:10.48550/arxiv.2310.04406 · Cited by 18
Self-citationUnderstanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17
CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
Lin, Z., Gou, Z., Liang, T., Luo, R., Liu, H., & Yang, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.14809 · Cited by 16
Self-citationGerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.01413 · Cited by 15
Iterative Reasoning Preference Optimization
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., & Weston, J. (2024) · arXiv · doi:10.48550/arxiv.2404.19733 · Cited by 15
AlphaMath Almost Zero: Process Supervision without Process
Chen, G., Liao, M., Li, C., & Fan, K. (2024) · arXiv · doi:10.48550/arxiv.2405.03553 · Cited by 14
Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
Putta, P., Mills, E., Garg, N., Motwani, S., Finn, C., Garg, D., & Rafailov, R. (2024) · arXiv · doi:10.48550/arxiv.2408.07199 · Cited by 10
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2308.01825 · Cited by 10
FireAct: Toward Language Agent Fine-tuning
Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2310.05915 · Cited by 9
Improve Mathematical Reasoning in Language Models by Automated Process Supervision
Luo, L., Liu, Y., Liu, R., Phatale, S., Guo, M., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.06592 · Cited by 8
Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models
Zhai, Y., Yang, T., Xu, K., Feng, D., Yang, C., Ding, B., & Wang, H. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i25.34924 · Cited by 7
Self-citationMutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers
Qi, Z., Ma, M., Xu, J., Zhang, L. L., Yang, F., & Yang, M. (2024) · arXiv · doi:10.48550/arxiv.2408.06195 · Cited by 4 · cited 1×
“Self-verification [21, 22] employs prompting techniques to generate self-diagnosed feedback, while rStar [23] introduces self-play mutual reasoning method, utilizing models to assess and refine each other’s results iteratively and ultimately selecting the most consistent answers.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training
Feng, X., Wan, Z., Wen, M., McAleer, S. M., Wen, Y., Zhang, W., & Wang, J. (2023) · arXiv · doi:10.48550/arxiv.2309.17179 · Cited by 4
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
Wang, P., Li, L., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., & Sui, Z. (2023) · arXiv · doi:10.48550/arxiv.2312.08935 · Cited by 3
Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
Song, Y., Yin, D., Yue, X., Huang, J., Li, S., & Lin, B. Y. (2024) · arXiv · doi:10.48550/arxiv.2403.02502 · Cited by 2
Zhiheng, X., Wenxiang, C., Xin, G., Wei, H., Yiwen, D., Boyang, H., Ming, Z., Junzhe, W., Senjie, J., Enyu, Z., et al. (2026) · SCIENTIA SINICA Informationis · doi:10.1360/ssi-2025-0532 · Cited by 1
Critique Ability of Large Language Models
Luo, L., Lin, Z., Liu, Y., Shu, L., Zhu, Y., Shang, J., & Meng, L. (2023) · arXiv · doi:10.48550/arxiv.2310.04815
Research works printed but not linked: 3
- Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, et al. 2024. Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning. International Conference on Machine Learning (2024).
- Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao. [n.,d.]. CriticEval: Evaluating Large-scale Language Model as Critic. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Pengfei Liu, et al. 2024. Generative Judge for Evaluating Alignment. In The Twelfth International Conference on Learning Representations.
Cited by
Evaluating LLMs on Kazakhstan's mathematics exam for university admission
Kadyrov, S., Abdrasilov, B. S., Sabyrov, A., Baizhanov, N., Makhmutova, A., & Kyllonen, P. C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1642570 · Cited by 6
FragLDM: Fragment-Guided Latent Diffusion Model for 3D Molecular Generation
Wang, R., Sun, R., Song, H., et al. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464993
Self-citation
Metadata
- Authors
- Zhai, Y., Liu, H., Zhang, Z., Lin, T., Xu, K., Yang, C., Feng, D., Ding, B., & Wang, H.
- Affiliations
- National University of Defense Technology (CN)
- Harbin Institute of Technology (CN)
- ORCID
- Venue
- Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2025-07-13
- Pages
- 1444-1454
- OpenAlex
- W4412394791
- Cited by
- 3
- References
- 10
- Citations by year
- 2026: 2 · 2025: 1
- FWCI
- 2.82
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Funders
- National Science and Technology Major Project (2023ZD0121101)
- National Key Laboratory under grant (231-HF-D04-01)
- the Open Fund of National Key Laboratory of Parallel and Distributed Computing (PDL) (NO.2024-KJWPDL-02)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLM agents, self-training, Monte Carlo tree search, multi-step decision-making, LLM fine-tuning, WebShop, HotpotQA, information retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.