CoChat
Sign Up
PreprintCited by 1Topic ModelingLogic, Reasoning, and KnowledgeDomain Adaptation and Few-Shot Learning

Better, Faster: Harnessing Self-Improvement in Large Reasoning Models

Open access
Zhong, Qihuang, Ding, Liang, Liu, Juhua, Du, Bo, Rutkowski, Leszek, Tao, Dacheng·2026·arXiv
DOI 10.48550/arxiv.2605.24998arXiv 2605.24998
Open this paper in CoChat — free accountOpen PDF

Abstract

Self-improvement training enables the large reasoning models (LRMs) to improve themselves by self-generating reasoning trajectories as training data without external supervision. However, we find that this method often falls short in complex reasoning tasks and even leads to model collapse. Through a series of preliminary analyses, we reveal two problems: (1) data imbalance, where most training samples are simple, but the challenging yet crucial samples are scarce; (2) overthinking, where many undesired samples with redundant reasoning steps are used for self-training. To this end, we propose HSIR, which effectively Harnesses Self-Improvement in large Reasoning models via two simple-yet-effective approaches. Specifically, HSIR introduces a verify-then-exit sampling strategy to mitigate data imbalance by efficiently collecting more accurate solutions for difficult queries, and designs an Intrinsic Diversity score to quantify overthinking and filter out the undesired solutions. We apply HSIR to various post-training paradigms, among which we further propose H-GRPO, an enhanced GRPO algorithm that leverages the intrinsic diversity as an external reward to encourage concise and diverse reasoning via reinforcement learning. Extensive results show that HSIR not only effectively enhances the reasoning performance, i.e., bringing up to +10.9% average performance gains, but also significantly improves the reasoning efficiency by reducing up to 42.4% relative inference overhead.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2605.24998

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Reasoning self-training collapses from identifiable data imbalance and overthinking, both fixable.

    “reasoning self-training collapses from identifiable data imbalance and overthinking, both fixable [148].”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [148]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 72

  1. Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2412.08905, 2024. · arXiv
  2. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Hu- mayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. Self-consuming generative models go mad. In The Twelfth International Conference on Learn- ing Representations, 2024.
  3. Azaria, A. and Mitchell, T. The internal state of an llm knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
  4. Bertrand, Q., Bose, J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of gener- ative models on their own data. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
  5. Cao, Y., Kang, Y., Wang, C., and Sun, L. Instruction mining: Instruction data selection for tuning large language mod- els. In First Conference on Language Modeling, 2024.
  6. Chen, C., Liu, K., Chen, Z., Gu, Y., Wu, Y., Tao, M., Fu, Z., and Ye, J. Inside: Llms' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024a.
  7. Chen, H., Fang, Z., Singla, Y., and Dredze, M. Benchmark- ing large language models on answering and explaining challenging medical questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025a.
  8. Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., Hou, J., and Wang, B. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925, 2024b. · arXiv
  9. Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. M3-embedding: Multi-linguality, multi- functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics ACL 2024, 2024c.
  10. Chen, J. C.-Y., Peng, B. X., Choubey, P. K., Huang, K.-H., Zhang, J., Bansal, M., and Wu, C.-S. Nudging the bound- aries of llm reasoning. arXiv preprint arXiv:2509.25666, 2025b. · arXiv
  11. Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Ma- toba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079, 2023. · arXiv
  12. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. · arXiv
  13. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
  14. Dai, M., Liu, S., and Si, Q. Stable reinforcement learning for efficient reasoning. arXiv preprint arXiv:2505.18086, 2025a. · arXiv
  15. Dai, M., Yang, C., and Si, Q. S-grpo: Early exit via rein- forcement learning in reasoning models. arXiv preprint arXiv:2505.07686, 2025b. · arXiv
  16. Ding, Y., Xi, Z., He, W., Lizhuoyuan, L., Zhai, Y., Xi- aowei, S., Cai, X., Gui, T., Zhang, Q., and Huang, X.-J. Mitigating tail narrowing in llm self-improvement via socratic-guided sampling. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025.
  17. Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transac- tions of the Association for Computational Linguistics, 2021.
  18. Giraldo, L. G. S., Rao, M., and Principe, J. C. Measures of entropy from data using infinitely divisible kernels. IEEE Transactions on Information Theory, 2014.
  19. Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023. · arXiv
  20. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. · arXiv
  21. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021.
  22. Horn, R. A. and Johnson, C. R. Matrix analysis. Cambridge university press, 2012.
  23. Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sor- doni, A., and Agarwal, R. V-star: Training verifiers for self-taught reasoners. In First Conference on Language Modeling, 2024.
  24. Huang, A., Block, A., Foster, D. J., Rohatgi, D., Zhang, C., Simchowitz, M., Ash, J. T., and Krishnamurthy, A. Self-improvement in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representations, 2025a.
  25. Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
  26. Huang, Z., Zou, H., Li, X., Liu, Y., Zheng, Y., Chern, E., Xia, S., Qin, Y., Yuan, W., and Liu, P. O1 replication journey-part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489, 2024. · arXiv
  27. Huang, Z., Geng, G., Hua, S., Huang, Z., Zou, H., Zhang, S., Liu, P., and Zhang, X. O1 replication journey-part 3: Inference-time scaling for medical reasoning. arXiv preprint arXiv:2501.06458, 2025b. · arXiv
  28. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. · arXiv
  29. Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Ham- bro, E., Grefenstette, E., and Raileanu, R. Understanding the effects of rlhf on llm generalisation and diversity. In The Twelfth International Conference on Learning Repre- sentations, 2024.
  30. Koh, W., Oh, W., Jang, J., Lee, M., Kim, H., Kim, A. Y., Kim, J., Lee, J., Kim, T., and Yun, S.-Y. Adastar: Adap- tive data sampling for training self-taught reasoners. In Proceedings of the 39th International Conference on Neu- ral Information Processing Systems, 2025.
  31. Li, G., Al Kader Hammoud, H. A., Itani, H., Khizbullin, D., and Ghanem, B. Camel: communicative agents for" mind" exploration of large language model society. In Proceedings of the 37th International Conference on Neu- ral Information Processing Systems, 2023.
  32. Li, Z.-Z., Zhang, D., Zhang, M.-L., Zhang, J., Liu, Z., Yao, Y., Xu, H., Zheng, J., Wang, P.-J., Chen, X., et al. From system 1 to system 2: A survey of reasoning large lan- guage models. arXiv preprint arXiv:2502.17419, 2025. · arXiv
  33. Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., and Smith, N. A. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, 2019.
  34. Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. What makes good data for alignment? a comprehensive study of auto- matic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2024.
  35. Manakul, P., Liusie, A., and Gales, M. Selfcheckgpt: Zero- resource black-box hallucination detection for genera- tive large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, 2023. Mathematical Association of America. American mathe- matics competitions, 2023. URL https://maa-amc. org/amc/amc-competitions/.
  36. Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Mat- sukawa, A., and Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, 2020.
  37. Munkhbat, T., Ho, N., Kim, S. H., Yang, Y., Kim, Y., and Yun, S.-Y. Self-training elicits concise reasoning in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, 2025.
  38. Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024.
  39. Peng, X., Xia, C., Yang, X., Xiong, C., Wu, C.-S., and Xing, C. Regenesis: Llms can grow into reasoning generalists via self-improvement. In The Thirteenth International Conference on Learning Representations, 2025.
  40. Plaat, A., Wong, A., Verberne, S., Broekens, J., van Stein, N., and Bäck, T. Reasoning with large language models, a survey. arXiv e-prints arXiv:2407.11511, 2024. · arXiv
  41. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Man- ning, C. D., and Finn, C. Direct preference optimization: your language model is secretly a reward model. In Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, 2023.
  42. Rahmath P, H., Srivastava, V., Chaurasia, K., Pacheco, R. G., and Couto, R. S. Early-exit deep neural network-a com- prehensive survey. ACM Computing Surveys, 2024.
  43. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
  44. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. · arXiv
  45. Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., et al. Beyond human data: Scaling self-training for problem- solving with language models. Transactions on Machine Learning Research, 2023.
  46. Skean, O., Osorio, J. K. H., Brockmeier, A. J., and Gi- raldo, L. G. S. Dime: Maximizing mutual information by a difference of matrix-based entropies. arXiv preprint arXiv:2301.08164, 2023. · arXiv
  47. Skean, O., Arefin, M. R., Zhao, D., Patel, N. N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R. Layer by layer: Uncov- ering hidden representations in language models. In Forty- second International Conference on Machine Learning, 2025.
  48. Song, Y., Zhang, H., Eisenach, C., Kakade, S. M., Foster, D., and Ghai, U. Mind the gap: Examining the self- improvement capabilities of large language models. In The Thirteenth International Conference on Learning Representations, 2025.
  49. Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
  50. Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. · arXiv
  51. Tong, Y., Zhang, X., Wang, R., Wu, R., and He, J. Dart- math: difficulty-aware rejection tuning for mathematical problem-solving. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024.
  52. Wang, T., Li, S., and Lu, W. Self-training with direct prefer- ence optimization improves chain-of-thought reasoning. In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics, 2024.
  53. Wang, W., Liu, X., Huang, B., Cui, H., Zhang, R., Yu, C., Jin, S., Yang, J., Yin, Q., Wang, Z., et al. Heapa: Difficulty-aware heap sampling and on-policy query aug- mentation for llm reinforcement learning. arXiv preprint arXiv:2601.22448, 2026. · arXiv
  54. Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. In International Conference on Learning Representations, 2022.
  55. Wen, L., Cai, Y., Xiao, F., He, X., An, Q., Duan, Z., Du, Y., Liu, J., Tanglifu, T., Lv, X., Zou, H., Deng, Y., Jia, S., and Zhang, X. Light-r1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. In Proceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025.
  56. Wu, T., Li, X., and Liu, P. Progress or regress? self- improvement reversal in post-training. In The Thirteenth International Conference on Learning Representations, 2025.
  57. Xiao, W. and Gan, L. Fast-slow thinking grpo for large vision-language model reasoning. Advances in Neural Information Processing Systems, 2026.
  58. Xu, F., Hao, Q., Zong, Z., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., Ouyang, T., Meng, F., et al. To- wards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686, 2025a. · arXiv
  59. Xu, H., Wu, X., Wang, W., Li, Z., Zheng, D., Chen, B., Hu, Y., Kang, S., Ji, J., Zhang, Y., et al. Redstar: Does scal- ing long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 2025b. · arXiv
  60. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. · arXiv
  61. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. · arXiv
  62. Yang, C., Si, Q., Duan, Y., Zhu, Z., Zhu, C., Li, Q., Lin, Z., Cao, L., and Wang, W. Dynamic early exit in reasoning models. arXiv preprint arXiv:2504.15895, 2025b. · arXiv
  63. Yang, S., Tong, Y., Niu, X., Neubig, G., and Yue, X. Demys- tifying long chain-of-thought reasoning. In Forty-second International Conference on Machine Learning, 2025c.
  64. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. · arXiv
  65. Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv e-prints arXiv: 2308.01825, 2023. · arXiv
  66. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D. Star: self- taught reasoner bootstrapping reasoning with reasoning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022.
  67. Zhang, C., Yang, Y., Liu, J., Wang, J., Xian, Y., Wang, B., and Song, D. Lifting the curse of capacity gap in distilling language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.
  68. Zhang, D., Wu, J., Lei, J., Che, T., Li, J., Xie, T., Huang, X., Zhang, S., Pavone, M., Li, Y., et al. Llama-berry: Pairwise optimization for olympiad-level mathematical reasoning via o1-like monte carlo tree search. In Pro- ceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025.
  69. Zhong, Q., Ding, L., Cai, X., Liu, J., Du, B., and Tao, D. Kaft: Knowledge-aware fine-tuning for boosting llms' domain-specific question-answering performance. In Findings of the Association for Computational Linguis- tics: ACL 2025, 2025a.
  70. Zhong, Q., Ding, L., Liao, F., Liu, J., Du, B., and Tao, D. Resolving knowledge conflicts in domain-specific data selection: A case study on medical instruction-tuning. arXiv preprint arXiv:2505.21958, 2025b. · arXiv
  71. Zhong, Q., Ding, L., Xuan, W., Liu, J., Du, B., and Tao, D. Learn to think: Improving multimodal reasoning through vision-aware self-improvement training. arXiv preprint, 2026a.
  72. Zuo, Y., Qu, S., Li, Y., Chen, Z.-R., Zhu, X., Hua, E., Zhang, K., Ding, N., and Zhou, B. Medxpertqa: Benchmark- ing expert-level medical reasoning and understanding. In Forty-second International Conference on Machine Learning, 2025.

Cited by

Sorted by citations

Authors
Zhong, Q., Ding, L., Liu, J., Du, B., Rutkowski, L., & Tao, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2605.24998
arXiv
2605.24998
OpenAlex
W7162432771
Cited by
0
References
2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Nanyang Technological University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
large reasoning models, self-training, class imbalance, overthinking, Group Relative Policy Optimization, reinforcement learning, reasoning efficiency, model collapse

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review