CoChat
Sign Up
PreprintCited by 1Reinforcement Learning in RoboticsSoftware Engineering ResearchDomain Adaptation and Few-Shot Learning

Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training

Open access
Lin, Jianzhe·2026·arXiv
DOI 10.48550/arxiv.2606.21090arXiv 2606.21090
Open this paper in CoChat — free accountOpen PDF

Abstract

Self-improvement can self-regress. In REINFORCE post-training for code, a model can quickly improve on its optimized metric and then collapse within the same training campaign. We study this in a controlled multi-seed testbed using Qwen-2.5-3B and Qwen-2.5-7B, trained on competitive-programming tasks with binary CodeGrader reward across 10 sequential 20-step campaigns. Across campaigns, pass@1 shows a robust rise-then-collapse pattern: it peaks within tens of gradient steps and then falls back, sometimes to near zero. This is not cross-task catastrophic forgetting, but within-task policy over-optimization on a fixed distribution; KL- and EWC-style constraints do not prevent it. We ask where the control loop should sit. We compare three levels: CARE, a between-campaign memory mechanism with a capability posterior, transfer gate, and regression-aware belief revision; ES, a within-campaign early-stop rule that rolls forward the peak checkpoint and sets the next budget to peak_step+3; and GRPO, which changes the RL update using group-relative reward normalization. The answer is regime-dependent. On Qwen-2.5-3B, where naive REINFORCE is fragile, CARE v2 nearly doubles end-of-chain pass@1 from 4.9% to 9.5%, with paired bootstrap 95% CI [+0.4,+8.9] and gains in 4/5 seeds. On Qwen-2.5-7B, CARE reaches parity with naive REINFORCE, 13.8% vs. 11.8%, while ES reaches 22.2% [14.1,28.0]. Out-of-the-box GRPO reaches 20.7% [15.7,25.1], nearly matching REINFORCE+ES. GRPO raises the floor but does not remove the cliff. Its 7B gain mainly comes from better between-campaign carryover, while the within-campaign peak-to-end gap remains about 17 points under both REINFORCE and GRPO. GRPO+ES gives mixed evidence: 2/3 seeds improve, but one final cliff lowers the mean to 17.0% [0.0,28.1]. A Gemma-3-4B pilot shows the same signature, suggesting the phenomenon is not limited to Qwen.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2606.21090

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Collapse is a default dynamic to be engineered against, not an edge case.

    “collapse is a default dynamic to be engineered against, not an edge case [93, 120]”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [93]

    The paper reports collapse in every campaign of its REINFORCE coding experiments and proposes an online predictor to intervene before it occurs. But the “Scope of the claim” section explicitly says rise-then-collapse is not claimed to be universal across algorithms, model families, or tasks. Calling it a default dynamic without that qualification overstates the source. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • In REINFORCE post-training for code, pass@1 climbs and then collapses within the same run, sometimes to near zero, under a genuinely verifiable binary reward, and KL- and EWC-style constraints do not prevent it.

    “Lin documents a rise-and-collapse failure mode in REINFORCE post-training for code: across sequential training campaigns, pass@1 climbs and then collapses within the same run - sometimes to near zero - under a genuinely verifiable binary reward, and KL- and EWC-style constraints do not prevent it [93].”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [93]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 39

  1. Anonymous. Exploration vs exploitation: Rethinking RLVR through clipping. In International Conference on Learning Representations (ICLR), 2026a. https://openreview.net/forum?id=sE8DCSJTzd. Anonymous. Prosperity before collapse: How far can off-policy RL reach with stale data on LLMs? In International Conference on Learning Representations (ICLR), 2026b. https://openreview.net/forum?id=IIgl5MWelz.
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. · arXiv
  3. Authors-TBD-from-arXiv-2602.09782. Flexible entropy control in RLVR with a gradient-preserving perspective. arXiv preprint arXiv:2602.09782, 2026. TODO: fill in author list from https://arxiv.org/abs/2602.09782 before camera-ready. · arXiv
  4. Authors-TBD-from-arXiv-2603.01162. Demystifying GRPO: Its policy gradient is equivalent to a reweighted REIN- FORCE. arXiv preprint arXiv:2603.01162, 2026. TODO: fill in author list from https://arxiv.org/abs/2603.01162 before camera-ready. · arXiv
  5. Authors-TBD-from-arXiv-2603.08660. How far can unsupervised RLVR scale LLM training? arXiv preprint arXiv:2603.08660, 2026. ICLR 2026 poster. TODO: fill in author list from https://arxiv.org/abs/2603.08660 before camera-ready. · arXiv
  6. Maximilian Balandat, Brian Karrer, Daniel R Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. Advances in Neural Information Processing Systems, 33:21524-21538, 2020.
  7. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, pages 41-48, 2009.
  8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
  9. Zhipeng Chen et al. Pass@k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508.10751, 2025. · arXiv
  10. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. SPIN: Self-play fine-tuning converts weak language models to strong language models. In International Conference on Machine Learning, 2024.
  11. Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective bayesian optimization. In Advances in Neural Information Processing Systems, volume 33, pages 9851-9864, 2020.
  12. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135, 2017.
  13. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. ReST meets ReAct: Self-improvement for multi-step reasoning LLM agent. arXiv preprint arXiv:2312.10003, 2023. · arXiv
  14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. · arXiv
  15. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with APPS. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
  16. Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. In arXiv preprint arXiv:1711.09846, 2017. · arXiv
  17. Florian Karl, Tobias Pielok, Julia Moosbauer, Florian Pfisterer, Stefan Coors, Martin Binder, Lennart Schneider, Janek Thomas, Jakob Richter, Michel Lang, Eduardo C. Garrido-Merchán, Juergen Branke, and Bernd Bischl. Multi-objective hyperparameter optimization in machine learning-an overview. ACM Transactions on Evolutionary Learning and Optimization, 3(4):1-50, 2022.
  18. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Oriol Vinyals, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. In Proceedings of the National Academy of Sciences, volume 114, pages 3521-3526, 2017.
  19. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024. · arXiv
  20. M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, volume 23, 2010.
  21. Sheikh Shafayat Kumar et al. Can large reasoning models self-train? arXiv preprint arXiv:2505.21444, 2025. · arXiv
  22. Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to learn quickly for few-shot learning. In arXiv preprint arXiv:1707.09835, 2017. · arXiv
  23. Siyi Liu, Ziran Chen, et al. AgentHPO: Large language model agent for hyperparameter optimization. arXiv preprint arXiv:2402.11427, 2024. · arXiv
  24. David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
  25. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024. · arXiv
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, 2022.
  27. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024. · arXiv
  28. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, 2023.
  29. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. · arXiv
  30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. · arXiv
  31. Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024.
  32. Manav Singhal et al. Reward design for code generation reinforcement learning: Binary vs. pass-rate signals. arXiv preprint arXiv:2502.18449, 2025. · arXiv
  33. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, volume 25, 2012.
  34. Ting Wu, Xuefeng Yuan, Xinghao Pan, et al. Progress or regress? Self-improvement reversal in post-training. arXiv preprint arXiv:2407.05013, 2024. · arXiv
  35. Sang Michael Xie, Hieu Pham, Xinyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S. Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems, volume 36, 2024.
  36. Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more CRINGE than others: Iterative preference optimization with human feedback. arXiv preprint arXiv:2312.16682, 2024. · arXiv
  37. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self- rewarding language models. arXiv preprint arXiv:2401.10020, 2024. · arXiv
  38. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35, 2022.
  39. Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995, 2017.

Cited by

Sorted by citations

  • Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

    Chen, M., Wang, L., & Qu, B. (2026) · arXiv

    “…three facts: (i) the loop’s ceiling is set by its verification signal (execution and proofs highest, learned judges lower, intrinsic signals lowest); (ii) collapse is a default dynamic to be engineered against, not an edge case [92, 120]; (iii) the loop transmits bias as efficiently as capability.”
    Influential

Authors
Lin, J.
Corresponding author
Lin, Jianzhe
ORCID
Lin, Jianzhe
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2606.21090
arXiv
2606.21090
OpenAlex
W7165628670
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
human development, catastrophic forgetting, REINFORCE, code generation, competitive programming, reward shaping, early stopping, belief revision, Qwen2.5, Gemma 3

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review