Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL
Open accessAbstract
Self-play reinforcement learning trains language models on their own generated tasks, co-evolving a proposer and solver without human labels. Recent systems report strong reasoning gains, but collapse and instability are widely observed and poorly understood. The dominant response treats this as a reward-design problem. We argue instead that self-play stability is governed by two distinct levers: a data-level gate that decides which proposer-generated tasks enter the training pool, and the reward signal that updates the policy on tasks already admitted. Through controlled experiments on a Python output-prediction task and a deterministic-DSL twin task that strips pretraining priors, output ambiguity, and executor noise, we find the two levers are asymmetric. A strict gate is sufficient for stability under every reward variant we test, including a self-consistency reward with no access to ground truth; while no reward variant is sufficient once the gate is removed. This asymmetry exposes a counter-intuitive coupling we call the Grounded Proposer Paradox: a proposer with ground-truth access accelerates collapse faster than an ungrounded one when paired with a self-consistency solver, by concentrating training on clean tasks that form the fastest path to a spurious self-consistent attractor. Replacing the binary gate with a continuous strictness parameter $\varepsilon$ further reveals a two-stage phase transition: training-side metrics decouple at low $\varepsilon$, while validation accuracy holds until $\varepsilon$ is much higher. Data-level gating, not reward calibration, is the binding constraint on self-play stability.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Self-play survival is governed by two asymmetric levers, data gating and reward grounding, and collapse is the default when either fails, not an occasional accident of reward design.
“self-play survival is governed by two asymmetric levers data gating (what enters the training set) and reward grounding (what anchors the signal to reality) - and that collapse is the default when either fails, not an occasional accident of reward design [120].”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [120]
The Introduction identifies data gating and reward as distinct, asymmetric levers, but directly contradicts the claim that collapse follows when either one fails. It reports that a strict gate kept every tested reward variant stable, including a self-consistency reward with no access to ground truth; removing the gate caused collapse regardless of reward design. These findings are limited to the variants tested, not a universal rule that either failure causes collapse. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredCollapse is a default dynamic to be engineered against, not an edge case.
“collapse is a default dynamic to be engineered against, not an edge case [93, 120]”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [120]
The papers the analysis leaned on hardest were disproportionately diagnostic: controlled decompositions of feedback value, placebo-controlled self-repair studies, stability analyses of self-play, and integrity benchmarks.
“the papers our analysis leaned on hardest were disproportionately diagnostic: controlled decompositions of feedback value [39], placebo-controlled self-repair studies [40], stability analyses of self-play [120], integrity benchmarks [143].”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [120]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Yue, Y., Lin, M., Wang, S., Wu, Q., Zheng, Z., et al. (2025) · arXiv · Cited by 13 · cited 3× · see citing works
“The coding task is the natural setting for self-play on code and matches the design of prior work (Zhao et al., 2025).”
Flagged first, then by citations
Research works printed but not linked: 17
- Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, and Tengyu Ma. Scaling self-play with self- guidance. arXiv preprint arXiv:2604.20209, 2026. URL https://arxiv.org/abs/2604.20209. · arXiv
- DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. URL https://arxiv.org/abs/2501.12948. · arXiv
- Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024. · arXiv
- Bingxiang He, Yuxin Zuo, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang, Yuchen Fan, Ganqu Cui, Xiusi Chen, Youbang Sun, Xingtai Lv, Xuekai Zhu, Li Sheng, Ran Li, Huan-ang Gao, Yuchen Zhang, Bowen Zhou, Zhiyuan Liu, and Ning Ding. How far can unsupervised RLVR scale LLM training? arXiv preprint arXiv:2603.08660, 2026. URL https://arxiv.org/abs/2603.08660. · arXiv
- Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Skalse. Goodhart's law in reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=5o9G4XF1LI.
- Thomas Kwa, Drake Thomas, and Adrià Garriga-Alonso. Catastrophic Goodhart: Regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. In Advances in Neural Information Pro- cessing Systems, volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/hash/1a8189929f3d7bd6183718f42c3f4309-Abstract-Conference.html.
- Xiao Liang, Zhong-Zhi Li, Yeyun Gong, Yang Wang, Hengyuan Zhang, Yelong Shen, Ying Nian Wu, and Weizhu Chen. Beyond Pass@1: Self-play with variational problem synthesis sustains RLVR. arXiv preprint arXiv:2508.14029, 2025. URL https://arxiv.org/abs/2508.14029. · arXiv
- Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum? id=1qvx610Cu7.
- Wei Liu, Siya Qi, Yali Du, and Yulan He. Self-play only evolves when self-synthetic pipeline ensures learnable information gain. arXiv preprint arXiv:2603.02218, 2026. URL https://arxiv.org/abs/2603.02218. · arXiv
- Hongliang Lu, Yuhang Wen, Pengyu Cheng, Ruijin Ding, Jiaqi Guo, Haotian Xu, Chutian Wang, Haonan Chen, Xiaoxi Jiang, and Guanjun Jiang. Search self-play: Pushing the frontier of agent capability without supervision. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=ZmGirmNJqE.
- Andreas Plesner, Francisco Guzmán, and Anish Athalye. An imperfect verifier is good enough: Learning with noisy rewards. arXiv preprint arXiv:2604.07666, 2026. URL https://arxiv.org/abs/2604.07666. · arXiv
- Ali Rad, Khashayar Filom, Darioush Keivan, Peyman Mohajerin Esfahani, and Ehsan Kamalinejad. Rate or fate? RLVεR: Reinforcement learning with verifiable noisy rewards. arXiv preprint arXiv:2601.04411, 2026. URL https://arxiv.org/abs/2601.04411. Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train? arXiv preprint arXiv:2505.21444, 2025. URL https://arxiv.org/abs/ 2505.21444. · arXiv
- Toby Simonds, Kevin Lopez, Akira Yoshiyama, and Dominique Garmier. RLSR: Reinforcement learning from self reward. arXiv preprint arXiv:2505.08827, 2025. URL https://arxiv.org/abs/2505.08827. · arXiv
- Rui Wu and Ruixiang Tang. When reward hacking rebounds: Understanding and mitigating it with representation-level signals. arXiv preprint arXiv:2604.01476, 2026. URL https://arxiv.org/abs/ 2604.01476. · arXiv
- Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. Overconfident errors need stronger correction: Asymmetric confidence penalties for reinforcement learning. arXiv preprint arXiv:2602.21420, 2026. URL https://arxiv.org/abs/2602.21420. · arXiv
- Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Vincent Wang, and Chris Lee. Spurious rewards paradox: Mechanistically understanding how rlvr activates memorization shortcuts in llms, 2026. URL https://arxiv.org/abs/2601.11061. · arXiv
- Ziyi Yang, Weizhou Shen, Chenliang Li, Ruijun Chen, Fanqi Wan, Ming Yan, Xiaojun Quan, and Fei Huang. SPELL: Self-play reinforcement learning for evolving long-context language models. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum? id=83F6YF4Hz6.
Cited by
Sorted by citations
Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
Chen, M., Wang, L., & Qu, B. (2026) · arXiv
“…three facts: (i) the loop’s ceiling is set by its verification signal (execution and proofs highest, learned judges lower, intrinsic signals lowest); (ii) collapse is a default dynamic to be engineered against, not an edge case [92, 120]; (iii) the loop transmits bias as efficiently as capability.”
Influential
- Authors
- Pu, S. X., Weng, Z., Liu, C., Srinivasa, J., Liu, G., Wang, W. Y., & Wang, X. E.
- Affiliations
- University of California, Santa Barbara (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2605.22217
- OpenAlex
- W7162075892
- Cited by
- 0
- References
- 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Keywords
- self-play reinforcement learning, model collapse, training stability, phase transitions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
