CoChat
Sign Up
PreprintCited by 1Topic ModelingDomain Adaptation and Few-Shot LearningMachine Learning and Algorithms

Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Models

Open access
Fu, Shi, Wang, Yingjie, Hu, Shengchao, Wang, Peng, Tao, Dacheng·2026·arXiv
DOI 10.48550/arxiv.2601.22513arXiv 2601.22513
Open this paper in CoChat — free accountOpen PDF

Abstract

Self-Rewarding Language Models (SRLMs) achieve notable success in iteratively improving alignment without external feedback. Yet, despite their striking empirical progress, the core mechanisms driving their capabilities remain unelucidated, leaving a critical gap in theoretical understanding. This paper provides the first rigorous theoretical guarantees for SRLMs. We first establish a lower bound that characterizes the fundamental limits of a single update step, revealing a critical dependence on the quality of the initial model. We then derive finite-sample error bounds for the full iterative paradigm, showing that performance improves at a rate of $\widetilde{\mathcal{O}}\left(1/\sqrt{n}\right)$ with sample size $n$. Crucially, our analysis reveals that the dependence on the initial model decays exponentially with the number of iterations $T$. This provides a formal explanation for why self-rewarding succeeds: it robustly overcomes poor initialization by steering the dynamics toward internal stability and consistency. Finally, we instantiate our theoretical framework for the linear softmax model class, yielding tailored guarantees that connect our high-level insights to practical model architectures.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2601.22513

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Theoretical guarantees for self-rewarding language models have begun to appear, characterizing when iterative self-alignment provably improves.

    “theoretical guarantees for self-rewarding language models have begun to appear, characterizing when iterative self-alignment provably improves [190]”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [190]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 66

  1. Agarwal, S., Zhang, Z., Yuan, L., Han, J., and Peng, H. The unreasonable effectiveness of entropy minimization in llm reasoning. arXiv preprint arXiv:2505.15134 , 2025. · arXiv
  2. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. Self-consuming generative models go mad. In The Twelfth International Conference on Learning Representations , 2024.
  3. Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations , 2023.
  4. Aminian, G., Abroshan, M., Khalili, M. M., Toni, L., and Rodrigues, M. An information-theoretical approach to semi-supervised learning under covariate-shift. In International Conference on Artificial Intelligence and Statistics , pp. 7433-7449. PMLR, 2022a.
  5. Aminian, G., Behnamnia, A., Vega, R., Toni, L., Shi, C., Rabiee, H. R., Rivasplata, O., and Rodrigues, M. R. Semi-supervised batch learning from logged data. arXiv preprint arXiv:2209.07148 , 2022b. · arXiv
  6. Aminian, G., Bagheri, A., JafariNodeh, M., Karimian, R., and Yassaee, M.-H. Robust semi-supervised learning via f-divergence and α -r´ enyi divergence. In 2024 IEEE International Symposium on Information Theory (ISIT) , pp. 1842-1847. IEEE, 2024.
  7. Amortila, P., Foster, D. J., and Krishnamurthy, A. Scalable online exploration via coverability. In International Conference on Machine Learning , pp. 1392-1455. PMLR, 2024.
  8. Behnamnia, A., Aminian, G., Aghaei, A., Shi, C., Tan, V. Y., and Rabiee, H. R. Batch learning via log-sumexponential estimator from logged bandit feedback. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists , 2024.
  9. Behnamnia, A., Aminian, G., Aghaei, A., Shi, C., Tan, V. Y., and Rabiee, H. R. Log-sum-exponential estimator for off-policy evaluation and learning. In Forty-second International Conference on Machine Learning , 2025.
  10. Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., and Raffel, C. Remixmatch: Semisupervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 , 2019a. · arXiv
  11. Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems , 32, 2019b.
  12. Bertrand, Q., Bose, J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data. In The Twelfth International Conference on Learning Representations , 2024.
  13. Boix-Adsera, E. Towards a theory of model distillation. arXiv preprint arXiv:2403.09053 , 2024. · arXiv
  14. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning , 2024.
  15. Chen, H., Feng, Y., Liu, Z., Yao, W., Prabhakar, A., Heinecke, S., Ho, R., Mui, P., Savarese, S., Xiong, C., et al. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint arXiv:2411.04282 , 2024. · arXiv
  16. Das, R. and Sanghavi, S. Understanding self-distillation in the presence of label noise. In International Conference on Machine Learning , pp. 7102-7140. PMLR, 2023.
  17. Dohmatob, E., Feng, Y., and Kempe, J. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.07712 , 2024. · arXiv
  18. Dong, Y., Cordonnier, J.-B., and Loukas, A. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning , pp. 2793-2803. PMLR, 2021.
  19. Du, H., Li, B., Xie, C., Gao, C., Chen, K., and Tao, D. Confidence as a reward: Transforming llms into reward models. arXiv preprint arXiv:2510.13501 , 2025. · arXiv
  20. Ethayarajh, K. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.00512 , 2019. · arXiv
  21. Feng, Y., Dohmatob, E., Yang, P., Charton, F., and Kempe, J. Beyond model collapse: Scaling up with synthesized data requires verification. In The Thirteenth International Conference on Learning Representations , 2025.
  22. Frei, S., Zou, D., Chen, Z., and Gu, Q. Self-training converts weak learners to strong learners in mixture models. In International Conference on Artificial Intelligence and Statistics , pp. 8003-8021. PMLR, 2022.
  23. Fu, S., Zhang, S., Wang, Y., Tian, X., and Tao, D. Towards theoretical understandings of self-consuming generative models. In Forty-first International Conference on Machine Learning , 2024.
  24. Fu, S., Wang, Y., Chen, Y., Shen, L., and Tao, D. Selfverification provably prevents model collapse in recursive synthetic training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems , 2025a. URL https://openreview.net/forum? id=X5Hk8aMs6w .
  25. Fu, S., Wang, Y., Chen, Y., Tian, X., and Tao, D. A theoretical perspective: How to prevent model collapse in self-consuming training loops. In The Thirteenth International Conference on Learning Representations , 2025b.
  26. Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pp. 10835-10866. PMLR, 2023.
  27. Gao, Z., Chang, J., Zhan, W., Oertell, O., Swamy, G., Brantley, K., Joachims, T., Bagnell, D., Lee, J. D., and Sun, W. Rebel: Reinforcement learning via regressing relative rewards. Advances in Neural Information Processing Systems , 37:52354-52400, 2024.
  28. Grandvalet, Y. and Bengio, Y. Semi-supervised learning by entropy minimization. Advances in neural information processing systems , 17, 2004.
  29. Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. A survey on llm-as-ajudge. arXiv preprint arXiv:2411.15594 , 2024. · arXiv
  30. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. · arXiv
  31. Huang, A., Block, A., Foster, D. J., Rohatgi, D., Zhang, C., Simchowitz, M., Ash, J. T., and Krishnamurthy, A. Self-improvement in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representations , 2025.
  32. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610 , 2022. · arXiv
  33. Jain, A., Montanari, A., and Sasoglu, E. Scaling laws for learning with real and surrogate data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.
  34. Joachims, T., Swaminathan, A., and De Rijke, M. Deep learning with logged bandit feedback. In International Conference on Learning Representations , 2018.
  35. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , pp. 1207-1216, Stanford, CA, 2000. Morgan Kaufmann.
  36. Lee, D.-H. et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, pp. 896. Atlanta, 2013.
  37. Li, Z., Yu, W., Huang, C., Liu, R., Liang, Z., Liu, F., Che, J., Yu, D., Boyd-Graber, J., Mi, H., et al. Selfrewarding vision-language model via reasoning decomposition. arXiv preprint arXiv:2508.19652 , 2025. · arXiv
  38. Marchi, M., Soatto, S., Chaudhari, P., and Tabuada, P. Heat death of generative models in closed-loop learning. arXiv preprint arXiv:2404.02325 , 2024. · arXiv
  39. Mobahi, H., Farajtabar, M., and Bartlett, P. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems , 33:3351-3361, 2020.
  40. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.
  41. Pareek, D., Du, S. S., and Oh, S. Understanding the gains from repeated self-distillation. Advances in Neural Information Processing Systems , 37:7759-7796, 2024.
  42. Prasad, A., Yuan, W., Pang, R. Y., Xu, J., Fazel-Zarandi, M., Bansal, M., Sukhbaatar, S., Weston, J. E., and Yu, J. Self-consistency preference optimization. In Fortysecond International Conference on Machine Learning , 2025.
  43. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems , 36: 53728-53741, 2023.
  44. Saglam, B., Kassianik, P., Nelson, B., Weerawardhena, S., Singer, Y., and Karbasi, A. Large language models encode semantics in low-dimensional linear subspaces. arXiv preprint arXiv:2507.09709 , 2025. · arXiv
  45. Shalev-Shwartz, S. and Ben-David, S. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.
  46. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y. Ai models collapse when trained on recursively generated data. Nature , 631(8022):755-759, 2024.
  47. Simchowitz, M. and Jamieson, K. G. Non-asymptotic gapdependent regret bounds for tabular mdps. Advances in Neural Information Processing Systems , 32, 2019.
  48. Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.
  49. Suresh, A. T., Thangaraj, A., and Khandavally, A. N. K. Rate of model collapse in recursive training. arXiv preprint arXiv:2412.17646 , 2024. · arXiv
  50. Swaminathan, A. and Joachims, T. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research , 16(1):1731-1755, 2015.
  51. Vapnik, V. The nature of statistical learning theory . Springer, 2013.
  52. Velegkas, G., Yang, Z., and Karbasi, A. The best of both worlds: Reinforcement learning with logarithmic regret and policy switches. arXiv preprint arXiv:2203.01491 , 2022. · arXiv
  53. Wang, Z., He, W., Liang, Z., Zhang, X., Bansal, C., Wei, Y., Zhang, W., and Yao, H. Cream: Consistency regularized self-rewarding language models. In The Thirteenth International Conference on Learning Representations , 2025.
  54. Wataoka, K., Takahashi, T., and Ri, R. Self-preference bias in llm-as-a-judge. In Neurips Safe Generative AI Workshop 2024 , 2024.
  55. Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self-improving alignment with llm-as-a-metajudge. arXiv preprint arXiv:2407.19594 , 2024. · arXiv
  56. Xie, T., Foster, D. J., Bai, Y ., Jiang, N., and Kakade, S. M. The role of coverage in online reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.
  57. Xiong, W., Zhang, H., Ye, C., Chen, L., Jiang, N., and Zhang, T. Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502.19613 , 2025. · arXiv
  58. Yoon, E., Yoon, H. S., Jang, J., Eom, S., Dai, Q., Luo, C., Hasegawa-Johnson, M. A., and Yoo, C. D. Pacr: Progressively ascending confidence reward for llm reasoning. arXiv preprint arXiv:2510.22255 , 2025. · arXiv
  59. Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. E. Self-rewarding language models. In Forty-first International Conference on Machine Learning , 2024.
  60. Zhang, Q., Qiu, L., Hong, I., Xu, Z., Liu, T., Li, S., Zhang, R., Li, Z., Li, L., Yin, B., et al. Self-rewarding ppo: Aligning large language models with demonstrations only. arXiv preprint arXiv:2510.21090 , 2025a. · arXiv
  61. Zhang, Q., Wu, H., Zhang, C., Zhao, P., and Bian, Y. Right question is already half the answer: Fully unsupervised llm reasoning incentivization. arXiv preprint arXiv:2504.05812 , 2025b. · arXiv
  62. Zhang, Y., Zhang, F., Yang, Z., and Wang, Z. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.19420 , 2023. · arXiv
  63. Zhao, X., Kang, Z., Feng, A., Levine, S., and Song, D. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590 , 2025. · arXiv
  64. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems , 36: 46595-46623, 2023.
  65. Zhou, X., Guo, Y., Ma, R., Gui, T., Zhang, Q., and Huang, X. Self-consistency of the internal reward models improves self-rewarding language models. arXiv preprint arXiv:2502.08922 , 2025. · arXiv
  66. Zhou, Y., Fan, Z., Cheng, D., Yang, S., Chen, Z., Cui, C., Wang, X., Li, Y., Zhang, L., and Yao, H. Calibrated selfrewarding vision language models. Advances in Neural Information Processing Systems , 37:51503-51531, 2024.

Cited by

Sorted by citations

Authors
Fu, S., Wang, Y., Hu, S., Wang, P., & Tao, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2601.22513
arXiv
2601.22513
OpenAlex
W7127090068
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
iterative alignment, theoretical guarantees, lower bounds, finite-sample error bounds, language model alignment, self-supervised learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review