CoChat
Sign Up
PreprintCited by 10Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; its reference list is not available.

LLM Post-Training: A Deep Dive into Reasoning Large Language Models

Open access
Kumar, Komal, Ashraf, Tajamul, Thawakar, Omkar, Anwer, Rao Muhammad, Cholakkal, Hisham, Shah, Mubarak, Yang, Ming-Hsuan, Torr, Phillip H. S., Khan, Fahad Shahbaz, Khan, Salman·2025·arXiv
DOI 10.48550/arxiv.2502.21321arXiv 2502.21321
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications. Pretraining on vast web-scale data has laid the foundation for these models, yet the research community is now increasingly shifting focus toward post-training techniques to achieve further breakthroughs. While pretraining provides a broad linguistic foundation, post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations. Fine-tuning, reinforcement learning, and test-time scaling have emerged as critical strategies for optimizing LLMs performance, ensuring robustness, and improving adaptability across various real-world tasks. This survey provides a systematic exploration of post-training methodologies, analyzing their role in refining LLMs beyond pretraining, addressing key challenges such as catastrophic forgetting, reward hacking, and inference-time trade-offs. We highlight emerging directions in model alignment, scalable adaptation, and inference-time reasoning, and outline future research directions. We also provide a public repository to continually track developments in this fast-evolving field: https://github.com/mbzuai-oryx/Awesome-LLM-Post-training.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.21321

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 10

Research works printed but not linked: 277

  1. [1][1] A. Vaswani, 'Attention is all you need,' Advances in Neural Information Processing Systems , 2017. 1 -
  2. [2][2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. , 'Language models are few-shot learners,' Advances in neural information processing systems , vol. 33, pp. 1877-1901, 2020. 1 -
  3. [3][3] Z. Yang, 'Xlnet: Generalized autoregressive pretraining for language understanding,' arXiv preprint arXiv:1906.08237 , 2019. 1, 3, 19 - · arXiv
  4. [4][4] J. Devlin, 'Bert: Pre-training of deep bidirectional transformers for language understanding,' arXiv preprint arXiv:1810.04805 , 2018. 1, 2, 3 - · arXiv
  5. [5][5] Z. Lan, 'Albert: A lite bert for self-supervised learning of language representations,' arXiv preprint arXiv:1909.11942 , 2019. 1 - · arXiv
  6. [6][6] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, 'Exploring the limits of transfer learning with a unified text-to-text transformer,' Journal of machine learning research , vol. 21, no. 140, pp. 167, 2020. 1 -
  7. [7][7] P. Verga, S. Hofstatter, S. Althammer, Y. Su, A. Piktus, A. Arkhangorodsky, M. Xu, N. White, and P. Lewis, 'Replacing judges with juries: Evaluating llm generations with a panel of diverse models,' arXiv preprint arXiv:2404.18796 , 2024. 1 - · arXiv
  8. [8][8] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. , 'Chain-of-thought prompting elicits reasoning in large language models,' Advances in neural information processing systems , vol. 35, pp. 24824-24837, 2022. 1, 2, 3, 11, 12, 15, 18 -
  9. [9][9] C. Wang, Y. Deng, Z. Lyu, L. Zeng, J. He, S. Yan, and B. An, 'Q*: Improving multi-step reasoning for llms with deliberative planning,' arXiv preprint arXiv:2406.14283 , 2024. 1 - · arXiv
  10. [10][10] Y. Wu, X. Han, W. Song, M. Cheng, and F. Li, 'Mindmap: Constructing evidence chains for multi-step reasoning in large language models,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, pp. 19270-19278, 2024. 1 -
  11. [11][11] Y. Ding, Z. Wang, W. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, et al. , 'Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,' Advances in Neural Information Processing Systems , vol. 36, 2024. 1 -
  12. [12][12] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , 'Gpt-4 technical report,' arXiv preprint arXiv:2303.08774 , 2023. 1 - · arXiv
  13. [13][13] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. , 'The llama 3 herd of models,' arXiv preprint arXiv:2407.21783 , 2024. 1, 5 - · arXiv
  14. [14][14] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. , 'Gemma 2: Improving open language models at a practical size,' arXiv preprint arXiv:2408.00118 , 2024. 1, 5 - · arXiv
  15. [15][15] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. , 'Gemini: a family of highly capable multimodal models,' arXiv preprint arXiv:2312.11805 , 2023. 1, 5 - · arXiv
  16. [16][16] A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, et al. , 'Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,' arXiv preprint arXiv:2405.04434 , 2024. 1, 2, 5, 6 - · arXiv
  17. [17][17] M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, et al. , 'Phi3 technical report: A highly capable language model locally on your phone,' arXiv preprint arXiv:2404.14219 , 2024. 1, 5 - · arXiv
  18. [18][18] A. Fan, M. Lewis, and Y. Dauphin, 'Hierarchical neural story generation,' arXiv preprint arXiv:1805.04833 , 2018. 1 - · arXiv
  19. [19][19] C. Chhun, P. Colombo, C. Clavel, and F. M. Suchanek, 'Of human criteria and automatic metrics: A benchmark of the evaluation of story generation,' arXiv preprint arXiv:2208.11646 , 2022. 1 - · arXiv
  20. [20][20] S. Arif, S. Farid, A. H. Azeemi, A. Athar, and A. A. Raza, 'The fellowship of the llms: Multi-agent workflows for synthetic preference optimization dataset generation,' arXiv preprint arXiv:2408.08688 , 2024. 1 - · arXiv
  21. [23][23] J. Ren, Y. Zhao, T. Vu, P. J. Liu, and B. Lakshminarayanan, 'Self-evaluation improves selective generation in large language models,' in Proceedings on , pp. 49-64, PMLR, 2023. 1 -
  22. [24][24] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. , 'Retrieval-augmented generation for knowledgeintensive nlp tasks,' Advances in Neural Information Processing Systems , vol. 33, pp. 9459-9474, 2020. 1, 2 -
  23. [25][25] Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.-t. Yih, D. Fried, S. Wang, and T. Yu, 'Ds-1000: A natural and reliable benchmark for data science code generation,' in International Conference on Machine Learning , pp. 1831918345, PMLR, 2023. 1 -
  24. [26][26] B. Zhu, E. Frick, T. Wu, H. Zhu, K. Ganesan, W.-L. Chiang, J. Zhang, and J. Jiao, 'Starling-7b: Improving helpfulness and harmlessness with rlaif,' in First Conference on Language Modeling , 2024. 1, 5 -
  25. [27][27] D. Paul, M. Ismayilzada, M. Peyrard, B. Borges, A. Bosselut, R. West, and B. Faltings, 'Refiner: Reasoning feedback on intermediate representations,' arXiv preprint arXiv:2304.01904 , 2023. 1 - · arXiv
  26. [28][28] Y. Xie, K. Kawaguchi, Y. Zhao, J. X. Zhao, M.-Y. Kan, J. He, and M. Xie, 'Self-evaluation guided beam search for reasoning,' Advances in Neural Information Processing Systems , vol. 36, 2024. 1 -
  27. [29][29] H. Ye and H. T. Ng, 'Self-judge: Selective instruction following with alignment self-evaluation,' arXiv preprint arXiv:2409.00935 , 2024. 1, 20 - · arXiv
  28. [30][30] Z. Luo, H. Wu, D. Li, J. Ma, M. Kankanhalli, and J. Li, 'Videoautoarena: An automated arena for evaluating large multimodal models in video analysis through user simulation,' arXiv preprint arXiv:2411.13281 , 2024. 1 - · arXiv
  29. [31][31] S. Deng, W. Zhao, Y.-J. Li, K. Wan, D. Miranda, A. Kale, and Y. Tian, 'Efficient self-improvement in multimodal large language models: A model-level judge-free approach,' arXiv preprint arXiv:2411.17760 , 2024. 1, 2 - · arXiv
  30. [32][32] T. Xiong, X. Wang, D. Guo, Q. Ye, H. Fan, Q. Gu, H. Huang, and C. Li, 'Llava-critic: Learning to evaluate multimodal models,' arXiv preprint arXiv:2410.02712 , 2024. - · arXiv
  31. [33][33] D. Chen, R. Chen, S. Zhang, Y. Liu, Y. Wang, H. Zhou, Q. Zhang, Y. Wan, P. Zhou, and L. Sun, 'Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language bench- mark,' arXiv preprint arXiv:2402.04788 , 2024. 1 - · arXiv
  32. [34][34] T. Hagendorff, S. Fabi, and M. Kosinski, 'Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt,' Nature Computational Science , vol. 3, no. 10, pp. 833-838, 2023. 1 -
  33. [35][35] Q. Pan, Z. Ashktorab, M. Desmond, M. S. Cooper, J. Johnson, R. Nair, E. Daly, and W. Geyer, 'Human-centered design recommendations for llm-as-a-judge,' arXiv preprint arXiv:2407.03479 , 2024. 1, 20 - · arXiv
  34. [36][36] G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang, 'Humans or llms as the judge? a study on judgement biases,' arXiv preprint arXiv:2402.10669 , 2024. 1 - · arXiv
  35. [38][38] X. Wang, H. Kim, S. Rahman, K. Mitra, and Z. Miao, 'Humanllm collaborative annotation through effective verification of llm labels,' in Proceedings of the CHI Conference on Human Factors in Computing Systems , pp. 1-21, 2024. 1 -
  36. [39][39] R. OpenAI, 'Gpt-4 technical report. arxiv 2303.08774,' View in Article , vol. 2, no. 5, 2023. 1, 5, 12 - · arXiv
  37. [40][40] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. , 'DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning,' arXiv preprint arXiv:2501.12948 , 2025. 1, 5, 11, 12, 20 - · arXiv
  38. [41][41] M. T. Hicks, J. Humphries, and J. Slater, 'Chatgpt is bullshit,' Ethics and Information Technology , vol. 26, no. 2, pp. 1-10, 2024. 1 -
  39. [43][43] A. Bruno, P. L. Mazzeo, A. Chetouani, M. Tliba, and M. A. Kerkouri, 'Insights into classifying and mitigating llms' hallucinations,' arXiv preprint arXiv:2311.08117 , 2023. 1 - · arXiv
  40. [44][44] S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal, 'Detecting hallucinations in large language models using semantic entropy,' Nature , vol. 630, no. 8017, pp. 625-630, 2024. 1 -
  41. [45][45] F. Leiser, S. Eckhardt, V. Leuthe, M. Knaeble, A. Maedche, G. Schwabe, and A. Sunyaev, 'Hill: A hallucination identifier for large language models,' in Proceedings of the CHI Conference on Human Factors in Computing Systems , pp. 1-13, 2024. 1 -
  42. [46][46] A. Gunjal, J. Yin, and E. Bas, 'Detecting and preventing hallucinations in large vision language models,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, pp. 1813518143, 2024. 1 -
  43. [47][47] S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu, 'Reasoning with language model is planning with world model,' in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 (H. Bouamor, J. Pino, and K. Bali, eds.), pp. 8154-8173, Association for Computational Linguistics, 2023. 1 -
  44. [50][50] T. Xia, B. Yu, Y. Wu, Y. Chang, and C. Zhou, 'Language models can evaluate themselves via probability discrepancy,' arXiv preprint arXiv:2405.10516 , 2024. 1 - · arXiv
  45. [51][51] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, 'Survey of hallucination in natural language generation,' ACM Computing Surveys , vol. 55, no. 12, pp. 1-38, 2023. 1, 3 -
  46. [53][53] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. , 'Constitutional ai: Harmlessness from ai feedback,' arXiv preprint arXiv:2212.08073 , 2022. 2, 20, 21 - · arXiv
  47. [55][55] E. Lobo, C. Agarwal, and H. Lakkaraju, 'On the impact of fine-tuning on chain-of-thought reasoning,' arXiv preprint arXiv:2411.15382 , 2024. 2, 3 - · arXiv
  48. [56][56] L. Trung, X. Zhang, Z. Jie, P. Sun, X. Jin, and H. Li, 'Reft: Reasoning with reinforced fine-tuning,' in Proceedings of the 62nd Annual Meeting of the Association for Computational
  49. [57][57] Linguistics (Volume 1: Long Papers) , pp. 7601-7614, 2024. 2 R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, 'Direct preference optimization: Your language model is secretly a reward model,' Advances in Neural Information Processing Systems , vol. 36, 2024. 2, 3, 6, 20 -
  50. [58][58] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. , 'Training language models to follow instructions with human feedback,' Advances in neural information processing systems , vol. 35, pp. 27730-27744, 2022. 2, 3, 5, 9, 13, 15 -
  51. [59][59] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. , 'Deepseekmath: Pushing the limits of mathematical reasoning in open language models,' arXiv preprint arXiv:2402.03300 , 2024. 2, 3, 6, 8, 10, 12, 22 - · arXiv
  52. [60][60] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, 'Lora: Low-rank adaptation of large language models,' arXiv preprint arXiv:2106.09685 , 2021. 2, 13, 20, 22 - · arXiv
  53. [61][61] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, 'Selfrag: Learning to retrieve, generate, and critique through selfreflection,' arXiv preprint arXiv:2310.11511 , 2023. 2 - · arXiv
  54. [62][62] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang, 'Retrieval-augmented generation for large language models: A survey,' arXiv preprint arXiv:2312.10997 , 2023. 2 - · arXiv
  55. [63][63] Z. Hu, L. Song, J. Zhang, Z. Xiao, J. Wang, Z. Chen, and H. Xiong, 'Rethinking llm-based preference evaluation,' arXiv preprint arXiv:2407.01085 , 2024. 2 - · arXiv
  56. [64][64] S. Yue, W. Chen, S. Wang, B. Li, C. Shen, S. Liu, Y. Zhou, Y. Xiao, S. Yun, X. Huang, et al. , 'Disc-lawllm: Fine-tuning large language models for intelligent legal services,' arXiv preprint arXiv:2309.11325 , 2023. 2 - · arXiv
  57. [65][65] Y. Luo, Z. Yang, F. Meng, Y. Li, J. Zhou, and Y. Zhang, 'An empirical study of catastrophic forgetting in large language models during continual fine-tuning,' arXiv preprint arXiv:2308.08747 , 2023. 2 - · arXiv
  58. [68][68] W. Zhang, Y. Deng, B. Liu, S. J. Pan, and L. Bing, 'Sentiment analysis in the era of large language models: A reality check,' arXiv preprint arXiv:2305.15005 , 2023. 2 - · arXiv
  59. [69][69] A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, 'Learning word vectors for sentiment analysis,' in Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pp. 142-150, 2011. 2 -
  60. [70][70] Y. Wang, S. Wang, Y. Li, and D. Dou, 'Recognizing medical search query intent by few-shot learning,' in Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 502-512, 2022. 2 -
  61. [71][71] R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu, 'Biogpt: generative pre-trained transformer for biomedical text generation and mining,' Briefings in bioinformatics , vol. 23, no. 6, p. bbac409, 2022. 2, 13 -
  62. [72][72] O. Wysocki, M. Wysocka, D. Carvalho, A. T. Bogatu, D. M. Gusicuma, M. Delmas, H. Unsworth, and A. Freitas, 'An llmbased knowledge synthesis and scientific reasoning framework for biomedical discovery,' arXiv preprint arXiv:2406.18626 , 2024. 2 - · arXiv
  63. [73][73] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, 'Proximal policy optimization algorithms,' arXiv preprint arXiv:1707.06347 , 2017. 2, 5, 8, 10 - · arXiv
  64. [74][74] Y. Xie, A. Goyal, W. Zheng, M.-Y. Kan, T. P. Lillicrap, K. Kawaguchi, and M. Shieh, 'Monte carlo tree search boosts reasoning via iterative preference learning,' arXiv preprint arXiv:2405.00451 , 2024. 2, 18, 22 - · arXiv
  65. [75][75] M. Shanahan, K. McDonell, and L. Reynolds, 'Role play with large language models,' Nature , vol. 623, no. 7987, pp. 493-498, 2023. 2 -
  66. [76][76] T. Xu, E. Helenowski, K. A. Sankararaman, D. Jin, K. Peng, E. Han, S. Nie, C. Zhu, H. Zhang, W. Zhou, et al. , 'The perfect blend: Redefining rlhf with mixture of judges,' arXiv preprint arXiv:2409.20370 , 2024. 2 - · arXiv
  67. [78][78] Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto, 'Alpacafarm: A simulation framework for methods that learn from human feedback,' Advances in Neural Information Processing Systems , vol. 36, 2024. 2, 15 -
  68. [79][79] L. Shani, A. Rosenberg, A. Cassel, O. Lang, D. Calandriello, A. Zipori, H. Noga, O. Keller, B. Piot, I. Szpektor, et al. , 'Multi-turn reinforcement learning from preference human feedback,' arXiv preprint arXiv:2405.14655 , 2024. 2 - · arXiv
  69. [80][80] Z. Li, Y. He, L. He, J. Wang, T. Shi, B. Lei, Y. Li, and Q. Chen, 'Falcon: Feedback-driven adaptive long/shortterm memory reinforced coding optimization system,' arXiv preprint arXiv:2410.21349 , 2024. 2 - · arXiv
  70. [81][81] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. , 'Webgpt: Browser-assisted question-answering with human feedback,' arXiv preprint arXiv:2112.09332 , 2021. 2, 11, 14 - · arXiv
  71. [82][82] L. Gao, J. Schulman, and J. Hilton, 'Scaling laws for reward model overoptimization,' in International Conference on Machine Learning , pp. 10835-10866, PMLR, 2023. 2, 15, 20 -
  72. [83][83] C. Snell, J. Lee, K. Xu, and A. Kumar, 'Scaling llm test-time compute optimally can be more effective than scaling model parameters,' arXiv preprint arXiv:2408.03314 , 2024. 2, 15, 21 - · arXiv
  73. [84][84] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, 'Tree of thoughts: Deliberate problem solving with large language models,' Advances in Neural Information Processing Systems , vol. 36, 2024. 2, 14, 16, 20 -
  74. [85][85] J. Jiang, D. He, and J. Allan, 'Searching, browsing, and clicking in a search session: Changes in user behavior by task and over time,' in Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval , pp. 607-616, 2014. 2 -
  75. [86][86] Y. Xie, K. Kawaguchi, Y. Zhao, J. X. Zhao, M.-Y. Kan, J. He, and M. Xie, 'Self-evaluation guided beam search for reasoning,' Advances in Neural Information Processing Systems , vol. 36, 2024. 2 -
  76. [87][87] Y. Tian, B. Peng, L. Song, L. Jin, D. Yu, H. Mi, and D. Yu, 'Toward self-improvement of llms via imagination, searching, and criticizing,' arXiv preprint arXiv:2404.12253 , 2024. 2, 18 - · arXiv
  77. [88][88] K. Gandhi, D. H. J. Lee, G. Grand, M. Liu, W. Cheng, A. Sharma, and N. Goodman, 'Stream of search (SoS): Learning to search in language,' in First Conference on Language Modeling , 2024. 2 -
  78. [91][91] E. Davis, 'Testing gpt-4-o1-preview on math and science problems: A follow-up study,' arXiv preprint arXiv:2410.22340 , 2024. 2 - · arXiv
  79. [92][92] J. Y. Wang, N. Sukiennik, T. Li, W. Su, Q. Hao, J. Xu, Z. Huang, F. Xu, and Y. Li, 'A survey on human-centric llms,' arXiv preprint arXiv:2411.14491 , 2024. 3 - · arXiv
  80. [93][93] J. Wu, S. Yang, R. Zhan, Y. Yuan, L. S. Chao, and D. F. Wong, 'A survey on llm-generated text detection: Necessity, methods, and future directions,' Computational Linguistics , pp. 1-65, 2025. 3, 14, 22 -
  81. [94][94] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. , 'A survey on evaluation of large language models,' ACM Transactions on Intelligent Systems and Technology , vol. 15, no. 3, pp. 1-45, 2024. 3 -
  82. [96][96] Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, 'A survey on in-context learning,' arXiv preprint arXiv:2301.00234 , 2022. 3 - · arXiv
  83. [97][97] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. , 'A survey of large language models,' arXiv preprint arXiv:2303.18223 , 2023. 3, 22 - · arXiv
  84. [101][101] J. Liu, C. Wang, C. Y. Liu, L. Zeng, R. Yan, Y. Sun, Y. Liu, and Y. Zhou, 'Improving multi-step reasoning abilities of large language models with direct advantage policy optimization,' arXiv preprint arXiv:2412.18279 , 2024. 3 - · arXiv
  85. [102][102] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, 'Large language models are zero-shot reasoners,' Advances in neural information processing systems , vol. 35, pp. 2219922213, 2022. 3 -
  86. [103][103] X. Deng, Y. Su, A. Lees, Y. Wu, C. Yu, and H. Sun, 'Reasonbert: Pre-trained to reason with distant supervision,' arXiv preprint arXiv:2109.04912 , 2021. 3 - · arXiv
  87. [106][106] I. J. Myung, 'Tutorial on maximum likelihood estimation,' Journal of mathematical Psychology , vol. 47, no. 1, pp. 90-100, 2003. 3 -
  88. [107][107] Y. Shao, J. Mao, Y. Liu, W. Ma, K. Satoh, M. Zhang, and S. Ma, 'Bert-pli: Modeling paragraph-level interactions for legal case retrieval.,' in IJCAI , pp. 3501-3507, 2020. 3 -
  89. [108][108] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. , 'Emergent abilities of large language models,' arXiv preprint arXiv:2206.07682 , 2022. 3 - · arXiv
  90. [109][109] R. Bellman, 'A markovian decision process,' Journal of mathematics and mechanics , pp. 679-684, 1957. 3, 10 -
  91. [112][112] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, 'Let's verify step by step,' arXiv preprint arXiv:2305.20050 , 2023. 4 - · arXiv
  92. [113][113] G. Chen, M. Liao, C. Li, and K. Fan, 'Step-level value preference optimization for mathematical reasoning,' arXiv preprint arXiv:2406.10858 , 2024. 4 - · arXiv
  93. [114][114] K. Nguyen, H. Daumé III, and J. Boyd-Graber, 'Reinforcement learning for bandit neural machine translation with simulated human feedback,' arXiv preprint arXiv:1707.07402 , 2017. 4, 5 - · arXiv
  94. [115][115] G. Williams, 'Substantive and adjectival law,' in Learning the Law , pp. 19-23, Law Book Company, Limited, 1982. 4 -
  95. [117][117] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, 'Self-critical sequence training for image captioning,' in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 7008-7024, 2017. 4, 5 -
  96. [118][118] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, 'Bleu: a method for automatic evaluation of machine translation,' in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pp. 311-318, 2002. 4 -
  97. [122][122] R. Team, A. Ormazabal, C. Zheng, C. d. M. d'Autume, D. Yogatama, D. Fu, D. Ong, E. Chen, E. Lamprecht, H. Pham, et al. , 'Reka core, flash, and edge: A series of powerful multimodal language models,' arXiv preprint arXiv:2404.12387 , 2024. 5 - · arXiv
  98. [123][123] B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhattacharya, A. Brundyn, J. Casper, B. Catanzaro, S. Clay, J. Cohen, et al. , 'Nemotron-4 340b technical report,' arXiv preprint arXiv:2406.11704 , 2024. 5 - · arXiv
  99. [125][125] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. , 'Qwen2. 5-coder technical report,' arXiv preprint arXiv:2409.12186 , 2024. 5 - · arXiv
  100. [128][128] R. Teknium, J. Quesnelle, and C. Guang, 'Hermes 3 technical report,' arXiv preprint arXiv:2408.11857 , 2024. 5 - · arXiv
  101. [130][130] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. , 'Palm 2 technical report,' arXiv preprint arXiv:2305.10403 , 2023. 5 - · arXiv
  102. [131][131] Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu, et al. , 'Internlm2 technical report,' arXiv preprint arXiv:2403.17297 , 2024. 5 - · arXiv
  103. [137][137] Fujitsu and T. I. of Technology, 'Fugaku-llm: The largest cpuonly trained language model,' Fujitsu Research Press Release , May 2024. 5 -
  104. [142][142] T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao, et al. , 'Chatglm: A family of large language models from glm-130b to glm-4 all tools,' arXiv preprint arXiv:2406.12793 , 2024. 5 - · arXiv
  105. [144][144] O. Lieber, O. Sharir, B. Lenz, and Y. Shoham, 'Jurassic-1: Technical details and evaluation,' White Paper. AI21 Labs , vol. 1, no. 9, pp. 1-17, 2021. 5 -
  106. [145][145] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. , 'Kimi k1. 5: Scaling reinforcement learning with llms,' arXiv preprint arXiv:2501.12599 , 2025. 5, 22 - · arXiv
  107. [147][147] C. Team, 'Chameleon: Mixed-modal early-fusion foundation models,' arXiv preprint arXiv:2405.09818 , 2024. 5 - · arXiv
  108. [152][152] V. Konda and J. Tsitsiklis, 'Actor-critic algorithms,' Advances in neural information processing systems , vol. 12, 1999. 4 -
  109. [153][153] S. Bhatnagar, M. Ghavamzadeh, M. Lee, and R. S. Sutton, 'Incremental natural actor-critic algorithms,' Advances in neural information processing systems , vol. 20, 2007. 4 -
  110. [154][154] V. Mnih, 'Asynchronous methods for deep reinforcement learning,' arXiv preprint arXiv:1602.01783 , 2016. 4, 5 - · arXiv
  111. [155][155] M. Babaeizadeh, I. Frosio, S. Tyree, J. Clemons, and J. Kautz, 'Reinforcement learning through asynchronous advantage actor-critic on a gpu,' arXiv preprint arXiv:1611.06256 , 2016. 4, 5 - · arXiv
  112. [156][156] S. M. Kakade, 'A natural policy gradient,' Advances in neural information processing systems , vol. 14, 2001. 4 -
  113. [157][157] A. G. Barto, R. S. Sutton, and C. W. Anderson, 'Neuronlike adaptive elements that can solve difficult learning control problems,' IEEE transactions on systems, man, and cybernetics , no. 5, pp. 834-846, 1983. 5 -
  114. [159][159] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. , 'Qwen2. 5 technical report,' arXiv preprint arXiv:2412.15115 , 2024. 5, 6 - · arXiv
  115. [161][161] N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist, 'Leverage the average: an analysis of kl regularization in reinforcement learning,' Advances in Neural Information Processing Systems , vol. 33, pp. 12163-12174, 2020. 6 -
  116. [162][162] S. Guo, B. Zhang, T. Liu, T. Liu, M. Khalman, F. Llinares, A. Rame, T. Mesnard, Y. Zhao, B. Piot, et al. , 'Direct language model alignment from online ai feedback,' arXiv preprint arXiv:2402.04792 , 2024. 6, 9, 10 - · arXiv
  117. [163][163] C. An, M. Zhong, Z. Wu, Q. Zhu, X. Huang, and X. Qiu, 'Colo: A contrastive learning based re-ranking framework for one-stage summarization,' arXiv preprint arXiv:2209.14569 , 2022. 6 - · arXiv
  118. [164][164] R. A. Bradley and M. E. Terry, 'Rank analysis of incomplete block designs: I. the method of paired comparisons,' Biometrika , vol. 39, no. 3/4, pp. 324-345, 1952. 6 -
  119. [165][165] R. L. Plackett, 'The analysis of permutations,' Journal of the Royal Statistical Society Series C: Applied Statistics , vol. 24, no. 2, pp. 193-202, 1975. 6 -
  120. [166][166] A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar, 'Rewarding progress: Scaling automated process verifiers for llm reasoning,' arXiv preprint arXiv:2410.08146 , 2024. 7, 20 - · arXiv
  121. [167][167] N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. , 'Rewardbench: Evaluating reward models for language modeling,' arXiv preprint arXiv:2403.13787 , 2024. 7, 19, 20 - · arXiv
  122. [170][170] R. Rafailov, Y. Chittepu, R. Park, H. S. Sikchi, J. Hejna, B. Knox, C. Finn, and S. Niekum, 'Scaling laws for reward model overoptimization in direct alignment algorithms,' ArXiv , vol. abs/2406.02900, 2024. 9 - · arXiv
  123. [172][172] G. Mukobi, P. Chatain, S. Fong, R. Windesheim, G. Kutyniok, K. Bhatia, and S. Alberti, 'Superhf: Supervised iterative learning from human feedback,' arXiv preprint arXiv:2310.16763 , 2023. 11 - · arXiv
  124. [173][173] C. Li, H. Zhou, G. Glavaš, A. Korhonen, and I. Vulić, 'On task performance and model calibration with supervised and self-ensembled in-context learning,' arXiv preprint arXiv:2312.13772 , 2023. 11 - · arXiv
  125. [175][175] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, , and J. Wei, 'Challenging big-bench tasks and whether chain-ofthought can solve them,' arXiv preprint arXiv:2210.09261 , 2022. 11, 19 - · arXiv
  126. [176][176] X. Zhang, C. Du, T. Pang, Q. Liu, W. Gao, and M. Lin, 'Chain of preference optimization: Improving chain-of-thought reasoning in LLMs,' in The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. 11 -
  127. [177][177] Z. Zhang, A. Zhang, M. Li, and A. Smola, 'Automatic chain of thought prompting in large language models,' arXiv preprint arXiv:2210.03493 , 2022. 11 - · arXiv
  128. [178][178] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al. , 'Multitask prompted training enables zero-shot task generalization,' arXiv preprint arXiv:2110.08207 , 2021. 12, 22 - · arXiv
  129. [179][179] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, 'Finetuned language models are zero-shot learners,' arXiv preprint arXiv:2109.01652 , 2021. 12 - · arXiv
  130. [183][183] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al. , 'Lamda: Language models for dialog applications,' arXiv preprint arXiv:2201.08239 , 2022. 12 - · arXiv
  131. [184][184] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, 'Self-consistency improves chain of thought reasoning in language models,' in The Eleventh International Conference on Learning Representations , 2023. 12, 15, 20 -
  132. [185][185] L. C. Magister, J. Mallinson, J. Adamek, E. Malmi, and A. Severyn, 'Teaching small language models to reason,' arXiv preprint arXiv:2212.08410 , 2022. 12, 13 - · arXiv
  133. [188][188] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, 'Qlora: Efficient finetuning of quantized llms,' arXiv preprint arXiv:2305.14314 , 2023. 13, 14 - · arXiv
  134. [189][189] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, 'GPTQ: Accurate post-training compression for generative pretrained transformers,' arXiv preprint arXiv:2210.17323 , 2022. 13 - · arXiv
  135. [190][190] E. Frantar and D. Alistarh, 'SparseGPT: Massive language models can be accurately pruned in one-shot,' arXiv preprint arXiv:2301.00774 , 2023. 13, 14 - · arXiv
  136. [192][192] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, 'Llm.int8(): 8-bit matrix multiplication for transformers at scale,' arXiv preprint arXiv:2208.07339 , 2022. 13 - · arXiv
  137. [193][193] Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao, 'Adaptive budget allocation for parameterefficient fine-tuning,' in The Eleventh International Conference on Learning Representations , 2023. 13, 20 -
  138. [194][194] X. Liu, K. Ji, Y. Fu, Z. Du, Z. Yang, and J. Tang, 'P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,' CoRR , vol. abs/2110.07602, 2021. 13 - · arXiv
  139. [195][195] Q. Lhoest, A. Villanova del Moral, Y. Jernite, A. Thakur, P. von Platen, S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. Šaško, G. Chhablani, B. Malik, S. Brandeis, T. Le Scao, V. Sanh, C. Xu, N. Patry, A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T. Matussière, L. Debut, S. Bekman, P. Cistac, T. Goehringer, V. Mustar, F. Lagunas, A. Rush, and T. Wolf, 'Datasets: A community library for natural language processing,' in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , (Online and Punta Cana, Dominican Republic), pp. 175-184, Association for Computational Linguistics, Nov. 2021. 13 -
  140. [207][207] Y. Zhou and K. Yang, 'Exploring tensorrt to improve realtime inference for deep learning,' in 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys) , pp. 2011-2018, 2022. 13 -
  141. [208][208] P. Tillet, H.-T. Kung, and D. Cox, 'Triton: an intermediate language and compiler for tiled neural network computations,' in Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages , pp. 10-19, 2019. 13 -
  142. [214][214] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, 'Automatic differentiation in pytorch,' in NIPS-W , 2017. 13 -
  143. [215][215] S. Hao, Y. Gu, H. Luo, T. Liu, X. Shao, X. Wang, S. Xie, H. Ma, A. Samavedhi, Q. Gao, et al. , 'Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models,' arXiv preprint arXiv:2404.05221 , 2024. 13 - · arXiv
  144. [216][216] S. Pieri, S. S. Mullappilly, F. S. Khan, R. M. Anwer, S. Khan, T. Baldwin, and H. Cholakkal, 'Bimedix: Bilingual medical mixture of experts llm,' arXiv preprint arXiv:2402.13253 , 2024. 13 - · arXiv
  145. [217][217] Y. Yang, M. C. S. Uy, and A. Huang, 'Finbert: A pretrained language model for financial communications,' arXiv preprint arXiv:2006.08097 , 2020. 13 - · arXiv
  146. [218][218] D. Thulke, Y. Gao, P. Pelser, R. Brune, R. Jalota, F. Fok, M. Ramos, I. van Wyk, A. Nasir, H. Goldstein, et al. , 'Climategpt: Towards ai synthesizing interdisciplinary research on climate change,' arXiv preprint arXiv:2401.09646 , 2024. 13 - · arXiv
  147. [219][219] S. S. Mullappilly, A. Shaker, O. Thawakar, H. Cholakkal, R. M. Anwer, S. Khan, and F. S. Khan, 'Arabic mini-climategpt: A climate change and sustainability tailored arabic llm,' arXiv preprint arXiv:2312.09366 , 2023. 13 - · arXiv
  148. [220][220] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, 'Codet5: Identifieraware unified pre-trained encoder-decoder models for code understanding and generation,' arXiv preprint arXiv:2109.00859 , 2021. 13 - · arXiv
  149. [221][221] K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan, 'Geochat: Grounded large vision-language model for remote sensing,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2783127840, 2024. 13 -
  150. [222][222] S. S. Mullappilly, M. I. Kurpath, S. Pieri, S. Y. Alseiari, S. Cholakkal, K. Aldahmani, F. Khan, R. Anwer, S. Khan, T. Baldwin, et al. , 'Bimedix2: Bio-medical expert lmm for diverse medical modalities,' arXiv preprint arXiv:2412.07769 , 2024. 13 - · arXiv
  151. [223][223] M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, 'Videochatgpt: Towards detailed video understanding via large vision and language models,' arXiv preprint arXiv:2306.05424 , 2023. 13 - · arXiv
  152. [224][224] B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan, 'Video-llava: Learning united visual representation by alignment before projection,' arXiv preprint arXiv:2311.10122 , 2023. 13 - · arXiv
  153. [225][225] H. Zhang, X. Li, and L. Bing, 'Video-llama: An instructiontuned audio-visual language model for video understanding,' arXiv preprint arXiv:2306.02858 , 2023. 13 - · arXiv
  154. [226][226] Y. Han, C. Zhang, X. Chen, X. Yang, Z. Wang, G. Yu, B. Fu, and H. Zhang, 'Chartllama: A multimodal llm for chart understanding and generation,' arXiv preprint arXiv:2311.16483 , 2023. 13 - · arXiv
  155. [227][227] X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang, 'A survey on model compression for large language models,' Transactions of the Association for Computational Linguistics , vol. 12, pp. 15561577, 2024. 13 -
  156. [228][228] Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, J. Liu, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, et al. , 'Efficient large language models: A survey,' arXiv preprint arXiv:2312.03863 , 2023. 13 - · arXiv
  157. [229][229] C.-Y. Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C.-Y. Lee, and T. Pfister, 'Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes,' arXiv preprint arXiv:2305.02301 , 2023. 13 - · arXiv
  158. [230][230] Y. Gu, L. Dong, F. Wei, and M. Huang, 'Minillm: Knowledge distillation of large language models,' arXiv preprint arXiv:2306.08543 , 2023. 13 - · arXiv
  159. [231][231] X. L. Li and P. Liang, 'Prefix-tuning: Optimizing continuous prompts for generation,' arXiv preprint arXiv:2101.00190 , 2021. 13 - · arXiv
  160. [233][233] B. P. Lowerre and B. R. Reddy, 'Harpy, a connected speech recognition system,' The Journal of the Acoustical Society of America , vol. 59, no. S1, pp. S97-S97, 1976. 14 -
  161. [234][234] A. Graves, 'Sequence transduction with recurrent neural networks,' arXiv preprint arXiv:1211.3711 , 2012. 14 - · arXiv
  162. [236][236] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. , 'A general language assistant as a laboratory for alignment,' arXiv preprint arXiv:2112.00861 , 2021. 14 - · arXiv
  163. [237][237] A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, et al. , 'Improving alignment of dialogue agents via targeted human judgements,' arXiv preprint arXiv:2209.14375 , 2022. 14 - · arXiv
  164. [238][238] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, 'Learning to summarize with human feedback,' Advances in Neural Information Processing Systems , vol. 33, pp. 3008-3021, 2020. 14 -
  165. [239][239] J. Q. Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami, 'Asymptotics of language model alignment,' arXiv preprint arXiv:2404.01730 , 2024. 15 - · arXiv
  166. [240][240] H. Sun, M. Haider, R. Zhang, H. Yang, J. Qiu, M. Yin, M. Wang, P. Bartlett, and A. Zanette, 'Fast best-of-n decoding via speculative rejection,' arXiv preprint arXiv:2410.20290 , 2024. 15 - · arXiv
  167. [242][242] L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.P. Lim, 'Plan-and-solve prompting: Improving zero-shot chainof-thought reasoning by large language models,' arXiv preprint arXiv:2305.04091 , 2023. 15 - · arXiv
  168. [243][243] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, 'Self-instruct: Aligning language models with self-generated instructions,' arXiv preprint arXiv:2212.10560 , 2022. 15 - · arXiv
  169. [244][244] X. Chen, R. Aksitov, U. Alon, J. Ren, K. Xiao, P. Yin, S. Prakash, C. Sutton, X. Wang, and D. Zhou, 'Universal selfconsistency for large language models,' in ICML 2024 Workshop on In-Context Learning . 15 -
  170. [246][246] F. Haji, M. Bethany, M. Tabar, J. Chiang, A. Rios, and P. Najafirad, 'Improving llm reasoning with multi-agent treeof-thought validator agent,' arXiv preprint arXiv:2409.11527 , 2024. 16 - · arXiv
  171. [247][247] M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler, 'Graph of thoughts: Solving elaborate problems with large language models,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, p. 17682-17690, Mar. 2024. 16 -
  172. [248][248] D. Wilson, 'Llm tree search,' arXiv preprint arXiv:2410.19117 , 2024. 17 - · arXiv
  173. [249][249] D. Hendrycks and K. Gimpel, 'A baseline for detecting misclassified and out-of-distribution examples in neural networks,' arXiv preprint arXiv:1610.02136 , 2016. 17 - · arXiv
  174. [250][250] G. Portillo Wightman, A. DeLucia, and M. Dredze, 'Strength in numbers: Estimating confidence of large language models by prompt agreement,' in Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) , pp. 326-362, 2023. 17 -
  175. [251][251] J. Qi, H. Tang, and Z. Zhu, 'Verifierq: Enhancing llm test time compute with q-learning-based verifiers,' arXiv preprint arXiv:2410.08048 , 2024. 17 - · arXiv
  176. [253][253] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al. , 'The rise and potential of large language model based agents: A survey,' arXiv preprint arXiv:2309.07864 , 2023. 17 - · arXiv
  177. [254][254] R. Coulom, 'Efficient selectivity and backup operators in monte-carlo tree search,' in International conference on computers and games , pp. 72-83, Springer, 2006. 18 -
  178. [255][255] J. X. Chen, 'The evolution of computing: Alphago,' Computing in Science & Engineering , vol. 18, no. 4, pp. 4-7, 2016. 18 -
  179. [256][256] L. Kocsis and C. Szepesvári, 'Bandit based monte-carlo planning,' in European conference on machine learning , pp. 282293, Springer, 2006. 18 -
  180. [257][257] H. W. Sprueill, C. Edwards, M. V. Olarte, U. Sanyal, H. Ji, and S. Choudhury, 'Monte carlo thought search: Large language model querying for complex scientific reasoning in catalyst design,' arXiv preprint arXiv:2310.14420 , 2023. 18, 20 - · arXiv
  181. [258][258] M. DeLorenzo, A. B. Chowdhury, V. Gohil, S. Thakur, R. Karri, S. Garg, and J. Rajendran, 'Make every move count: Llm-based high-quality rtl code generation using mcts,' arXiv preprint arXiv:2402.03289 , 2024. 18 - · arXiv
  182. [259][259] S. Park, X. Liu, Y. Gong, and E. Choi, 'Ensembling large language models with process reward-guided tree search for better complex reasoning,' arXiv preprint arXiv:2412.15797 , 2024. 18 - · arXiv
  183. [260][260] M. Shen, G. Zeng, Z. Qi, Z.-W. Hong, Z. Chen, W. Lu, G. Wornell, S. Das, D. Cox, and C. Gan, 'Satori: Reinforcement learning with chain-of-action-thought enhances llm reasoning via autoregressive search,' arXiv preprint arXiv:2502.02508 , 2025. 18 - · arXiv
  184. [262][262] U. Anwar, A. Saparov, J. Rando, D. Paleka, M. Turpin, P. Hase, E. S. Lubana, E. Jenner, S. Casper, O. Sourbut, et al. , 'Foundational challenges in assuring alignment and safety of large language models,' arXiv preprint arXiv:2404.09932 , 2024. 18 - · arXiv
  185. [263][263] W. Zhang, P. H. Torr, M. Elhoseiny, and A. Bibi, 'Bi-factorial preference optimization: Balancing safety-helpfulness in language models,' arXiv preprint arXiv:2408.15313 , 2024. 18 - · arXiv
  186. [264][264] C. Li, W. Wu, H. Zhang, Y. Xia, S. Mao, L. Dong, I. Vulić, and F. Wei, 'Imagine while reasoning in space: Multimodal visualization-of-thought,' arXiv preprint arXiv:2501.07542 , 2025. 19 - · arXiv
  187. [265][265] F. Nowak, A. Svete, A. Butoi, and R. Cotterell, 'On the representational capacity of neural language models with chainof-thought reasoning,' arXiv preprint arXiv:2406.14197 , 2024. 19 - · arXiv
  188. [266][266] Z. Li, H. Liu, D. Zhou, and T. Ma, 'Chain of thought empowers transformers to solve inherently serial problems,' arXiv preprint arXiv:2402.12875 , vol. 1, 2024. 19 - · arXiv
  189. [267][267] W. Merrill and A. Sabharwal, 'The expressive power of transformers with chain of thought,' arXiv preprint arXiv:2310.07923 , 2023. 19 - · arXiv
  190. [268][268] C. V. Snell, J. Lee, K. Xu, and A. Kumar, 'Scaling testtime compute optimally can be more effective than scaling llm parameters,' in The Thirteenth International Conference on Learning Representations . 19 -
  191. [269][269] Saxton et al. , 'Analysing mathematical reasoning abilities of neural models,' arXiv:1904.01557 , 2019. 19 - · arXiv
  192. [270][270] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, 'Training verifiers to solve math word problems,' arXiv preprint arXiv:2110.14168 , 2021. 19 - · arXiv
  193. [271][271] L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu, 'Metamath: Bootstrap your own mathematical questions for large language models,' arXiv preprint arXiv:2309.12284 , 2023. 19 - · arXiv
  194. [272][272] Z. Xie, S. Thiem, J. Martin, E. Wainwright, S. Marmorstein, and P. Jansen, 'WorldTree v2: A corpus of science-domain structured explanations and inference patterns supporting multi-hop inference,' in Proceedings of the Twelfth Language Resources and Evaluation Conference (N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, eds.), (Marseille, France), pp. 5456-5473, European Language Resources Association, May 2020. 17, 19 -
  195. [273][273] F. Liu, E. Bugliarello, E. M. Ponti, S. Reddy, N. Collier, and D. Elliott, 'Visually grounded reasoning across languages and cultures,' in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , (Online and Punta Cana, Dominican Republic), pp. 10467-10485, Association for Computational Linguistics, Nov. 2021. 19 -
  196. [274][274] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen, 'Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,' in Proceedings of CVPR , 2024. 19 -
  197. [275][275] S. Lin, J. Hilton, and O. Evans, 'Truthfulqa: Measuring how models mimic human falsehoods,' arXiv preprint arXiv:2109.07958 , 2021. 19 - · arXiv
  198. [276][276] X. Yue et al. , 'Mammoth: Building math generalist models through hybrid instruction tuning,' arXiv preprint arXiv:2309.05653 , 2023. 19 - · arXiv
  199. [277][277] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, 'Measuring massive multitask language understanding,' Proceedings of the International Conference on Learning Representations (ICLR) , 2021. 19, 20 -
  200. [278][278] D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt, 'Aligning ai with shared human values,' Proceedings of the International Conference on Learning Representations (ICLR) , 2021. 19, 20 -

200 of 277 shown

Metadata

Authors
Kumar, K., Ashraf, T., Thawakar, O., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., Torr, P. H. S., Khan, F. S., & Khan, S.
Affiliations
  • Mohamed bin Zayed University of Artificial Intelligence (AE)
  • University of Central Florida (US)
  • University of California, Merced (US)
  • University of Oxford (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Volume
48
Issue
11
Pages
14968-14987
ISSN
0162-8828 (ISSN-L)
DOI
10.48550/arxiv.2502.21321
arXiv
2502.21321
OpenAlex
W4417305769
Cited by
10
References
0
Citations by year
2026: 8 · 2025: 2
FWCI
27.16
Type
journal article
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
large language models, post-training, fine-tuning, reinforcement learning, test-time scaling, catastrophic forgetting, reward hacking, model alignment, factual accuracy, LLM reasoning
MeSH terms
Large Language Models, Reinforcement Machine Learning, Humans, Natural Language Processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review