LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Open accessAbstract
Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMs and 34 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing multilingual benchmarks that extend the Python-only HumanEval/MBPP benchmark, such as MultiPL-E, are challenged by the number of languages and the data leaking problem.
“Some works focus on the multilingual scenarios (Wang et al., 2023; Athiwaratkun et al., 2023; Peng et al., 2024; Zheng et al., 2023b) by extending the Python-only HumanEval/MBPP benchmark (e.g. MultiPL-E (Cassano et al., 2023)), which is challenged by the number of the languages and data leaking problem (Jain et al., 2024).”
From McEval: Massively Multilingual Code Evaluation (2024) · cites this paper as Jain et al., 2024
Section 6.1 says Python-focused code-generation benchmarks have variants covering more languages. The Introduction says HumanEval and MBPP may be subject to contamination because benchmark samples appear in training data. Our copy does not identify the number of languages as a challenge for multilingual variants or specifically establish data leakage in MultiPL-E. (LLM, conf=0.82)
Experimental — accuracy not yet measuredExisting multilingual code benchmarks such as MultiPL-E and MBXP are challenged by the number of covering languages and data leaking problem.
“such as MultiPL-E (Cassano et al., 2023) and MBXP (Athiwaratkun et al., 2023), which is challenged by the number of the covering languages and data leaking problem (Li et al., 2023; Jain et al., 2024).”
From McEval: Massively Multilingual Code Evaluation (2024) · cites this paper as Jain et al., 2024
The Introduction (page unavailable) says existing code benchmarks may suffer contamination or overfitting because samples appear in training data. It names HumanEval, MBPP, and APPS in that discussion, not MultiPL-E or MBXP. The cited paper does not establish that those two multilingual benchmarks have limited language coverage or data leakage. (LLM, conf=0.82)
Experimental — accuracy not yet measuredMulti-LCB extends LCB to twelve programming languages while preserving LCB's contamination controls and evaluation protocol.
“We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol.”
From Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages (2026) · cites this paper as Jain et al., 2024
The paper’s “Focus on Python” section says LCB evaluates Python and could be extended to other languages. Its Introduction describes contamination controls based on release dates and model cutoff dates. The paper does not describe Multi-LCB, its twelve-language implementation, or whether that implementation preserves LCB’s controls and evaluation protocol. (LLM, conf=0.85)
Experimental — accuracy not yet measuredEarlier methods evaluate code quality by generating test cases, but this suffers from inaccurate test generation, inability to handle special judges, and failure to assess non-code parts of responses.
“Previous works (Chen et al., 2023;Zhang et al., 2024a) rely on generating test cases to evaluate code quality.However, this method has issues with inaccurate test case generation (Jain et al., 2024), inability to handle special judge (Quan et al., 2025), and failure to evaluate non-code content in response such as code comments and explanations.”
From RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation (2025) · cites this paper as Jain et al., 2024
The Jain et al. citation is attached to the concern about inaccurate test-case generation, not to the separate claims about special judges or non-code content. Section 3 describes insufficient tests in existing benchmarks, and Appendix A.2 says randomly sampled inputs may miss corner cases. These passages support a concern about test coverage, but do not establish that generated test cases are inaccurate. (LLM, conf=0.75)
Experimental — accuracy not yet measuredIn repository-level software engineering, an agent may pass visible tests while exploiting weak or incomplete test suites; in GUI/OS tasks, a scripted checker may miss unsafe or undesirable intermediate actions; and in scientific or embodied settings, successful execution in a simulator may not imply scientific validity or physical safety.
“This is especially visible in repository-level software engineering, where an agent may pass visible tests while exploiting weak or incomplete test suites; in GUI/OS tasks, where a scripted checker may miss unsafe or undesirable intermediate actions; and in scientific or embodied settings, where successful execution in a simulator may not imply that the result is scientifically valid or physically”
From Code as Agent Harness (2026) · cites this paper as [133]
Split into 3 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
DevBench, CrossCodeBench, LiveCodeBench, KernelBench, and NL2Repo each advance scope but none separate proxy from true objective.
“DevBench (Li et al., 2025), CrossCodeBench (Niu et al., 2023), LiveCodeBench (Jain et al., 2024), KernelBench (Ouyang et al., 2025), and NL2Repo (Ding et al., 2025) each advance scope but none separate proxy from true objective.”
From SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (2026) · cites this paper as Jain et al., 2024
The Introduction and §6.1 describe LiveCodeBench as broadening code evaluation beyond generation to repair, execution, and test-output prediction, supporting the claim that it advances scope. The paper’s “Robustness to Prompts” discussion cautions that competition problems may not represent general programming ability, but it does not establish that LiveCodeBench fails to separate a proxy measure from a true objective. The other named benchmarks are not assessed by this citation. (LLM, conf=0.83)
Experimental — accuracy not yet measuredWe re-use the evaluation codebase maintained by Jain et al. [31], which wraps the candidate function in a test module, executes given the inputs, and checks equivalence of output with expected output, with a default timeout set to prevent the evaluator from getting trapped during evaluation.
“We re-use the evaluation codebase maintained by Jain et al. [31], which (1) wraps the candidate function in a test module, (2) execute given the inputs, and (3) checks the equivalence of the output from the expected output, with a default timeout set to prevent the evaluator from getting trapped during evaluation ( e.g. , brute-force implementation may not pass under the set time budget).”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as Jain et al. [31]
Section 4 (Experiment Setup; page unavailable) says generated programs are checked against tests and must pass all of them. Section 4.1 (Scenario-specific setup; page unavailable) includes time-limit errors among the evaluation feedback. Our copy does not describe wrapping candidate functions in a test module or specify a default timeout intended to prevent the evaluator from getting stuck. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLCB provides contamination-aware evaluation by continuously curating competitive-programming problems and filtering them by release date.
“LiveCodeBench (LCB) (Jain et al., 2024) addresses these shortcomings by continuously curating competitive-programming problems, filtering them by release date, and enabling contamination-aware, continuously updatable evaluation”
From Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages (2026) · cites this paper as Jain et al., 2024
LiveCodeBench (LCB) evaluates only Python.
“Despite these strengths, LCB (Jain et al., 2024) evaluates only Python.”
From Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages (2026) · cites this paper as Jain et al., 2024
Multi-LCB reuses the entire LCB task pool and inherits its contamination controls.
“Multi-LCB builds directly on this foundation. It reuses the entire LCB (Jain et al., 2024) task pool and inherits its contamination controls.”
From Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages (2026) · cites this paper as Jain et al., 2024
LiveCodeBench provides a continuously updated evaluation for code capabilities.
“LiveCodeBench provides a continuously updated evaluation for code capabilities [11]; SWE-bench turns real GitHub issues into executable repository-level tasks [12].”
From HELIX: Model-Harness Co-evolution for Recursive Self-Improvement (2026) · cites this paper as [11]
Recent works have proposed novel benchmarks to address contamination.
“Recent works have proposed novel benchmarks to address contamination (Li et al., 2024b; Zhu et al., 2023a; Jain et al., 2024; Li et al., 2024a; Qian et al., 2024; Wu et al., 2024).”
From The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (2025) · cites this paper as Jain et al., 2024
LiveCodeBench introduces continuously updated evaluation pipelines that assess execution, self-repair, and runtime reasoning capabilities under evolving problem distributions.
“LiveCodeBench [133] introduces continuously updated evaluation pipelines that assess execution, self-repair, and runtime reasoning capabilities under evolving problem distributions.”
From Code as Agent Harness (2026) · cites this paper as [133]
On APPS Introductory and LiveCodeBench v6, JEPA-Anything-guided candidate selection achieved final full-test success rates of 46.37% and 33.05%, respectively, exceeding random selection and the static verifier.
“On APPS Introductory [17] and LiveCodeBench v6 [18], JEPA-Anything-guided candidate selection achieved final full-test success rates of 46 . 37% and 33 . 05%, respectively, exceeding random selection and the static verifier (Figure 3e,f).”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [18]
Similar trends are observed in the test-output-prediction subtask of LiveCodeBench.
“We again observe similar trends in the test-output-prediction subtask of LiveCodeBench (Jain et al., 2024).”
From Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies (2025) · cites this paper as Jain et al., 2024
Both context and semantic caching lack flexibility for handling slight variations in input.
“Both context and semantic caching lack flexibility for handling slight variations in input.”
From Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents (2025) · cites this paper as [26]
The Introduction discusses exact and fuzzy matching for benchmark decontamination, while “Robustness to Prompts” discusses performance variation from prompts; neither addresses context or semantic caching. Searches for those caching terms and for flexibility with slight input variations found no passage supporting the claim. Page numbers were not available in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Part of this paper’s reference list could not be split into single entries.
Research works printed but not linked: 89
- Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. 2019. Juice: A large scale distantly supervised dataset for open domain context-based code generation. arXiv preprint arXiv:1910.02216. (Cited on pg. 15 · arXiv
- Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988. (Cited on pg. 2, 15) · arXiv
- Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400. (Cited on pg. 16 · arXiv
- Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868. (Cited on pg. 15 · arXiv
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. (Cited on pg. 2, 4, 15) · arXiv
- Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275. (Cited on pg. · arXiv
- Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. (Cited on pg. 15 · arXiv
- Barry Boehm. 2006. A view of 20th and 21st century software engineering. In Proceedings of the 28th International Conference on Software Engineering, ICSE '06, page 12-29, New York, NY, USA. Association for Computing Machinery. (Cited on pg. 5 · DOI
- Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227. (Cited on pg. · arXiv
- Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397. (Cited on pg. 6, 16) · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. (Cited on pg. 2, 9, 15) · arXiv
- Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. (Cited on pg. 5, 15) · arXiv
- Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallap- ati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. Cocomic: Code completion by jointly modeling in-file and cross-file context. arXiv preprint arXiv:2212.10007. (Cited on pg. 15 · arXiv
- Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. preprint arXiv:2204.05999. (Cited on pg. · arXiv
- A Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. (Cited on pg. 15 · arXiv
- Shahriar Golchin and Mihai Surdeanu. 2023. Time travel in llms: Tracing data contamination in large language models. arXiv preprint arXiv:2308.08493. (Cited on pg. 16 · arXiv
- Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065. (Cited on pg. 6, 9, 15, 34) · arXiv
- Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196. (Cited on pg. 2, 5, 10) · arXiv
- Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md Mahim Anjum Haque, Tahmid Hasan, Wasi Uddin Ahmad, Anindya Iqbal, and Rifat Shahriyar. 2021. Codesc: A large code-description parallel dataset. arXiv preprint arXiv:2105.14220. (Cited on pg. 16 · arXiv
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. (Cited on pg. 9, 10, 15, 26) · arXiv
- Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, et al. 2023. Competition-level problems are effective llm evaluators. arXiv preprint arXiv:2312.02143. (Cited on pg. · arXiv
- Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016, pages 2073-2083. Association for Computational Linguistics. (Cited on pg. 16
- Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. 2022. Jigsaw: Large language models meet program synthesis. In Proceedings of the 44th International Conference on Software Engineering, pages 1219-1231. (Cited on pg. 15
- Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E Gonzalez, Koushik Sen, and Ion Sto- ica. 2023. Llm-assisted code cleaning for training accurate code generators. arXiv preprint arXiv:2311.14904. (Cited on pg. 15 · arXiv
- Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770. (Cited on pg. 15 · arXiv
- Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2023. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint arXiv:2303.03004. (Cited on pg. 15 · arXiv
- Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32. (Cited on pg. 9
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. (Cited on pg. 9)
- Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319- 18345. PMLR. (Cited on pg.
- Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019. A neural model for generating natural language summaries of program subroutines. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 795-806. IEEE. (Cited on pg. 16
- Jierui Li, Szymon Tworkowski, Yingying Wu, and Raymond Mooney. 2023a. Explaining competitive-level programming solutions using llms. arXiv preprint arXiv:2307.05337. (Cited on pg. · arXiv
- Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023b. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. (Cited on pg. 2, 15) · arXiv
- Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023c. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852. (Cited on pg. 5, 15) · arXiv
- Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023d. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463. (Cited on pg. · arXiv
- Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092-1097. (Cited on pg. 2, 15)
- Pietro Liguori, Erfan Al-Hossami, Domenico Cotroneo, Roberto Natella, Bojan Cukic, and Samira Shaikh. 2022. Can we generate shellcodes via natural language? an empirical study. Automated Software Engineering, 29(1):30. (Cited on pg. 16
- Changshu Liu, Shizhuo Dylan Zhang, and Reyhaneh Jabbarvand. 2024. Codemind: A framework to challenge large language models for code reasoning. arXiv preprint arXiv:2402.09664. (Cited on pg. · arXiv
- Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023a. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439. (Cited on pg. · arXiv
- Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023b. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210. (Cited on pg. 7, 15) · arXiv
- Tianyang Liu, Canwen Xu, and Julian McAuley. 2023c. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091. (Cited on pg. 15 · arXiv
- Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, et al. 2024. Repoagent: An llm-powered open-source framework for repository-level code documentation generation. arXiv preprint arXiv:2402.16667. (Cited on pg. · arXiv
- Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. (Cited on pg. 2, 15) · arXiv
- Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. 2023a. Learning performance-improving code edits. arXiv preprint arXiv:2302.07867. (Cited on pg. 2 · arXiv
- Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023b. Self-refine: Iterative re- finement with self-feedback. arXiv preprint arXiv:2303.17651. (Cited on pg. 15 · arXiv
- Rabee Sohail Malik, Jibesh Patra, and Michael Pradel. 2019. Nl2type: inferring javascript function types from natural language information. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 304-315. IEEE. (Cited on pg. 16
- Amir M Mir, Evaldas Latoškinas, Sebastian Proksch, and Georgios Gousios. 2022. Type4py: Prac- tical deep similarity learning-based type inference for python. In Proceedings of the 44th Inter- national Conference on Software Engineering, pages 2241-2252. (Cited on pg. 16
- Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, et al. 2023. L2ceval: Evaluating language-to-code generation capabilities of large language models. arXiv preprint arXiv:2309.17446. (Cited on pg. 15 · arXiv
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations. (Cited on pg.
- Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114. (Cited on pg. 15 · arXiv
- Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. 2023. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306.09896. (Cited on pg. 2, 5, 10, 15) · arXiv
- R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article. (Cited on pg. · arXiv
- Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. 2024. Proving test set contamination for black-box language models. In The Twelfth International Conference on Learning Representations. (Cited on pg. 16
- Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334. (Cited on pg. 2, 15) · arXiv
- Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot's code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754-768. IEEE. (Cited on pg. 16
- Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Im- proving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813. (Cited on pg. 15 · arXiv
- Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations. (Cited on pg. 2)
- Tal Ridnik, Dedy Kredo, and Itamar Friedman. 2024. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500. (Cited on pg. 2, 5, 15) · arXiv
- Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. 2024. To the cutoff... and beyond? a longitudinal perspective on LLM data contamination. In The Twelfth International Conference on Learning Representations. (Cited on pg. 16
- Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. (Cited on pg. 2, 15) · arXiv
- Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An empirical evaluation of us- ing large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85-105. (Cited on pg. · DOI · Paper page
- Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789. (Cited on pg. 16 · arXiv
- Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems. (Cited on pg. 5, 15)
- Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. In International Conference on Machine Learning, pages 31693-31715. PMLR. (Cited on pg.
- Manav Singhal, Tushar Aggarwal, Abhijeet Awasthi, Nagarajan Natarajan, and Aditya Kanade. 2024. Nofuneval: Funny how code lms falter on requirements beyond functional correctness. arXiv preprint arXiv:2401.15963. (Cited on pg. 5 · arXiv
- Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. Rein- forcement learning from automatic feedback for high-quality unit test generation. arXiv preprint arXiv:2310.02368. (Cited on pg. · arXiv
- Ruoxi Sun, Sercan O Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. 2023. Sql-palm: Improved large language modeladaptation for text-to-sql. arXiv preprint arXiv:2306.00739. (Cited on pg. 2) · arXiv
- Catherine Tony, Markus Mutas, Nicolás E Díaz Ferreyra, and Riccardo Scandariato. 2023. Llm- seceval: A dataset of natural language prompts for security evaluations. arXiv preprint arXiv:2303.09384. (Cited on pg. 16 · arXiv
- Michele Tufano, Shao Kun Deng, Neel Sundaresan, and Alexey Svyatkovskiy. 2022. Methods2test: A dataset of focal methods mapped to test cases. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 299-303. (Cited on pg. 16
- Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. 2022a. Recode: Robustness evaluation of code generation models. arXiv preprint arXiv:2212.10264. (Cited on pg. · arXiv
- Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922. (Cited on pg. · arXiv
- Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696-8708. (Cited on pg. 15
- Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. 2022b. Execution-based evaluation for open-domain code generation. arXiv preprint arXiv:2212.10481. (Cited on pg. 15 · arXiv
- Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 1398-1409. (Cited on pg. 16
- Jiayi Wei, Greg Durrett, and Isil Dillig. 2023a. Typet5: Seq2seq type inference using static analysis. arXiv preprint arXiv:2303.09564. (Cited on pg. 16 · arXiv
- Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023b. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120. (Cited on pg. 2, 15) · arXiv
- Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2023. " according to..." prompting language models improves quoting from pre- training data. arXiv preprint arXiv:2305.13252. (Cited on pg. 16 · arXiv
- Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weis- han Zhao, Li Zhu, Shuiguang Deng, et al. 2023. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation. arXiv preprint arXiv:2311.08588. (Cited on pg. 15 · arXiv
- Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H Chi, and Denny Zhou. 2023. Large language models as analogical reasoners. arXiv preprint arXiv:2310.01714. (Cited on pg. 15 · arXiv
- Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua How- land, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. 2022. Natural language to code generation in interactive data science notebooks. arXiv preprint arXiv:2212.09248. (Cited on pg. · arXiv
- Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023. No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.04207. (Cited on pg. 16 · arXiv
- Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. 2022. Parsel: A unified natural language framework for algorithmic reasoning. arXiv preprint arXiv:2212.10561. (Cited on pg. 15 · arXiv
- Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023a. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570. (Cited on pg. 15 · arXiv
- Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. 2023b. Toolcoder: Teach code generation models to use apis with search tools. arXiv preprint arXiv:2305.04032. (Cited on pg. 15 · arXiv
- Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023c. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 769-787, Toronto, Canada. Association for Computational Linguistics. (Cited on pg. 15
- Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. 2023d. Algo: Syn- thesizing algorithmic programs with generated oracle verifiers. arXiv preprint arXiv:2305.14591. (Cited on pg. 15 · arXiv
- Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568. (Cited on pg. 15 · arXiv
- Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. https://arxiv.org/abs/2402.14658. (Cited on pg. · arXiv
- Maosheng Zhong, Gen Liu, Hongwei Li, Jiangling Kuang, Jinshan Zeng, and Mingwen Wang. 2022. Codegen-test: An automatic code generation model integrating program test information. arXiv preprint arXiv:2202.07612. (Cited on pg. · arXiv
- Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don't make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964. (Cited on pg. 16 · arXiv
Cited by
Sorted by citations
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 538 · see citing works
Self-citationDeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294 · see citing works
Self-citationYang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candes, E., & Hashimoto, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1025 · Cited by 76 · see citing works
Qwen2.5-Coder Technical Report
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12186 · Cited by 65 · see citing works
“5 (Bai et al., 2023a), and Code-Stral (team, 2024), have demonstrated superior performance in coding evaluations (Chen et al., 2021; Austin et al., 2021; Cassano et al., 2022; Jain et al., 2024).”
BackgroundSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31 · see citing works
Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.21631 · Cited by 27 · see citing works
“…(Team, 2025), (2) Reasoning: AIME-25 (AIME, 2025), HMMT-25 (HMMT, 2025), LiveBench (2024-11-25) (White et al., 2024), (3) Code: LiveCodeBench v6 (Jain et al., 2024), CFEval, OJBench (Wang et al., 2025c), (4) Alignment Tasks: IFEval (Zhou et al., 2023), Arena-Hard v2 (Li et al., 2024d) 1 ,…”
Agentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24 · see citing works
“This is done by providing the previously described repository structure and asking the LLM to produce a list of irrelevant folders that do not need to be further inspected or modified to resolve the issue.”
MethodAbsolute Zero: Reinforced Self-play Reasoning with Zero Data
Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Yue, Y., Lin, M., Wang, S., Wu, Q., Zheng, Z., et al. (2025) · arXiv · Cited by 13 · see citing works
Self-citationMuennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12 · see citing works
Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01939 · Cited by 14
“Here, we test whether DAPO, when trained on a math dataset and using only a small fraction of high-entropy tokens in the policy gradient loss, can still surpass vanilla DAPO on out-of-distribution test sets, such as LiveCodeBench (Jain et al., 2024).”
InfluentialSelf-citationLLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 10 · see citing works
“The original instructions were sourced from the HumanEval [10] and LiveCodeBench [31] datasets, two popular benchmarks used to evaluate LLM programming aptitude.”
Helping LLMs improve code generation using feedback from testing and static analysis
Dolcetti, G., Arceri, V., Iotti, E., Maffeis, S., Cortesi, A., & Zaffanella, E. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01009-5 · Cited by 8 · see citing works
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patwardhan, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.07095 · Cited by 7 · see citing works
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6 · see citing works
DafnyBench: A Benchmark for Formal Software Verification
Loughridge, C., Sun, Q., Ahrenbach, S., Cassano, F., Sun, C., Sheng, Y., Mudide, A., Misu, M. R. H., Amin, N., & Tegmark, M. (2024) · arXiv · doi:10.48550/arxiv.2406.08467 · Cited by 6 · see citing works
“…of code Unverified program synthesis APPS [17] 10,000 programs HumanEvalX [18, 19] 165 programs MBPP [20] 974 programs SWEBench [21] 2,294 programs LiveCodeBench [22] grows weekly Formal software verification Clover [9] 66 programs Dafny-synthesis [10] 153 programs 3 DafnyBench Dataset Construction”
MethodSWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Miserendino, S., Wang, M., Patwardhan, T., & Heidecke, J. (2025) · arXiv · doi:10.48550/arxiv.2502.12115 · Cited by 5 · see citing works
“…al., 2024; Liu et al., 2023), with benchmarks such as HumanEval (Chen et al., 2021), HumanEvalPack (Muennighoff et al., 2024), APPS, LiveCodeBench (Jain et al., 2024), NaturalCodeBench (Zhang et al., 2024), Big-CodeBench (Zhuo et al., 2024), MBPP, MathQA-Python (Austin et al., 2021), CodeXGlue…”
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., & Huang, G. (2025) · arXiv · doi:10.48550/arxiv.2504.13837 · Cited by 3 · see citing works
“For evaluation, Models are assessed on LiveCodeBench v5, comprising 880 problems that span from May 2023 to January 2025 (Jain et al., 2024), as well as HumanEval+ and MBPP+ (Liu et al., 2023).”
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3 · see citing works
Self-citationLarge Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3 · see citing works
VERINA: Benchmarking Verifiable Code Generation
Ye, Z., Yan, Z., He, J., Kasriel, T., Yang, K., & Song, D. (2025) · arXiv · doi:10.48550/arxiv.2505.23135 · Cited by 3 · see citing works
“It draws problems from various sources, including MBPP [18, 24], LiveCodeBench [1], and LeetCode, offering a diverse range of difficulty levels.”
InfluentialSeed-Coder: Let the Code Model Curate Data for Itself
Seed, B., Zhang, Y., Su, J., Sun, Y., Xi, C., Xiao, X., Zheng, S., Zhang, A., Liu, K., Zan, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.03524 · Cited by 3 · see citing works
“To mitigate these issues, LiveCodeBench [Jain et al., 2025] introduces live updates by continuously collecting new problems from coding contests across prominent competitive programming platforms – LeetCode, AtCoder, and CodeForces – and tagging each problem with a release date.”
DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
Du, M., Xu, B., Zhu, C., Wang, X., & Mao, Z. (2025) · arXiv · doi:10.48550/arxiv.2506.11763 · Cited by 2 · see citing works
“These include evaluations for agents in scientific domains [2, 12, 17, 3], creative writing [31, 1, 23], code generation and software engineering [10, 40, 22, 9, 32], and in their roles as human assistants, often enhanced by capabilities such as web Browse and tool-use [28, 38, 33, 6, 26].”
Kimi K2.5: Visual Agentic Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., Cai, S. H., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.02276 · Cited by 2 · see citing works
Self-citationFullStack Bench: Evaluating LLMs as Full Stack Coders
Bytedance-Seed-Foundation-Code-Team, :, Cheng, Y., Chen, J., Chen, J., Chen, L., Chen, L., Chen, W., Chen, Z., Geng, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.00535 · Cited by 2 · see citing works
Self-citationReasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
Wu, M., Zhang, Z., Dong, Q., Xi, Z., Zhao, J., Jin, S., Fan, X., Zhou, Y., Lv, H., Zhang, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.10532 · Cited by 1 · see citing works
“Because our evaluation matches only the final numeric answer, the model can sometimes output the correct value by accident, which explains the anomalous accuracy of Llama on AIME2025 (solving exactly one problem with a faulty reasoning process).”
InfluentialMiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1 · see citing works
“On LiveCodeBench v5, it significantly outperforms OpenAI o1-mini, while on the latest LiveCodeBench v6, our model achieves a score of 49.3%, surpassing QwQ-32B-Preview by over 10 points, demonstrating its robust and stable capabilities.”
InfluentialDeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
Huang, W., Lee, C., Tng, L., & Ge, S. (2026) · arXiv · doi:10.48550/arxiv.2607.07946 · Cited by 2 · see citing works
“…by issue or pull-request date and restrict contamination-free evaluation to instances post-dating a model’s training cutoff, while LiveCodeBench [Jain et al., 2025] applies the same release-date windowing to competitive-programming problems; this requires continual re-collection as tasks age…”
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1 · see citing works
“Where earlier we could only post-train on human demonstrations (Mishra et al., 2021), language models can now be trained with just rewards (Shao et al., 2024), enabling sophisticated reasoning (Guo et al., 2025; Jain et al., 2024) and agents (Kimi Team et al., 2025).”
SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1 · see citing works
“• Recall (Li et al., 2024): we calculate recall scores specifically for intra-file dependencies, cross-file dependencies, and external APIs.”
McEval: Massively Multilingual Code Evaluation
Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07436 · Cited by 1 · see citing works
“…scenarios (Wang et al., 2023; Athiwaratkun et al., 2023; Peng et al., 2024; Zheng et al., 2023b) by extending the Python-only HumanEval/MBPP benchmark (e.g. MultiPL-E (Cassano et al., 2023)), which is challenged by the number of the languages and data leaking problem (Jain et al., 2024).”
MethodAgentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Lin, J., Liu, S., Pan, C., Lin, L., Dou, S., Xi, Z., Huang, X., Yan, H., Han, Z., Gui, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.25850 · Cited by 1 · see citing works
OpenThoughts: Data Recipes for Reasoning Models
Guha, E., Marten, R., Keh, S., Raoof, N., Smyrnis, G., Bansal, H., Nezhurina, M., Mercat, J., Vu, T., Sprague, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.04178 · Cited by 1 · see citing works
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
Zhao, B., Srikanth, D., Wu, Y., & Jiang, Z. (2026) · arXiv · doi:10.48550/arxiv.2605.21384 · Cited by 1 · see citing works
Search Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1 · see citing works
“The dataset and evaluation landscape has progressed from standardized, static evaluations [22, 28, 23, 70] toward more challenging settings such as deep reasoning [35, 30, 68], coding [27, 26], and open-ended dialogue under crowd-sourced evaluation [10, 70].”
The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
Iacob, A., Jovanović, A., Shen, W. F., Burkhardt, D., Kurmanji, M., Tastan, N., Sani, L., Venanzi, N. A. E., Nie, J., Odonnat, A., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.26294 · Cited by 1 · see citing works
“C.2).”
R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
Jain, N., Singh, J., Shetty, M., Zheng, L., Sen, K., & Stoica, I. (2025) · arXiv · doi:10.48550/arxiv.2504.07164 · Cited by 1 · see citing works
Self-citationRobeyns, M., Szummer, M., & Aitchison, L. (2025) · arXiv · doi:10.48550/arxiv.2504.15228 · Cited by 1 · see citing works
Towards a Science of Scaling Agent Systems
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.08296 · Cited by 1 · see citing works
Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
Liu, R., Sun, Y., Zhang, M., Bai, H., Yu, X., Yu, T., Yuan, C., & Hou, L. (2025) · arXiv · doi:10.48550/arxiv.2504.04823
“5 (LI et al., 2024) and LiveCodeBench (Jain et al., 2024)) and pre-training (i.e. WikiText2 (Merity et al., 2016)) data.”
Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
Qiu, E. S., & Gill, J. (2026) · arXiv · doi:10.48550/arxiv.2608.18167
“We first evaluate this construction on LiveCodeBench (LCB) (Jain et al., 2024).”
HELIX: Model-Harness Co-evolution for Recursive Self-Improvement
Fan, T., & Huang, C. (2026) · arXiv
“LiveCodeBench provides a continuously updated evaluation for code capabilities [11]; SWE-bench turns real GitHub issues into executable repository-level tasks [12].”
A Unified Approach to Routing and Cascading for LLMs
Dekoninck, J., Baader, M., & Vechev, M. (2024) · arXiv · doi:10.48550/arxiv.2410.10347
“For the Minerva Math and LiveCodeBench benchmark, we only include the Algebra portion of Minerva Math to ensure that both benchmarks have a comparable number of samples for evaluation.”
InfluentialMulti-LCB: Extending LiveCodeBench to Multiple Programming Languages
Ivanova, M., Zadorozhny, P., Levichev, R., Petrov, I., Pavel, A., Lopatin, I., Kutalev, A., & Babaev, D. (2026) · arXiv · doi:10.48550/arxiv.2606.20517
“LiveCodeBench (LCB) (Jain et al., 2024) addresses these shortcomings by continuously curating competitive-programming problems, filtering them by release date, and enabling contamination-aware, continuously updatable evaluation .”
InfluentialJiang, J., Shen, J., Kim, S., Yoo, K. M., Kim, J., & Kim, S. (2026) · arXiv · doi:10.48550/arxiv.2603.05863
“…establishes a new state of the art among leading open-source models, achieving 94.51% on HumanEval (Chen et al., 2021), 35.00% on BigCodeBench (Zhuo et al., 2024), 52.21% on LiveCodeBench (Naman Jain et al., 2024), and 37.34% on CodeForces (Quan et al., 2025) in a single-attempt setting.”
InfluentialProcess Supervision via Verbal Critique Improves Reasoning in Large Language Models
Chen, H.-Y. (2026) · arXiv · doi:10.48550/arxiv.2604.21611
“LiveCodeBench V6 [Jain et al., 2024] is a contamination-free competitive programming benchmark (latest release; SOTA 91.7%); we report pass@1.”
InfluentialWhat Makes In-Context Examples Effective for Code Generation?
Li, D., Chen, S., Cao, J., & Cheung, S.-C. (2026) · Proceedings of the ACM on Software Engineering · doi:10.1145/3832275
Sun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025) · arXiv · doi:10.48550/arxiv.2503.16402
“Recent works have proposed novel benchmarks to address contamination (Li et al., 2024b; Zhu et al., 2023a; Jain et al., 2024; Li et al., 2024a; Qian et al., 2024; Wu et al., 2024).”
Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents
Zhang, Q., Wornow, M., Wan, G., & Olukotun, K. (2025) · arXiv · doi:10.48550/arxiv.2506.14852
“Agent applications based on Large Language Models (LLMs) have shown early promise in replicating human performance on a broad range of workflows, from coding [26, 28, 61] to web navigation [23, 70] to open-ended research [2, 7] to social interactions [42, 57].”
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Yang, X., Gu, H., Xu, B., Li, L., Liu, B., Liu, J., Zhu, Q., Guo, Y., & Han, S. (2026) · arXiv · doi:10.48550/arxiv.2605.25475
“In contrast, heuristic compression baselines (e.g., PyramidKV and SnapKV) show clear position-dependent failure modes: they perform reasonably when the needle lies in more favorable regions, but degrade sharply for harder placements, consistent with a strong recency bias and coarse token-selection granularity.”
InfluentialEvilGenie: A Reward Hacking Benchmark
Gabor, J., Lynch, J., & Rosenfeld, J. (2025) · arXiv · doi:10.48550/arxiv.2511.21654
“Problems are sourced from L IVE C ODE B ENCH [4] and adapted to permit complex agentic behavior, including reward hacking.”
Zhou, C., Zhang, X., Song, D., Chen, X., Gu, W., Ma, H., Tian, Y., Zhang, M., & Hu, L. (2025) · arXiv · doi:10.48550/arxiv.2502.09183
“We restrict data usage to the range from 240701 to 250201 of the LiveCodeBench to maintain its contamination-free nature.”
InfluentialHow to Steal Reasoning Without Reasoning Traces
Zhang, T., Morris, J. X., & Shmatikov, V. (2026) · arXiv · doi:10.48550/arxiv.2603.07267
“…inversion model I sum ( x, y, b ) = ˆ t is trained to maximize the likelihood that its output matches the true trace t ′ : The no-summary setting is similar: I nosum ( x, y ) = ˆ t and the training objective is: Both objectives are implemented using teacher forcing with token-level cross-entropy.”
Self-citationSWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Training
Song, H., Huang, L., Sun, S., Jiang, J., Le, R., Cheng, D., Chen, G., Hu, Y., Chen, Z., Jia, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.03411
“Unlike static benchmarks ( e.g., code genreation [20], websearch [21]), such tasks demand interactive execution environments with terminal access, persistent file systems, and package management support, allowing agents to compile, run, and debug code under realistic conditions.”
Self-citationMergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
“Tasks are grouped into several categories, including: (a) general knowledge/reasoning (ARC (Bhak-thavatsalam et al., 2021), AGIEval (Zhong et al., 2024), OpenBookQA (Mihaylov et al., 2018), BBH (Suzgun et al., 2023), WorldSense (Hong et al., 2025), PIQA (Bisk et al., 2020), hellaswag (Zellers et al., 2019) and KOR-Ben…”
InfluentialXie, S., Shi, Z., Shen, H., Wang, J., Zhong, S., Liu, M., Pan, C., Jia, P., Sun, B., Jing, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.21482
“B ENCH currently comprises 100 tasks spanning six domain categories: Technology (25), Energy & Materials (20), Industrials & Transport (18), Consumer (16), Finance (12), and Healthcare & Pharma (9).”
Li, Z., Li, Z., Shi, Y., Wang, R., Yang, J., Liu, Z., Wu, X., Li, A., Huang, C., Shang, E., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.08964
Ning, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
Self-citationBetter Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation
Yang, C., Zhao, X., Wu, T., & Kästner, C. (2026) · arXiv · doi:10.48550/arxiv.2607.08938
Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies
Zhou, H., Wan, X., Sun, R., Palangi, H., Iqbal, S., Vulić, I., Korhonen, A., & Arık, S. Ö. (2025) · arXiv · doi:10.48550/arxiv.2502.02533
CLEVER: A Curated Benchmark for Formally Verified Code Generation
Thakur, A., Lee, J., Tsoukalas, G., Sistla, M., Zhao, M., Zetzsche, S., Durrett, G., Yue, Y., & Chaudhuri, S. (2025) · arXiv · doi:10.48550/arxiv.2505.13938
AlphaResearch: Accelerating New Algorithm Discovery with Language Models
Yu, Z., Feng, K., Zhao, Y., He, S., Zhang, X.-P., & Cohan, A. (2025) · arXiv · doi:10.48550/arxiv.2511.08522
Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model
Lab, N., :, Yang, C., Huang, C., Lan, F., Chen, H., Zhou, H., Song, H., Cao, J., Zhu, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.22083
Self-citationImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
Zhong, Z., Raghunathan, A., & Carlini, N. (2025) · arXiv · doi:10.48550/arxiv.2510.20270
LiveMedBench: A Contamination-Limited Medical Benchmark for LLMs with Automated Rubric Evaluation
Yan, Z., Song, D., Fang, Z., Ji, Y., Li, X., Li, Q., & Sun, L. (2026) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3817579
MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution
Shang, Z., Huo, Y., Gong, M., Yan, A., Qu, Z., Dong, J., Low, B. K. H., Wu, C., & Dai, Z. (2026) · arXiv · doi:10.48550/arxiv.2610.05300
The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute
Jin, Y., Wei, G.-Y., & Brooks, D. (2025) · arXiv · doi:10.48550/arxiv.2505.14733
MARFT: Multi-Agent Reinforcement Fine-Tuning
Liao, J., Wen, M., Wang, J., & Zhang, W. (2025) · arXiv · doi:10.48550/arxiv.2504.16129
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Seo, M., Baek, J., Lee, S., & Hwang, S. J. (2025) · arXiv · doi:10.48550/arxiv.2504.17192
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Bogin, B., Yang, K., Gupta, S., Richardson, K., Bransom, E., Clark, P., Sabharwal, A., & Khot, T. (2024) · arXiv · doi:10.48550/arxiv.2409.07440
AIOpsLab in Action: An Open Platform for AIOps Research
Ma, M., Clark, J., & Zhang, S. (2025) · Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3696630.3728619
Detecting Data Contamination in Code Generation Benchmarks via DiffMIA
Yamashita, K. (2025) · 2025 32nd Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00128
RSI-Anything: Unlocking Recursive Self-Improvement with World Models
Cui, T., Zeng, P., Xu, W., Xiao, H., Xue, S., Gao, Q., Ouyang, W., Heng, P. A., Yin, Z., Wu, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2610.13912
Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis
Dolcetti, G., Arceri, V., Iotti, E., Maffeis, S., Cortesi, A., & Zaffanella, E. (2024) · arXiv · doi:10.48550/arxiv.2412.14841 · Cited by 8
Zhang, L., He, S., Zhang, C., Kang, Y., Li, B., Xie, C., Wang, J., Wang, M., Huang, Y., Fu, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.23419
- Authors
- Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., & Stoica, I.
- Affiliations
- Cornell University (US)
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2403.07974
- OpenAlex
- W4392822190
- Cited by
- 42
- References
- 0
- Citations by year
- 2026: 15 · 2025: 20 · 2024: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1908870, 2141064, 1900968, 2217064)
- Intel Corporation
- VMware
- Zayed University
- Samsung
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- LiveCodeBench, code large language models, benchmark contamination, programming competition, LeetCode, Codeforces, program execution, instruction-tuned LLMs, code generation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
