Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs), such as GPT-4, StarCoder, and Code Llama, are transforming the way developers approach programming by automatically generating code based on given contexts, such as natural language descriptions or incomplete surrounding code. Despite advancements, generating syntactically and semantically correct code remains challenging, especially for complex programming tasks. Existing approaches typically generate multiple candidate solutions using LLMs to increase the likelihood of producing correct code. However, selecting the correct code from these candidates — a process known as code ranking — remains a major challenge. Current research on code ranking can be cate-gorized into execution-based and non-execution-based methods. Execution-based methods, although effective, encounter notable limitations, such as scarcity of quality unit tests and security risks. Non-execution-based methods like CodeRanker, which rely solely on classification labels to train a code ranker, struggle to capture subtle errors and provide detailed error insights. Recognizing the strengths and limitations of both approaches, we propose a new method that integrates the advantages of execution-based and non-execution-based techniques. The key insight of our work is that an effective code ranker is expected to truly comprehend the underlying causes of erroneous code, as relying solely on classification labels is insufficient. Inspired by this, this paper puts forward RankEF, an innov…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“The detailed process is illustrated in Figure 2.”
MethodGolub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
Feature Selection: A Data Perspective
Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., & Liu, H. (2018) · ACM Computing Surveys · doi:10.1145/3136625 · Cited by 2,442
Self-citationSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 1×
“…learning [22, 62, 65], multi-task learning [72], dual In contrast, our approach trains an independent code ranker using a multi-task learning framework, eliminating the dependence on execution feedback during the inference phase, thereby overcoming the limitations of the aforementioned methods.”
MethodWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 1×
“…as the output, such neural comment generation (NCG) approaches automatically learn a function, which is usually a DNN model such as the neural machine Through the integration of execution feedback and classification labels using multi-task learning, RankEF not only distinguishes between…”
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · cited 1×
“…as the output, such neural comment generation (NCG) approaches automatically learn a function, which is usually a DNN model such as the neural machine Through the integration of execution feedback and classification labels using multi-task learning, RankEF not only distinguishes between…”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 13×
“Execution-based code ranking methods, which involve sampling a large number of code candidates and executing a few unit tests to filter out those that pass, have notably improved the code generation capabilities of LLMs [4, 6, 7].”
BackgroundMethodInfluentialData Preprocessing in Data Mining
García, S., Luengo, J., & Herrera, F. (2015) · Intelligent Systems Reference Library · doi:10.1007/978-3-319-10247-4 · Cited by 1,368
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Research works printed but not linked: 12
- OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. · arXiv
- Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. · arXiv
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. · arXiv
- Mohammed Latif Siddiq, Joanna Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. Exploring the effectiveness of large language models in generating unit tests. arXiv preprint arXiv:2305.00418, 2023. · arXiv
- Mohammed Latif Siddiq and Joanna Santos. Generate and pray: Using sallms to evaluate the security of llm generated code. arXiv preprint arXiv:2311.00889, 2023. · arXiv
- Rebeka Tóth, Tamas Bisztray, and László Erdodi. Llms in web-development: Evaluating llm-generated php code unveiling vulnerabilities and limitations. arXiv preprint arXiv:2404.14459, 2024. · arXiv
- Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. · arXiv
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. · arXiv
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999, 2022. · arXiv
- Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. · arXiv
- Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Cited by
Large Language Model-Aware In-Context Learning for Code Generation
Li, J., Tao, C., Li, J., Li, G., Jin, Z., Zhang, H., Fang, Z., & Liu, F. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715908 · Cited by 32
Self-citationLarge Language Model-Aware In-Context Learning for Code Generation
Li, J., Li, G., Tao, C., Li, J., Zhang, H., Liu, F., & Jin, Z. (2023) · arXiv · doi:10.48550/arxiv.2310.09748 · Cited by 32
Self-citationMulti-stage guided code generation for Large Language Models
Han, Y., & Lyu, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109491 · Cited by 36
Self-citationWebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Gui, Y., Li, Z., Wan, Y., Shi, Y., Zhang, H., Chen, B., Su, Y., Chen, D., Wu, S., Zhou, X., et al. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714889 · Cited by 17
Self-citationUIC opilot : Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs
Gui, Y., Wan, Y., Li, Z., Zhang, Z., Chen, D., Zhang, H., Su, Y., Chen, B., Zhou, X., Jiang, W., et al. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714891 · Cited by 62
Self-citationLaTCoder: Converting Webpage Design to Code with Layout-as-Thought
Gui, Y., Li, Z., Zhang, Z., Wang, G., Lv, T., Jiang, G., Liu, Y., Chen, D., Wan, Y., Zhang, H., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737016 · Cited by 28
Self-citationCode Ranking with Structure Awareness Contrastive Learning
Huang, H., Cao, L., Wang, J., Yu, T., & Cai, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00053 · Cited by 1
“• RankEF [24] employs multi-task learning to integrate code classification with execution feedback generation.”
Duan, Y., Zhang, D., Jiang, Z., Xie, Z., Liu, Y., Xu, Y., Li, R., & Cui, D. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786872 · Cited by 1
A Code Ranking Framework with Human-Inspired Agent-Based Tests Generation
Cao, L., Liang, J., Wang, J., & Cai, Y. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794820
SkelDPO: A Skeleton-Guided Direct Preference Optimization Framework for Efficient Code Generation
Yu, Y., & Lyu, C. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794824 · Cited by 1
Self-citationSun, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787758
Self-citation
Metadata
- Authors
- Sun, Z., Wan, Y., Li, J., Zhang, H., Jin, Z., Li, G., & Lyu, C.
- Affiliations
- Shandong Normal University (CN)
- Huazhong University of Science and Technology (CN)
- Peking University (CN)
- Chongqing University (CN)
- Venue
- Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2024-10-27
- Pages
- 229-241
- OpenAlex
- W4403536858
- Semantic Scholar
- de630124d259fec0133263bd51124a72a5f3b4ed
- Cited by
- 10
- References
- 29
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 1
- FWCI
- 3.81
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Science Foundation of Shandong Province, China?ZR2021MF059
- National Natural Science Foundation of China?62192731, 62072007, 62192733, 61832009, 62192730
- National Key R&D Program?2023YFB4503801
- Key Program of Hubei?JD2023008
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- execution feedback, large language models, code generation, multi-task learning, code classification, code quality evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.