On the Impacts of Contexts on Repository-Level Code Generation
Open accessAbstract
CodeLLMs have gained widespread adoption for code generation tasks, yet their capacity to handle repository-level code generation with complex contextual dependencies remains underexplored. Our work underscores the critical importance of leveraging repository-level contexts to generate executable and functionally correct code. We present RepoExec, a novel benchmark designed to evaluate repository-level code generation, with a focus on three key aspects: executability, functional correctness through comprehensive test case generation, and accurate utilization of cross-file contexts. Our study examines a controlled scenario where developers specify essential code dependencies (contexts), challenging models to integrate them effectively. Additionally, we introduce an instruction-tuned dataset that enhances CodeLLMs' ability to leverage dependencies, along with a new metric, Dependency Invocation Rate (DIR), to quantify context utilization. Experimental results reveal that while pretrained LLMs demonstrate superior performance in terms of correctness, instruction-tuned models excel in context utilization and debugging capabilities. RepoExec offers a comprehensive evaluation framework for assessing code functionality and alignment with developer intent, thereby advancing the development of more reliable CodeLLMs for real-world applications. The dataset and source code are available at https://github.com/FSoft-AI4Code/RepoExec.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Constraint-reconciliation decay occurs: as sessions lengthen and dependencies accumulate, success rates fall.
“This results in constraintreconciliation decay : as sessions lengthen and dependencies accumulate, success rates fall.”
From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [10]
Table 5 in §6.1 reports that Pass@1 rose across three debugging rounds for GPT-3.5 and WizardCoder and remained unchanged for CodeLlama-13b-Python; it did not fall as rounds accumulated. The multi-round debugging discussion also reports improved dependency use. The paper does not establish a separate trend for sessions in which dependencies accumulate. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredRepository-level evaluations reveal cross-file integration and dependency breaks.
“Repository-level evaluations reveal similar cross-file integration and dependency breaks [10, 22].”
From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [10]
The Introduction and Section 3 describe repository-level evaluation of functional correctness and dependency use; the dependency-extraction section explicitly includes cross-file dependencies. The Additional information section gives examples of models failing to use available dependencies, but our copy does not establish distinct cross-file integration breaks. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · see citing works
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 538 · see citing works
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 2× · see citing works
“…for assisting with coding tasks (Wang et al., 2021, 2023; Feng et al., 2020; Allal et al., 2023; Li et al., 2023; Lozhkov et al., 2024; Guo et al., 2024; Pinnaparaju et al., 2024; Zheng et al., 2024; Roziere et al., 2023; Nijkamp et al., 2022; Luo et al., 2023; Xu et al., 2022; Bui et al., 2023).”
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2305.07922 · Cited by 459
Self-citationProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · see citing works
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · see citing works
OpenCodeInterpreter: Integrating Code Generation with Execution and RefinementPreprint
Zheng, T., Zhang, G., Shen, T., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.14658 · Cited by 322 · see citing works
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294 · see citing works
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · see citing works
DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence
Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y. K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.14196 · Cited by 180 · see citing works
Gemma 2: Improving Open Language Models at a Practical Size
Gemma Team, Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.00118 · Cited by 149 · see citing works
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143 · see citing works
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code RepositoriesPreprint
Li, J., Li, G., Zhao, Y., Li, Y.-M., Liu, H.-Y., Zhu, H., Wang, L., Liu, K., Fang, Z., Wang, L.-S., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2405.19856 · Cited by 141 · see citing works
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11644 · Cited by 138 · see citing works
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.15115 · Cited by 128 · see citing works
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023) · arXiv · doi:10.48550/arxiv.2306.08568 · Cited by 120 · see citing works
Zhang, K., Li, J., Li, G., Shi, X., & Jin, Z. (2024) · arXiv · doi:10.48550/arxiv.2401.07339 · Cited by 97 · see citing works
Qwen2.5-Coder Technical Report
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12186 · Cited by 65 · see citing works
SantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61 · see citing works
Yi: Open Foundation Models by 01.AI
AI, 0., :, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Wang, G., Li, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04652 · Cited by 50 · see citing works
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation
Huang, D., Zhang, J. M., Luck, M., Bu, Q., Qing, Y., & Cui, H. (2023) · arXiv · doi:10.48550/arxiv.2312.13010 · Cited by 34 · see citing works
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.08905 · Cited by 31 · see citing works
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29 · cited 1× · see citing works
“This criterion has been widely employed for evaluating code generation in numerous studies (Chen et al., 2021; Austin et al., 2021; Zhuo et al., 2024; Li et al., 2024).”
CodeBenchGen: Creating Scalable Execution-based Code Generation BenchmarksPreprint
Xie, Y.-Q., Xie, A., Sheth, D., Liu, P.-F., Fried, D., & Rosé, C. P. (2024) · arXiv.org · doi:10.48550/arxiv.2404.00566 · Cited by 25 · see citing works
RepoBench: Benchmarking Repository-Level Code Auto-Completion SystemsPreprint
Liu, T., Xu, C., & McAuley, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.03091 · Cited by 23 · see citing works
Execution-Based Evaluation for Open-Domain Code Generation
Wang, Z., Zhou, S., Fried, D., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2212.10481 · Cited by 20 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., & Chen, W. (2023) · arXiv · Cited by 10 · see citing works
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6 · see citing works
Execution-based Evaluation for Data Science Code Generation ModelsPreprint
Huang, J., Wang, C., Zhang, J., Yan, C., Cui, H., Inala, J. P., Clement, C., Duan, N., & Gao, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09374 · Cited by 4 · see citing works
CodeTF: One-stop Transformer Library for State-of-the-art Code LLMs
Bui, N. D. Q., Le, H., Wang, Y., Li, J., Gotmare, A. D., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2306.00029 · Cited by 4
Self-citationHyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale
Phan, H. N., Nguyen, T. N., Nguyen, P. X., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2409.16299 · Cited by 2 · see citing works
Self-citationBetter Language Models of Code through Self-ImprovementPreprint
To, H. Q., Bui, N. D. Q., Guo, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.01228 · Cited by 2 · see citing works
Self-citationAnalyzing the Performance of Large Language Models on Code SummarizationPreprint
Haldar, R., & Hockenmaier, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.08018 · Cited by 2 · see citing works
Manh, D. N., Chau, T. P., Hai, N. L., Doan, T. T., Nguyen, N. V., Pham, Q., & Bui, N. D. Q. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.01999
Self-citationStable Code Technical ReportPreprint
Pinnaparaju, N., Adithyan, R., Phung, D., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.01226
Flagged first, then by citations
Research works printed but not linked: 31
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
- Yangruibo Ding, Zijian Wang, Wasi U. Ahmad, Mu- rali Krishna Ramanathan, Ramesh Nallapati, Par- minder Bhatia, Dan Roth, and Bing Xiang. 2024. CoCoMIC: Code completion by jointly modeling in-file and cross-file context. In Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evalua- tion (LREC-COLING 2024).
- Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Han- tian Ding, Ming Tan, Nihal Jain, Murali Krishna Ra- manathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. Crosscodeeval: A di- verse and multilingual benchmark for cross-file code completion. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
- Nam Le Hai, Anh MT Bui, Phuong T Nguyen, Davide Di Ruscio, and Rick Kazman. 2024a. Improving the detection of technical debt in java source code with an enriched dataset. arXiv preprint arXiv:2411.05457. · arXiv
- Nam Le Hai, Trang Nguyen, Linh Ngo Van, Thien Huu Nguyen, and Khoat Than. 2024b. Continual varia- tional dropout: a view of auxiliary local variables in continual learning. Machine Learning, 113(1):281- 323.
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge com- petence with apps. NeurIPS.
- Maor Ivgi, Uri Shaham, and Jonathan Berant. 2023. Ef- ficient long-text understanding with short-text mod- els. Transactions of the Association for Computa- tional Linguistics, 11:284-299.
- Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1643-1652.
- Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. 2024. R2e: Turning any github repository into a programming agent envi- ronment. In Forty-first International Conference on Machine Learning.
- Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Es- caping coverage plateaus in test generation with pre- trained large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineer- ing (ICSE), pages 919-931. IEEE.
- Dianshu Liao, Shidong Pan, Qing Huang, Xiaoxue Ren, Zhenchang Xing, Huan Jin, and Qinying Li. 2023. Context-aware code generation framework for code repositories: Local, global, and third-party library awareness. arXiv preprint arXiv:2312.05772. · arXiv
- Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. 2023a. Is your code gen- erated by chatGPT really correct? rigorous evalu- ation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems.
- Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024b. Lost in the middle: How language models use long contexts. Transactions of the Asso- ciation for Computational Linguistics, 12:157-173.
- Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. 2024c. No need to lift a finger anymore? assessing the quality of code generation by chatgpt. IEEE Transactions on Software Engineer- ing.
- Anton Lozhkov, Raymond Li, Loubna Ben Allal, Fed- erico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173. · arXiv
- Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. 2023. An empirical study of automated unit test gen- eration for python. Empirical Software Engineering, 28(2):36.
- Everton da S Maldonado and Emad Shihab. 2015. Detecting and quantifying different types of self- admitted technical debt. In 2015 IEEE 7Th interna- tional workshop on managing technical debt (MTD), pages 9-15. IEEE.
- Debanjan Mondal, Abhilasha Lodha, Ankita Sahoo, and Beena Kumari. 2023. Robust code summarization. In Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP, pages 65-75, Singapore. Association for Computational Linguis- tics.
- Dung Nguyen, Le Nam, Anh Dau, Anh Nguyen, Khanh Nghiem, Jin Guo, and Nghi Bui. 2023. The vault: A comprehensive multilingual dataset for advanc- ing code understanding and generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4763-4788, Singapore. Associ- ation for Computational Linguistics.
- Huy N Phan, Hoang N Phan, Tien N Nguyen, and Nghi DQ Bui. 2024a. Repohyper: Better context retrieval is all you need for repository-level code completion. arXiv preprint arXiv:2403.06095. · arXiv
- José Amancio M Santos, João B Rocha-Junior, Luciana Carla Lins Prates, Rogeres Santos Do Nascimento, Mydiã Falcão Freitas, and Manoel Gomes De Men- donça. 2018. A systematic review on the code smell effect. Journal of Systems and Software, 144:450- 477.
- Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. Adaptive test generation using a large language model. arXiv e-prints, pages arXiv-2302.
- Disha Shrivastava, Hugo Larochelle, and Daniel Tar- low. 2023. Repository-level prompt generation for large language models of code. In International Con- ference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 31693-31715.
- PMLR.
- Ankita Nandkishor Sontakke, Manasi Patwardhan, Lovekesh Vig, Raveendra Kumar Medicherla, Ravin- dra Naik, and Gautam Shroff. 2022. Code summa- rization: Do transformers really understand code? In Deep Learning for Code Workshop.
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- Quyen Tran, Nguyen Thanh, Nguyen Anh, Nam Hai, Trung Le, Linh Ngo, and Thien Nguyen. 2024. Pre- serving generalization of language models in few- shot continual relation extraction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13771-13784.
- Frank F Xu, Uri Alon, Graham Neubig, and Vincent Jo- sua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1-10.
- Prateek Yadav, Qing Sun, Hantian Ding, Xiaopeng Li, Dejiao Zhang, Ming Tan, Parminder Bhatia, Xiaofei Ma, Ramesh Nallapati, Murali Krishna Ramanathan, Mohit Bansal, and Bing Xiang. 2023. Exploring continual learning for code generation models. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 782-792, Toronto, Canada. As- sociation for Computational Linguistics. · DOI
- Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In Proceedings of the 15th interna- tional conference on mining software repositories, pages 476-486.
- Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023b. Repocoder: Repository-level code completion through iterative retrieval and gen- eration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2023, Singapore, December 6-10, 2023, pages 2471-2484. Association for Computational Linguistics.
Cited by
Sorted by citations
Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification
Mitchell, J., & Shaaban, Y. (2025) · Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages · doi:10.1145/3759425.3763390 · Cited by 8 · see citing works
- Authors
- Hai, N. L., Nguyen, D. M., & Bui, N. D. Q.
- Affiliations
- FPT University (VN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.11927
- OpenAlex
- W4399836661
- Cited by
- 3
- References
- 40
- Citations by year
- 2026: 1 · 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/legalcode
- Locations
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- repository-level code generation, test case generation, functional correctness, instruction-tuned LLMs, debugging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
