The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair
Open accessAbstract
During Automated Program Repair (APR), it can be challenging to synthesize correct patches for real-world systems in general-purpose programming languages. Recent Large Language Models (LLMs) have been shown to be helpful “copilots” in assisting developers with various coding tasks, and have also been directly applied for patch synthesis. However, most LLMs treat programs as sequences of tokens, meaning that they are ignorant of the underlying semantics constraints of the target programming language. This results in plenty of statically invalid generated patches, impeding the practicality of the technique. Therefore, we propose Repilot, a framework to further copilot the AI “copilots” (i.e., LLMs) by synthesizing more valid patches during the repair process. Our key insight is that many LLMs produce outputs autoregressively (i.e., token by token), resembling human writing programs, which can be significantly boosted and guided through a Completion Engine. Repilot synergistically synthesizes a candidate patch through the interaction between an LLM and a Completion Engine, which 1) prunes away infeasible tokens suggested by the LLM and 2) proactively completes the token based on the suggestions provided by the Completion Engine. Our evaluation on a subset of the widely-used Defects4j 1.2 and 2.0 datasets shows that Repilot fixes 66 and 50 bugs, respectively, surpassing the best-performing baseline by 14 and 16 bugs fixed. More importantly, Repilot is capable of producing more valid and correct patches than the base LLM when given the same generation budget.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“This approach enhances the compilation rate and correctness of patches while reducing the number of invalid attempts in the generation process.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Wei et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Large Language Models (LLMS) trained on code (LLMCS) have demonstrated significant effectiveness in generating program fixes (Joshi et al. 2023; Xia, Ding, and Zhang 2023; Wei, Xia, and Zhang 2023; Fan et al. 2023; Xia, Wei, and Zhang 2023; Brancas, Manquinho, and Martins 2024).”
From Counterexample Guided Program Repair Using Zero-Shot Learning and MaxSAT-based Fault Localization (2025) · cites this paper as Wei, Xia, and Zhang 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
GenProg: A Generic Method for Automatic Software Repair
Goues, C. L., Nguyen, T. H., Forrest, S., & Weimer, W. R. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,084
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Unified Pre-training for Program Understanding and GenerationPreprint
Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.06333 · Cited by 1,000
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Flagged first, then by citations
Research works printed but not linked: 11
- 10.18653/v1 · DOI
- Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=hQwb-lbM6EL
- Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Representations with Data Flow. In International Conference on Learning Representations. https://openreview.net/forum?id=jLoC4ez43PZ
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations. https://openreview.net/forum?id=rygGQyrFvH
- Sophia D Kolak, Ruben Martins, Claire Le Goues, and Vincent Josua Hellendoorn. 2022. Patch Generation with Language Models: Feasibility and Scaling Behavior. In Deep Learning for Code Workshop. https://openreview.net/forum?id=rHlzJh_b1-5
- Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable Code Generation from Pre-trained Language Models. In International Conference on Learning Representations. https://openreview.net/forum?id=KmtVD97J43e
- 10.18653/v1 · DOI
- Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (Eds.). 27, Curran Associates, Inc.. https://proceedings.neurips.cc/paper_files/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf
- 10.5555/3295222.3295349 · DOI
- 10.18653/v1 · DOI
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. Curran Associates Inc., Red Hook, NY, USA.
Cited by
Sorted by citations
Items 101–118 of 118
An empirical assessment of go linters on real-world issues
Wu, J., & Clause, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112797
An Empirical Study of Python Library Migration Using Large Language Models
Islam, M., Jha, A. K., Mahmoud, M., Akhmetov, I., & Nadi, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00077 · Cited by 7
Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat
Ji, K., Fu, S., Gao, C., Chen, Y., Yang, Z., Wang, C. J., & Deng, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00285
Benchmarking Large Language Models for Automated Code Changes: A Systematic Review
Sandhu, P., Khurana, P., & Singla, J. (2026) · International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) · doi:10.1109/idciot67589.2026.11455819
An Environment-Aware Verification Framework for LLM-Generated Robot Control Programs
Nie, Z., Wang, W., Liu, X., Zhang, Y., Chen, Z. T., Wang, Z., Huang, K., & Zhao, S. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539304
PortGPT: Towards Automated Backporting Using Large Language Models
Li, Z. L., Yu, Z., Song, J., Xu, M., Luo, Y., & Mu, D. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00034 · Cited by 2
SliceRepair: Constraining LLM-Based Vulnerability Repair with Code Property Graph Slices
Bian, Y., Chen, Y., Li, S., Wang, H., & Cui, Z. (2026) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd69640.2026.11637294
The impact of Software Model Slicing on Software Model Completion with Large Language Models
Welter, A., Bliem, B., Iqbal, O., & Apel, S. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3830319
Automatic Dockerfile Generation with Large Language Models
Lyu, J., Zhang, H., Yuan, Y., Yang, L., Li, Y., & Rigger, M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773162
Mining Invariants to Enhance Fuzzing for Detecting Functional Bugs in Smart Contracts
Lin, X., Zhao, B., Xie, Q., Ruan, N., Li, J., & Ji, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3849091
Closed-Loop LLM-Driven Self-Healing for Enterprise Data Pipelines
Sivaraman, M. K. (2026) · Technology and Innovation Conference (TIC) · doi:10.1109/tic68483.2026.11703022
NSync: Automated Cloud Infrastructure-as-Code Reconciliation with AI Agents
Yang, Z., Guan, H., Nicolet, V., Paulsen, B., Dodds, J., Kroening, D., & Chen, A. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832121
Automated Program Repair for UI-Centric Android Bugs: How Far Are We?
Mahmud, J., Pandey, S., Silva, N. D., Dipongkor, A. K., Wu, J., Chaparro, O., Fazzini, M., & Moran, K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832272
Pattern Synergistic Repair: Multi-expert LoRA fusion for automated program repair
Wang, S., & Shen, Z. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108344
LaMAR: Latent Multi-agent Collaboration via KV-Cache Communication for Automated Program Repair
Jiao, R., Zhang, Y., Feng, Z., & Li, J. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837423
CiL-Agent: Compiler-in-the-Loop LLM Agent for Memory-Safe C Program Repair
Shen, Y., Zhang, Y., Yao, J., Liu, M., & Dong, W. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837493
Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification
Mitchell, J., & Shaaban, Y. (2025) · arXiv · Cited by 8
Type-Constrained Code Generation with Language Models
Mündler, N., He, J., Wang, H., Sen, K., Song, D., & Vechev, M. (2025) · arXiv · doi:10.48550/arxiv.2504.09246
Metadata
- Authors
- Wei, Y., Xia, C. S., & Zhang, L.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Venue
- Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2023-11-30
- Pages
- 172-184
- OpenAlex
- W4386436496
- Cited by
- 115
- References
- 45
- Citations by year
- 2026: 27 · 2025: 62 · 2024: 24 · 2023: 2
- FWCI
- 22.26
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (National Science Foundation) (CCF-2131943,CCF-2141474)
- Kwai Inc.
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Keywords
- automated program repair, large language models, patch synthesis, token pruning, autoregressive generation, Defects4J, bug fixes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
