CoChat
Sign Up
Cited by 126Software Engineering ResearchSoftware Testing and Debugging TechniquesTeaching and Learning Programming

Repair Is Nearly Generation: Multilingual Program Repair with LLMs

Open access
Joshi, Harshit, Cambronero Sanchez, José, Gulwani, Sumit, Le, Vu, Verbruggen, Gust, Radiček, Ivan·2023·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v37i4.25642
Open this paper in CoChat — free accountOpen PDF

Abstract

Most programmers make mistakes when writing code. Some of these mistakes are small and require few edits to the original program – a class of errors recently termed last mile mistakes. These errors break the flow for experienced developers and can stump novice programmers. Existing automated repair techniques targeting this class of errors are language-specific and do not easily carry over to new languages. Transferring symbolic approaches requires substantial engineering and neural approaches require data and retraining. We introduce RING, a multilingual repair engine powered by a large language model trained on code (LLMC) such as Codex. Such a multilingual engine enables a flipped model for programming assistance, one where the programmer writes code and the AI assistance suggests fixes, compared to traditional code suggestion technology. Taking inspiration from the way programmers manually fix bugs, we show that a prompt-based strategy that conceptualizes repair as localization, transformation, and candidate ranking, can successfully repair programs in multiple languages with minimal effort. We present the first results for such a multilingual repair engine by evaluating on 6 different languages and comparing performance to language-specific repair engines. We show that RING can outperform language-specific repair engines for three of these languages.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v37i4.25642

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Without adequate context to LLMs via prompt crafting, they may only serve as next code token predictors, without understanding the overall task.

    “However, research has shown that without providing adequate context to LLMs via prompt crafting, they may only serve as”

    From Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code (2024) · cites this paper as [44]

    The Related Work section says autoregressive models such as Codex are trained to predict the next token. The RQ3 Code Transformation section reports that relevant examples in prompts improve repair performance. The paper does not establish that, without adequate prompt context, LLMs only predict code tokens or fail to understand the overall task. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Large Language Models trained on code have demonstrated significant effectiveness in generating program fixes.

    “Large Language Models (LLMS) trained on code (LLMCS) have demonstrated significant effectiveness in generating program fixes (Joshi et al. 2023; Xia, Ding, and Zhang 2023; Wei, Xia, and Zhang 2023; Fan et al. 2023; Xia, Wei, and Zhang 2023; Brancas, Manquinho, and Martins 2024).”

    From Counterexample Guided Program Repair Using Zero-Shot Learning and MaxSAT-based Fault Localization (2025) · cites this paper as Joshi et al. 2023

  • The syntax used in our prompts is similar to that in other works on LLM-driven program repair.

    “The syntax used in our prompts is similar to that in other works on LLM-driven program repair (Joshi et al. 2023).”

    From Counterexample Guided Program Repair Using Zero-Shot Learning and MaxSAT-based Fault Localization (2025) · cites this paper as Joshi et al. 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 43 · Cited by 126

Research works printed but not linked: 35

  1. Ahmed, T.; Ledesma, N. R.; and Devanbu, P. 2021. SYN- FIX: Automatically Fixing Syntax Errors using Compiler Diagnostics. arXiv preprint arXiv:2104.14671. · arXiv
  2. Ahmed, U. Z.; Kumar, P.; Karkare, A.; Kar, P.; and Gulwani, S. 2018. Compilation error repair: for the student programs, from the student programs. In Proceedings of the 40th In- ternational Conference on Software Engineering: Software Engineering Education and Training, 78-87.
  3. Altadmri, A.; and Brown, N. C. 2015. 37 million compila- tions: Investigating novice programming mistakes in large- scale student data. In Proceedings of the 46th ACM technical symposium on computer science education, 522-527.
  4. Arcuri, A. 2008. On the automation of fixing software bugs. In Companion of the 30th international conference on Soft- ware engineering, 1003-1006.
  5. Bareiß, P.; Souza, B.; d'Amorim, M.; and Pradel, M. 2022. Code Generation Tools (Almost) for Free? A Study of Few- Shot, Pre-Trained Language Models on Code. arXiv preprint arXiv:2206.01335. · arXiv
  6. Bavishi, R.; Joshi, H.; Cambronero, J.; Fariha, A.; Gulwani, S.; Le, V.; Radiček, I.; and Tiwari, A. 2022. Neurosym- bolic Repair for Low-Code Formula Languages. Proc. ACM Program. Lang., 6(OOPSLA2).
  7. Berabi, B.; He, J.; Raychev, V.; and Vechev, M. 2021. Tfix: Learning to fix coding errors with a text-to-text transformer. In International Conference on Machine Learning, 780-791.
  8. PMLR.
  9. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877- 1901.
  10. Chowdhury, J. R.; Zhuang, Y.; and Wang, S. 2022. Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuning. Proceedings of the AAAI Con- ference on Artificial Intelligence, 36(10): 10535-10544.
  11. Debroy, V.; and Wong, W. E. 2010. Using Mutation to Au- tomatically Suggest Fixes for Faulty Programs. 2010 Third International Conference on Software Testing, Verification and Validation, 65-74.
  12. Diekmann, L.; and Tratt, L. 2020. Don't Panic! Better, Fewer, Syntax Errors for LR Parsers. In 34th European Conference on Object-Oriented Programming, ECOOP 2020, volume 166 of LIPIcs, 6:1-6:32.
  13. Dormann, C. F. 2020. Calibration of probability predictions from machine-learning and statistical models. Global ecology and biogeography, 29(4): 760-765.
  14. Drori, I.; Zhang, S.; Shuttleworth, R.; Tang, L.; Lu, A.; Ke, E.; Liu, K.; Chen, L.; Tran, S.; Cheng, N.; et al. 2022. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences, 119(32): e2123433119.
  15. Drosos, I.; Guo, P. J.; and Parnin, C. 2017. HappyFace: Identifying and predicting frustrating obstacles for learning programming at scale. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 171- 179. IEEE.
  16. Feng, Z.; Guo, D.; Tang, D.; Duan, N.; Feng, X.; Gong, M.; Shou, L.; Qin, B.; Liu, T.; Jiang, D.; et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. · arXiv
  17. Gazzola, L.; Micucci, D.; and Mariani, L. 2019. Automatic Software Repair: A Survey. IEEE Transactions on Software Engineering, 45: 34-67.
  18. Goues, C. L.; Pradel, M.; and Roychoudhury, A. 2019. Auto- mated program repair. Communications of the ACM, 62(12): 56-65.
  19. Gupta, R.; Pal, S.; Kanade, A.; and Shevade, S. K. 2017. DeepFix: Fixing Common C Language Errors by Deep Learn- ing. In AAAI.
  20. Hajipour, H.; Bhattacharyya, A.; and Fritz, M. 2020. Sam- pleFix: Learning to Correct Programs by Efficient Sampling of Diverse Fixes. In NeurIPS 2020 Workshop on Computer- Assisted Programming.
  21. Inala, J. P.; Wang, C.; Yang, M.; Codas, A.; Encarnación, M.; Lahiri, S. K.; Musuvathi, M.; and Gao, J. 2022. Fault-Aware Neural Code Rankers. arXiv preprint arXiv:2206.03865. · arXiv
  22. Murphy, L.; Lewandowski, G.; McCauley, R.; Simon, B.; Thomas, L.; and Zander, C. 2008. Debugging: the good, the bad, and the quirky-a qualitative analysis of novices' strategies. ACM SIGCSE Bulletin, 40(1): 163-167.
  23. Nguyen, H. D. T.; Qi, D.; Roychoudhury, A.; and Chandra, S. 2013. SemFix: Program repair via semantic analysis. International Conference on Software Engineering, 772-781.
  24. Nixon, J.; Dusenberry, M. W.; Zhang, L.; Jerfel, G.; and Tran, D. 2019. Measuring Calibration in Deep Learning. In CVPR Workshops, volume 2.
  25. Parihar, S.; Dadachanji, Z.; Singh, P. K.; Das, R.; Karkare, A.; and Bhattacharya, A. 2017. Automatic grading and feedback using program repair for introductory programming courses. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, 92-97.
  26. Prenner, J. A.; and Robbes, R. 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. arXiv preprint arXiv:2111.03922. · arXiv
  27. Pu, Y.; Narasimhan, K.; Solar-Lezama, A.; and Barzilay, R. 2016. sk p: a neural program corrector for MOOCs. In Companion Proceedings of the 2016 ACM SIGPLAN Inter- national Conference on Systems, Programming, Languages and Applications: Software for Humanity, 39-40.
  28. Qi, Y.; Mao, X.; Lei, Y.; Dai, Z.; and Wang, C. 2014. The strength of random search on automated program repair. In Proceedings of the 36th International Conference on Software Engineering, 254-265.
  29. Shannon, C. E. 1948. A mathematical theory of communica- tion. The Bell system technical journal, 27(3): 379-423.
  30. Tómasdóttir, K. F.; Aniche, M.; and Van Deursen, A. 2018. The adoption of javascript linters in practice: A case study on eslint. IEEE Transactions on Software Engineering, 46(8): 863-891.
  31. Wexelblat, R. L. 1976. Maxims for malfeasant designers, or how to design languages to make programming as difficult as possible. In Proceedings of the 2nd international conference on Software engineering, 331-336.
  32. Yasunaga, M.; and Liang, P. 2020. Graph-based, self- supervised program repair from diagnostic feedback. In In- ternational Conference on Machine Learning, 10799-10808.
  33. PMLR.
  34. PMLR.
  35. Zhong, H.; and Su, Z. 2015. An Empirical Study on Real Bug Fixes. 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, 1: 913-923.

Authors
Joshi, H., Cambronero Sanchez, J., Gulwani, S., Le, V., Verbruggen, G., & Radiček, I.
Affiliations
  • Microsoft (India) (IN)
  • Microsoft (United States) (US)
  • Microsoft (Belgium) (BE)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2023-06-26
Volume
37
Issue
4
Pages
5131-5140
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v37i4.25642
OpenAlex
W4382239980
Cited by
126
References
43
Citations by year
2026: 23 · 2025: 44 · 2024: 40 · 2023: 19
FWCI
32.38
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
program repair, large language models, bug localization, program transformation, candidate ranking, programming assistance, rings

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review