CoChat
Sign Up
Cited by 202Software Testing and Debugging TechniquesSoftware Engineering ResearchSoftware Reliability and Analysis Research

Impact of Code Language Models on Automated Program Repair

Open access
Jiang, Nan, Liu, Kevin, Lutellier, Thibaud, Tan, Lin·2023·2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)
DOI 10.1109/icse48619.2023.00125
Open this paper in CoChat — free accountOpen PDF

Abstract

Automated program repair (APR) aims to help developers improve software reliability by generating patches for buggy programs. Although many code language models (CLM) are developed and effective in many software tasks such as code completion, there has been little comprehensive, in-depth work to evaluate CLMs' fixing capabilities and to fine-tune CLMs for the APR task. Firstly, this work is the first to evaluate ten CLMs on four APR benchmarks, which shows that surprisingly, the best CLM, as is, fixes 72% more bugs than the state-of-the-art deep-learning (DL)-based APR techniques. Secondly, one of the four APR benchmarks was created by us in this paper to avoid data leaking for a fair evaluation. Thirdly, it is the first work to fine-tune CLMs with APR training data, which shows that fine-tuning brings 31%-1,267% improvement to CLMs and enables them to fix 46%-164 % more bugs than existing DL-based APR techniques. Fourthly, this work studies the impact of buggy lines, showing that CLMs, as is, cannot make good use of the buggy lines to fix bugs, yet fine-tuned CLMs could potentially over-rely on buggy lines. Lastly, this work analyzes the size, time, and memory efficiency of different CLMs. This work shows promising directions for the APR domain, such as fine-tuning CLMs with APR-specific designs, and also raises awareness of fair and comprehensive evaluations of CLMs and calls for more transparent reporting of open-source repositories used in the pre-training data to address the data leaking problem.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icse48619.2023.00125

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. M. Monperrus, "The living review on automated program repair," 2020.
  2. M. Monperrus, "Automatic software repair: A bibliography," ACM Comput. Surv., vol. 51, no. 1, pp. 17:1-17:24, 2018. [Online]. Available: https://doi.org/10.1145/3105906 · DOI
  3. Y. Li, S. Wang, and T. N. Nguyen, "DLFix: Context-Based Code Transformation Learning for Automated Program Repair," in ICSE. ACM, 2020, p. 602-614.
  4. T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, "CoCoNuT: Combining Context-Aware Neural Translation Models Using Ensemble for Program Repair," in ISSTA. ACM, 2020, p. 101-114.
  5. N. Jiang, T. Lutellier, and L. Tan, "Cure: Code-aware neural machine translation for automatic program repair," in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 2021, pp. 1161-1173.
  6. S. Chakraborty, Y. Ding, M. Allamanis, and B. Ray, "Codit: Code editing with tree-based neural models," IEEE Transactions on Software Engineering, vol. 48, no. 4, pp. 1385-1399, 2022.
  7. H. Ye, M. Martinez, and M. Monperrus, "Neural program repair with execution-based backpropagation," in Proceedings of the International Conference on Software Engineering, 2022. [Online]. Available: http://arxiv.org/pdf/2105.04123 · arXiv
  8. Q. Zhu, Z. Sun, Y.-a. Xiao, W. Zhang, K. Yuan, Y. Xiong, and L. Zhang, "A syntax-guided edit decoder for neural program repair," in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. New York, NY, USA: Association for Computing Machinery, 2021, p. 341-353. [Online]. Available: https://doi.org/10.1145/3468264.3468544 · DOI
  9. N. Jiang, T. Lutellier, Y. Lou, L. Tan, D. Goldwasser, and X. Zhang, "Knod: Domain knowledge distilled tree decoder for automated program repair," in Proceedings of the International Conference on Software Engineering, 2023.
  10. Z. Chen, S. Kommrusch, and M. Monperrus, "Neural transfer learning for repairing security vulnerabilities in c code," IEEE Transactions on Software Engineering, 2022. [Online]. Available: http://arxiv.org/pdf/2104.08308 · arXiv
  11. H. Ye, M. Martinez, X. Luo, T. Zhang, and M. Monperrus, "Selfapr: Self-supervised program repair with test execution diagnostics," in Proceedings of ASE, 2022. [Online]. Available: http://arxiv.org/pdf/ 2203.12755
  12. K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, "TBar: Revisiting Template-Based Automated Program Repair," in ISSTA. ACM, 2019.
  13. Y. Yuan and W. Banzhaf, "ARJA: automated repair of java programs via multi-objective genetic programming," IEEE Trans. Software Eng., vol. 46, no. 10, pp. 1040-1067, 2020. [Online]. Available: https://doi.org/10.1109/TSE.2018.2874648 · DOI
  14. M. Wen, J. Chen, R. Wu, D. Hao, and S. Cheung, "Context-aware patch generation for better automated program repair," in Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 -June 03, 2018, M. Chaudron, I. Crnkovic, M. Chechik, and M. Harman, Eds. ACM, 2018, pp. 1-11. [Online]. Available: https://doi.org/10.1145/3180155.3180233 · DOI
  15. R. K. Saha, Y. Lyu, H. Yoshida, and M. R. Prasad, "ELIXIR: effective object oriented program repair," in Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 -November 03, 2017, G. Rosu, M. D. Penta, and T. N. Nguyen, Eds. IEEE Computer Society, 2017, pp. 648-659. [Online]. Available: https://doi.org/10.1109/ASE.2017.8115675 · DOI
  16. Y. Xiong, J. Wang, R. Yan, J. Zhang, S. Han, G. Huang, and L. Zhang, "Precise condition synthesis for program repair," in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), 2017, pp. 416-426.
  17. J. Xuan, M. Martinez, F. Demarco, M. Clement, S. R. L. Marcote, T. Durieux, D. L. Berre, and M. Monperrus, "Nopol: Automatic repair of conditional statement bugs in java programs," IEEE Trans. Software Eng., vol. 43, no. 1, pp. 34-55, 2017. [Online]. Available: https://doi.org/10.1109/TSE.2016.2560811 · DOI
  18. M. Martinez and M. Monperrus, "Ultra-large repair search space with automatically mined templates: The cardumen mode of astor," in Search-Based Software Engineering -10th International Symposium, SSBSE 2018, Montpellier, France, September 8-9, 2018, Proceedings, ser. Lecture Notes in Computer Science, T. E. Colanzi and P. McMinn, Eds., vol. 11036. Springer, 2018, pp. 65-86. [Online]. Available: https://doi.org/10.1007/978-3-319-99241-9 3 · DOI
  19. Y. Noller, R. Shariffdeen, X. Gao, and A. Roychoudhury, "Trust enhancement issues in program repair," in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE '22. New York, NY, USA: Association for Computing Machinery, 2022, p. 2228-2240. [Online]. Available: https://doi.org/10.1145/3510003. 3510040 · DOI
  20. W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, "Unified pre- training for program understanding and generation," in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics, Jun. 2021, pp. 2655-2668. [Online]. Available: https://aclanthology.org/2021.naacl-main.211
  21. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, "Language models are few- shot learners," CoRR, vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165 · arXiv
  22. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized BERT pretraining approach," CoRR, vol. abs/1907.11692, 2019. [Online]. Available: http://arxiv.org/abs/1907.11692 · arXiv
  23. J. Zhang, H. Zhang, C. Xia, and L. Sun, "Graph-bert: Only attention is needed for learning graph representations," CoRR, vol. abs/2001.05140, 2020. [Online]. Available: https://arxiv.org/abs/2001.05140 · arXiv
  24. S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zheng, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro, "Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model," CoRR, vol. abs/2201.11990, 2022. [Online]. Available: https://arxiv.org/abs/2201.11990 · arXiv
  25. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, "A conversational paradigm for program synthesis," arXiv preprint, 2022.
  26. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, "Evaluating large language models trained on code," CoRR, vol. abs/2107.03374, 2021. [Online]. Available: https://arxiv.org/abs/2107.03374 · arXiv
  27. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, "Codebert: A pre-trained model for programming and natural languages," CoRR, vol. abs/2002.08155, 2020. [Online]. Available: https://arxiv.org/abs/2002.08155 · arXiv
  28. D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, "Graphcodebert: Pre-training code representations with data flow," CoRR, vol. abs/2009.08366, 2020. [Online]. Available: https://arxiv.org/abs/2009.08366 · arXiv
  29. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, "Codexglue: A machine learning benchmark dataset for code understanding and generation," CoRR, vol. abs/2102.04664, 2021. · arXiv
  30. M. Tan, L. Tan, S. Dara, and C. Mayeux, "Online defect prediction for imbalanced data," in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2. IEEE, 2015, pp. 99-108.
  31. E.-S. A. Lee, S. Thillainathan, S. Nayak, S. Ranathunga, D. I. Adelani, R. Su, and A. D. McCarthy, "Pre-trained multilingual sequence-to- sequence models: A hope for low-resource language translation?" 2022. [Online]. Available: https://arxiv.org/abs/2203.08850 · arXiv
  32. H. Mehrafarin, S. Rajaee, and M. T. Pilehvar, "On the importance of data size in probing fine-tuned models," in Findings of the Association for Computational Linguistics: ACL 2022. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 228-238. [Online]. Available: https://aclanthology.org/2022.findings-acl.20
  33. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," CoRR, vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762 · arXiv
  34. H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, "Codesearchnet challenge: Evaluating the state of semantic code search," CoRR, vol. abs/1909.09436, 2019. [Online]. Available: http://arxiv.org/abs/1909.09436 · arXiv
  35. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, "The pile: An 800gb dataset of diverse text for language modeling," CoRR, vol. abs/2101.00027, 2021. [Online]. Available: https://arxiv.org/abs/2101.00027 · arXiv
  36. X. V. Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, R. Pasunuru, S. Shleifer, P. S. Koura, V. Chaudhary, B. O'Horo, J. Wang, L. Zettlemoyer, Z. Kozareva, M. T. Diab, V. Stoyanov, and X. Li, "Few-shot learning with multilingual language models," CoRR, vol. abs/2112.10668, 2021. [Online]. Available: https://arxiv.org/abs/2112.10668 · arXiv
  37. C. S. Xia and L. Zhang, "Less training, more repairing please: Revisiting automated program repair via zero-shot learning," in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 959-971. [Online]. Available: https: //doi.org/10.1145/3540250.3549101 · DOI · Paper page
  38. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, "fairseq: A fast, extensible toolkit for sequence modeling," in Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
  39. V. P. Dwivedi and X. Bresson, "A generalization of transformer networks to graphs," CoRR, vol. abs/2012.09699, 2020. [Online]. Available: https://arxiv.org/abs/2012.09699 · arXiv
  40. Z. Hu, X. Ma, Z. Liu, E. Hovy, and E. Xing, "Harnessing deep neural networks with logic rules," in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 2410-2420. [Online]. Available: https://aclanthology.org/P16-1228
  41. C.-Y. Lin and F. J. Och, "ORANGE: a method for evaluating automatic evaluation metrics for machine translation," in COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics. Geneva, Switzerland: COLING, aug 23-aug 27 2004, pp. 501-507. [Online]. Available: https://aclanthology.org/C04-1072
  42. A. Mesbah, A. Rice, E. Johnston, N. Glorioso, and E. Aftandilian, "DeepDelta: Learning to Repair Compilation Errors," in ESEC/FSE. ACM, 2019, pp. 925-936.
  43. R. Gupta, S. Pal, A. Kanade, and S. Shevade, "Deepfix: Fixing Common C Language Errors by Deep Learning," in AAAI, 2017, pp. 1345-1351.
  44. E. A. Santos, J. C. Campbell, A. Hindle, and J. N. Amaral, "Finding and Correcting Syntax Errors Using Recurrent Neural Networks," PeerJ PrePrints, vol. 5, p. e3123v1, 2017.

Cited by

Sorted by citations

Authors
Jiang, N., Liu, K., Lutellier, T., & Tan, L.
Affiliations
  • Purdue University West Lafayette (US)
  • Lynbrook High School,San Jose,USA
  • University of Alberta (CA)
ORCID
Lutellier, Thibaud
Venue
2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)
Publisher
IEEE
Published
2023-05
Pages
1430-1442
ISSN
0270-5257 (ISSN-L)
DOI
10.1109/icse48619.2023.00125
OpenAlex
W4384304728
Cited by
202
References
12
Citations by year
2026: 41 · 2025: 81 · 2024: 62 · 2023: 18
FWCI
39.07
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
Keywords
automated program repair, code language models, fine-tuning, bug fixes, benchmark evaluation, data leakage, model efficiency, software reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review