CoChat
Sign Up
Cited by 338Software Testing and Debugging TechniquesSoftware Engineering ResearchEvolutionary Algorithms and Applications

An Analysis of the Automatic Bug Fixing Performance of ChatGPT

Open access
Sobania, Dominik, Briesch, Martin, Hanna, Carol, Petke, Justyna·2023·2023 IEEE/ACM International Workshop on Automated Program Repair (APR)
DOI 10.1109/apr59189.2023.00012
Open this paper in CoChat — free accountOpen PDF

Abstract

To support software developers in finding and fixing software bugs, several automated program repair techniques have been introduced. Given a test suite, standard methods usually either synthesize a repair, or navigate a search space of software edits to find test-suite passing variants. Recent program repair methods are based on deep learning approaches. One of these novel methods, which is not primarily intended for automated program repair, but is still suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard bug fixing benchmark set, QuixBugs, and compare the performance with the results of several other approaches reported in the literature. We find that ChatGPT's bug fixing performance is competitive to the common deep learning approaches CoCoNut and Codex and notably better than the results reported for the standard program repair approaches. In contrast to previous approaches, ChatGPT offers a dialogue system through which further information, e.g., the expected output for a certain input or an observed error message, can be entered. By providing such hints to ChatGPT, its success rate can be further increased, fixing 31 out of 40 bugs, outperforming state-of-the-art.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/apr59189.2023.00012

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SWE-bench unites many separate software engineering tasks under a single task formulation that faithfully mirrors practical software engineering.

    “We use SWE-bench because it unites many separate SE tasks, such as automated program repair [10, 40, 55], bug localization [4, 58], and testing [22, 46, 56] under a single task formulation that faithfully mirrors practical SE.”

    From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [40]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 23 · Cited by 338

Research works printed but not linked: 20

  1. W. E. Wong, X. Li, and P. A. Laplante, "Be more familiar with our enemies and pave the way forward: A review of the roles bugs played in software failures," Journal of Systems and Software, vol. 133, pp. 68-94, 2017.
  2. C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, "GenProg: A generic method for automatic software repair," Ieee transactions on software engineering, vol. 38, no. 1, pp. 54-72, 2011.
  3. L. Gazzola, D. Micucci, and L. Mariani, "Automatic software repair: a survey," in Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 -June 03, 2018, M. Chaudron, I. Crnkovic, M. Chechik, and M. Harman, Eds. ACM, 2018, p. 1219. [Online]. Available: https://doi.org/10.1145/3180155.3182526 · DOI
  4. S. O. Haraldsson, J. R. Woodward, A. E. Brownlee, and K. Siggeirs- dottir, "Fixing bugs in your sleep: How genetic improvement became an overnight success," in Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2017, pp. 1513-1520.
  5. J. Petke, S. O. Haraldsson, M. Harman, W. B. Langdon, D. R. White, and J. R. Woodward, "Genetic improvement of software: a comprehensive survey," IEEE Transactions on Evolutionary Computation, vol. 22, no. 3, pp. 415-432, 2017.
  6. Q. Zhang, C. Fang, Y. Ma, W. Sun, and Z. Chen, "A survey of learning-based automated program repair," 2023. [Online]. Available: https://arxiv.org/abs/2301.03270 · arXiv
  7. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in neural information processing systems, 2017, pp. 5998-6008.
  8. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., "CodeBERT: A pre-trained model for programming and natural languages," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, 2020, pp. 1536-1547.
  9. C. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan, "PyMT5: Multi-mode translation of natural language and Python code with transformers," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 9052- 9065.
  10. D. Sobania, M. Briesch, and F. Rothlauf, "Choose your programming copilot: a comparison of the program synthesis performance of GitHub Copilot and genetic programming," in Proceedings of the Genetic and Evolutionary Computation Conference, 2022, pp. 1019-1027.
  11. D. Lin, J. Koppel, A. Chen, and A. Solar-Lezama, "QuixBugs: A multi- lingual program repair benchmark set based on the Quixey Challenge," in Proceedings Companion of the 2017 ACM SIGPLAN international conference on systems, programming, languages, and applications: software for humanity, 2017, pp. 55-56.
  12. H. Ye, M. Martinez, T. Durieux, and M. Monperrus, "A comprehensive study of automatic program repair on the QuixBugs benchmark," Journal of Systems and Software, vol. 171, p. 110825, 2021.
  13. T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, "CoCoNuT: combining context-aware neural translation models using ensemble for program repair," in ISSTA '20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020, S. Khurshid and C. S. Pasareanu, Eds. ACM, 2020, pp. 101-114. [Online]. Available: https://doi.org/10.1145/3395363.3397369 · DOI
  14. J. A. Prenner, H. Babii, and R. Robbes, "Can OpenAI's codex fix bugs? an evaluation on QuixBugs," in Proceedings of the Third International Workshop on Automated Program Repair, 2022, pp. 69-75.
  15. Y. Yuan and W. Banzhaf, "ARJA: Automated repair of java programs via multi-objective genetic programming," IEEE Transactions on software engineering, vol. 46, no. 10, pp. 1040-1067, 2018.
  16. M. Martinez and M. Monperrus, "Ultra-large repair search space with automatically mined templates: The cardumen mode of astor," in Inter- national Symposium on Search Based Software Engineering. Springer, 2018, pp. 65-86.
  17. T. Durieux and M. Monperrus, "Dynamoth: dynamic code synthesis for automatic program repair," in Proceedings of the 11th International Workshop on Automation of Software Test, 2016, pp. 85-91.
  18. M. Martinez and M. Monperrus, "Astor: Exploring the design space of generate-and-validate program repair beyond GenProg," Journal of Systems and Software, vol. 151, pp. 65-80, 2019.
  19. J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus, "Nopol: Automatic repair of conditional statement bugs in Java programs," IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 34-55, 2016.
  20. B. Cornu, T. Durieux, L. Seinturier, and M. Monperrus, "NPEfix: Automatic runtime repair of null pointer exceptions in Java," arXiv preprint arXiv:1512.07423, 2015. · arXiv

Authors
Sobania, D., Briesch, M., Hanna, C., & Petke, J.
Affiliations
  • Johannes Gutenberg University Mainz (DE)
  • University College London (GB)
ORCID
Venue
2023 IEEE/ACM International Workshop on Automated Program Repair (APR)
Publisher
IEEE
Published
2023-05
Pages
23-30
DOI
10.1109/apr59189.2023.00012
OpenAlex
W4385302156
Cited by
338
References
23
Citations by year
2026: 35 · 2025: 71 · 2024: 129 · 2023: 103
FWCI
65.37
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/ACM International Workshop on Automated Program Repair (APR) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • UK Research and Innovation (EP/P023991/1)
  • Engineering and Physical Sciences Research Council (EP/P023991/1)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
automated program repair, ChatGPT, bug fixes, large language models, coconut, codices

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review