CoChat
Sign Up
PreprintCited by 12Software Testing and Debugging TechniquesSoftware Engineering ResearchSoftware System Performance and Reliability

The DOI matches its DataCite record; its reference list is not available.

CigaR: Cost-efficient Program Repair with LLMs

Open access
Hidvégi, Dávid, Etemadi, Khashayar, Bobadilla, Sofia, Monperrus, Martin·2024·arXiv
DOI 10.48550/arxiv.2402.06598arXiv 2402.06598
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLM) have proven to be effective at automated program repair (APR). However, using LLMs can be costly, with companies invoicing users by the number of tokens. In this paper, we propose CigaR, the first LLM-based APR tool that focuses on minimizing the repair cost. CigaR works in two major steps: generating a first plausible patch and multiplying plausible patches. CigaR optimizes the prompts and the prompt setting to maximize the information given to LLMs using the smallest possible number of tokens. Our experiments on 429 bugs from the widely used Defects4J and HumanEval-Java datasets shows that CigaR reduces the token cost by 73%. On average, CigaR spends 127k tokens per bug while the baseline uses 467k tokens per bug. On the subset of bugs that are fixed by both, CigaR spends 20k per bug while the baseline uses 608k tokens, a cost saving of 96%. Our extensive experiments show that CigaR is a cost-effective LLM-based program repair tool that uses a low number of tokens to automatically generate patches.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.06598

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 12

Research works printed but not linked: 51

  1. M. Monperrus, "Automatic software repair: A bibliography," ACM Computing Surveys (CSUR), vol. 51, no. 1, pp. 1-24, 2018.
  2. R. C. Seacord, D. Plakosh, and G. A. Lewis, Modernizing legacy systems: software technologies, engineering processes, and business practices. Addison-Wesley Professional, 2003.
  3. C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer, "A Systematic Study of Automated Program Repair: Fixing 55 Out of 105 Bugs for $8 Each," in Proceedings of the International Conference on Software Engineering, 2012, pp. 3-13.
  4. C. S. Xia and L. Zhang, "Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt," arXiv preprint arXiv:2304.00385, 2023. · arXiv
  5. N. Jiang, K. Liu, T. Lutellier, and L. Tan, "Impact of code language models on automated program repair," in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). Los Alamitos, CA, USA: IEEE Computer Society, may 2023, pp. 1430- 1442. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ ICSE48619.2023.00125 · DOI · Paper page
  6. L. Chen, M. Zaharia, and J. Zou, "Frugalgpt: How to use large language models while reducing cost and improving performance," arXiv preprint arXiv:2305.05176, 2023. · arXiv
  7. Q. Zhang, T. Zhang, J. Zhai, C. Fang, B. Yu, W. Sun, and Z. Chen, "A critical review of large language model on software engineering: An example from chatgpt and automated program repair," arXiv preprint arXiv:2310.08879, 2023. · arXiv
  8. R. Just, D. Jalali, and M. D. Ernst, "Defects4j: A database of existing faults to enable controlled testing studies for java programs," in Pro- ceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437-440.
  9. A. S. Luccioni, S. Viguier, and A.-L. Ligozat, "Estimating the carbon footprint of bloom, a 176b parameter language model," Journal of Machine Learning Research, vol. 24, no. 253, pp. 1-15, 2023.
  10. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
  11. A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, "Large language models in medicine," Nature medicine, vol. 29, no. 8, pp. 1930-1940, 2023.
  12. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  13. M. Zhang and J. Li, "A commentary of gpt-3 in mit technology review 2021," Fundamental Research, vol. 1, no. 6, pp. 831-833, 2021.
  14. L. K. Umapathi, A. Pal, and M. Sankarasubbu, "Med-halt: Medical domain hallucination test for large language models," arXiv preprint arXiv:2307.15343, 2023. · arXiv
  15. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485-5551, 2020.
  16. A. Roberts, C. Raffel, and N. Shazeer, "How much knowledge can you pack into the parameters of a language model?" arXiv preprint arXiv:2002.08910, 2020. · arXiv
  17. Z. Cheng, J. Kasai, and T. Yu, "Batch prompting: Efficient inference with large language model apis," arXiv preprint arXiv:2301.08721, 2023. · arXiv
  18. C. Wang, S. X. Liu, and A. H. Awadallah, "Cost-effective hyperparam- eter optimization for large language model generation inference," arXiv preprint arXiv:2303.04673, 2023. · arXiv
  19. J. Oppenlaender and J. Hämäläinen, "Mapping the challenges of hci: An application and evaluation of chatgpt and gpt-4 for cost-efficient question answering," arXiv preprint arXiv:2306.05036, 2023. · arXiv
  20. H. Ye, J. Gu, M. Martinez, T. Durieux, and M. Monperrus, "Automated classification of overfitting patches with statically extracted code fea- tures," IEEE Transactions on Software Engineering, vol. 48, no. 8, pp. 2920-2938, 2021.
  21. A. Ghanbari and A. Marcus, "Patch correctness assessment in automated program repair based on the impact of patches on production and test code," in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 654-665.
  22. S. Fakhoury, S. Chakraborty, M. Musuvathi, and S. K. Lahiri, "Towards generating functionally correct code edits from natural language issue descriptions," arXiv preprint arXiv:2304.03816, 2023. · arXiv
  23. Y. Zhang, Z. Jin, Y. Xing, and G. Li, "Steam: Simulating the interactive behavior of programmers for automatic bug fixing," arXiv preprint arXiv:2308.14460, 2023. · arXiv
  24. K. Huang, Z. Xu, S. Yang, H. Sun, X. Li, Z. Yan, and Y. Zhang, "A survey on automated program repair techniques," arXiv preprint arXiv:2303.18184, 2023. · arXiv
  25. C. S. Xia and L. Zhang, "Less training, more repairing please: revisiting automated program repair via zero-shot learning," in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 959- 971.
  26. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., "Codebert: A pre-trained model for programming and natural languages," arXiv preprint arXiv:2002.08155, 2020. · arXiv
  27. C. H. Lampert, H. Nickisch, and S. Harmeling, "Learning to detect unseen object classes by between-class attribute transfer," in 2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009, pp. 951-958.
  28. Q. Zhang, C. Fang, T. Zhang, B. Yu, W. Sun, and Z. Chen, "Gamma: Re- visiting template-based automated program repair via mask prediction," arXiv preprint arXiv:2309.09308, 2023. · arXiv
  29. H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, "Examining zero-shot vulnerability repair with large language models," in 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023, pp. 2339-2356.
  30. J. Cao, M. Li, M. Wen, and S.-c. Cheung, "A study on prompt design, advantages and limitations of chatgpt for deep learning program repair," arXiv preprint arXiv:2304.08191, 2023. · arXiv
  31. C. S. Xia, Y. Wei, and L. Zhang, "Automated program repair in the era of large pre-trained language models," in Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery, 2023.
  32. N. Nashid, M. Sintaha, and A. Mesbah, "Retrieval-based prompt se- lection for code-related few-shot learning," in Proceedings of the 45th International Conference on Software Engineering (ICSE'23), 2023.
  33. N. Reimers and I. Gurevych, "Sentence-bert: Sentence embeddings using siamese bert-networks," arXiv preprint arXiv:1908.10084, 2019. · arXiv
  34. S. Robertson, H. Zaragoza et al., "The probabilistic relevance frame- work: Bm25 and beyond," Foundations and Trends® in Information Retrieval, vol. 3, no. 4, pp. 333-389, 2009.
  35. T. Ahmed and P. Devanbu, "Better patching using llm prompting, via self-consistency," in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Computer Society, 2023, pp. 1742-1746.
  36. S. Chakraborty and B. Ray, "On multi-modal learning of editing source code," in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 443-455.
  37. W. Yuan, Q. Zhang, T. He, C. Fang, N. Q. V. Hung, X. Hao, and H. Yin, "Circle: Continual repair across programming languages," in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 678-690.
  38. D. Sobania, M. Briesch, C. Hanna, and J. Petke, "An analysis of the automatic bug fixing performance of chatgpt," arXiv preprint arXiv:2301.08653, 2023. · arXiv
  39. Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan, "Automated repair of programs from large language models," in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1469-1481.
  40. C. Liu, P. Cetin, Y. Patodia, S. Chakraborty, Y. Ding, and B. Ray, "Automated code editing with search-generate-modify," arXiv preprint arXiv:2306.06490, 2023. · arXiv
  41. M. M. A. Haque, W. U. Ahmad, I. Lourentzou, and C. Brown, "Fixeval: Execution-based evaluation of program fixes for programming problems," in 2023 IEEE/ACM International Workshop on Automated Program Repair (APR). IEEE, 2023, pp. 11-18.
  42. Y. Wu, N. Jiang, H. V. Pham, T. Lutellier, J. Davis, L. Tan, P. Babkin, and S. Shah, "How effective are neural networks for fixing security vulnerabilities," arXiv preprint arXiv:2305.18607, 2023. · arXiv
  43. Q. Zhang, C. Fang, B. Yu, W. Sun, T. Zhang, and Z. Chen, "Pre-trained model-based automated software vulnerability repair: How far are we?" IEEE Transactions on Dependable and Secure Computing, 2023.
  44. E. Mashhadi and H. Hemmati, "Applying codebert for automated pro- gram repair of java simple bugs," in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021, pp. 505-509.
  45. A. Zirak and H. Hemmati, "Improving automated program repair with domain adaptation," ACM Transactions on Software Engineering and Methodology, 2022.
  46. D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, "Incoder: A generative model for code infilling and synthesis," arXiv preprint arXiv:2204.05999, 2022. · arXiv
  47. K. Huang, X. Meng, J. Zhang, Y. Liu, W. Wang, S. Li, and Y. Zhang, "An empirical study on fine-tuning large language models of code for automated program repair," in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Computer Society, 2023, pp. 1162-1174.
  48. J. Zhang, R. Krishna, A. H. Awadallah, and C. Wang, "Ecoassistant: Using llm assistant more affordably and accurately," arXiv preprint arXiv:2310.03046, 2023. · arXiv
  49. M. A. Arefeen, B. Debnath, and S. Chakradhar, "Leancontext: Cost- efficient domain-specific question answering using llms," arXiv preprint arXiv:2309.00841, 2023. · arXiv
  50. J. Mu, X. L. Li, and N. Goodman, "Learning to compress prompts with gist tokens," arXiv preprint arXiv:2304.08467, 2023. · arXiv
  51. J. Lin, M. Diesendruck, L. Du, and R. Abraham, "Batchprompt: Ac- complish more with less," arXiv preprint arXiv:2309.00384, 2023. · arXiv

Metadata

Authors
Hidvégi, D., Etemadi, K., Bobadilla, S., & Monperrus, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.06598
arXiv
2402.06598
OpenAlex
W4391766982
Cited by
12
References
0
Citations by year
2026: 4 · 2025: 8
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
Keywords
automated program repair, large language models, token cost reduction, prompt optimization, patch generation, Defects4J

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review