CoChat
Sign Up
Cited by 79Software Testing and Debugging TechniquesSoftware Engineering ResearchAdvanced Software Engineering Methodologies

A critical review on the evaluation of automated program repair systems

Open access
Liu, Kui, Li, Li, Koyuncu, Anil, Kim, Dongsun, Liu, Zhe, Klein, Jacques, Bissyandé, Tegawendé F.·2021·Journal of Systems and Software
DOI 10.1016/j.jss.2020.110817
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jss.2020.110817

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. Bhatia, S., Singh, R., Automated correction for syntax errors in programming assignments using recurrent neural networks, arXiv preprint arXiv:1603. 06129.
  2. Bohme, M., Soremekun, E.O., Chattopadhyay, S., Ugherughe, E., Zeller, A., 2017. Where is the bug and how is it fixed? An experiment with practitioners. In: Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. ACM, pp. 117-128. http://dx.doi.org/10.1145/3106237.3106255. · DOI
  3. Campos, J., Ge, Y., Albunian, N., Fraser, G., Eler, M., Arcuri, A., 2018. An empirical evaluation of evolutionary algorithms for unit test suite generation. Inf. Softw. Technol. 104, 207-235. http://dx.doi.org/10.1016/j.infsof.2018.08.010. · DOI
  4. Chen, Z., Kommrusch, S., Tufano, M., Pouchet, L., Poshyvanyk, D., Monperrus, M., 2019. SequenceR: Sequence-to-sequence learning for end-to-end program repair. IEEE Trans. Softw. Eng. 1. http://dx.doi.org/10.1109/TSE.2019.2940179. · DOI
  5. Chen, L., Pei, Y., Furia, C.A., 2017. Contract-based program repair without the contracts. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE, pp. 637-647. http://dx.doi.org/10. 1109/ASE.2017.8115674. · DOI
  6. Chen, L., Pei, Y., Furia, C.A., 2020. Contract-based program repair without the contracts: An extended study. IEEE Trans. Softw. Eng. 1. http://dx.doi.org/10. 1109/TSE.2020.2970009. · DOI
  7. Durieux, T., Cornu, B., Seinturier, L., Monperrus, M., 2017. Dynamic patch generation for null pointer exceptions using metaprogramming. In: Proceed- ings of the 24th International Conference on Software Analysis, Evolution and Reengineering. IEEE, pp. 349-358. http://dx.doi.org/10.1109/SANER.2017. 7884635. · DOI
  8. Falleri, J.-R., Morandat, F., Blanc, X., Martinez, M., Monperrus, M., 2014. Fine- grained and accurate source code differencing. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering. ACM, pp. 313-324. http://dx.doi.org/10.1145/2642937.2642982. · DOI
  9. Fry, Z.P., Landau, B., Weimer, W., 2012. A human study of patch maintainability. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, ISSTA 2012. ACM, New York, NY, USA, pp. 177-187. http: //dx.doi.org/10.1145/04000800.2336775. · DOI
  10. Ghanbari, A., Benton, S., Zhang, L., 2019. Practical program repair via bytecode mutation. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 19-30. http://dx.doi.org/10.1145/ 3293882.3330559. · DOI
  11. Gupta, R., Pal, S., Kanade, A., Shevade, S., 2017. DeepFix: Fixing common c language errors by deep learning. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, pp. 1345-1351. http://dx. doi.org/10.5555/3298239.3298436. · DOI
  12. Hua, J., Zhang, M., Wang, K., Khurshid, S., 2018. Towards practical program repair with on-demand candidate generation. In: Proceedings of the 40th International Conference on Software Engineering. ACM, pp. 12-23. http: //dx.doi.org/10.1145/3180155.3180245. · DOI
  13. Just, R., Parnin, C., Drosos, I., Ernst, M.D., 2018. Comparing developer-provided to user-provided tests for fault localization and automated program repair. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, pp. 287-297. http://dx.doi.org/10.1145/3213846. 3213870. · DOI
  14. Kim, D., Nam, J., Song, J., Kim, S., 2013. Automatic patch generation learned from human-written patches. In: Proceedings of the 35th International Conference on Software Engineering. IEEE, pp. 802-811. http://dx.doi.org/10.1109/ICSE. 2013.6606626. · DOI
  15. Koyuncu, A., Liu, K., Bissyandé, T.F., Kim, D., Klein, J., Monperrus, M., Y. L, 2020. Traon fixminer: Mining relevant fix patterns for automated program repair. Empir. Softw. Eng. 25 (3), 1980-2024. http://dx.doi.org/10.1007/s10664-019- 09780-z. · DOI
  16. Koyuncu, A., Liu, K., Bissyandé, T.F., Kim, D., Monperrus, M., Klein, J., Traon, Y.L., 2019. iFixR: bug report driven program repair. In: Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, pp. 314-325. http://dx.doi.org/10.1145/3338906.3338935. · DOI
  17. Le, X.B.D., Lo, D., Le Goues, C., 2016. History driven program repair. In: Proceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering. IEEE, pp. 213-224. http://dx.doi.org/10.1109/ SANER.2016. · DOI
  18. Le Goues, C., Dewey-Vogt, M., Forrest, S., Weimer, W., 2012a. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In: 2012 34th International Conference on Software Engineering. pp. 3-13. http://dx.doi.org/10.1109/ICSE.2012.6227211. · DOI
  19. Le Goues, C., Nguyen, T., Forrest, S., Weimer, W., 2012b. GenProg: A generic method for automatic software repair. IEEE Trans. Softw. Eng. 38 (1), 54-72. http://dx.doi.org/10.1109/TSE.2011. · DOI
  20. Lee, J., Kim, D., Bissyandé, T.F., Jung, W., Le Traon, Y., 2018. Bench4bl re- producibility study on the performance of ir-based bug localization. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018. ACM, New York, NY, USA, pp. 61-72. http://dx.doi.org/10.1145/3213846.3213856. · DOI
  21. Liu, K., Kim, D., Bissyandé, T.F., Yoo, S., Le Traon, Y., 2018a. Mining fix patterns for findbugs violations. IEEE Trans. Softw. Eng. 1. http://dx.doi.org/10.1109/ TSE.2018.2884955. · DOI
  22. Liu, K., Kim, D., Koyuncu, A., Li, L., Bissyandé, T.F., Le Traon, Y., 2018b. A closer look at real-world patches. In: Proceedings of the 34th IEEE International Conference on Software Maintenance and Evolution. IEEE, pp. 275-286. http://dx.doi.org/10.1109/ICSME.2018.00037. · DOI
  23. Liu, K., Koyuncu, A., Bissyandé, T.F., Kim, D., Klein, J., Traon, Y.L., 2019a. You cannot fix what you cannot find! an investigation of fault localization bias in benchmarking automated program repair systems. In: Proceedings of the 12th IEEE International Conference on Software Testing, Verification and Validation. IEEE, pp. 102-113. http://dx.doi.org/10.1109/ICST.2019.00020. · DOI
  24. Liu, K., Koyuncu, A., Kim, D., Bissyandé, T.F., 2019b. AVATAR: fixing semantic bugs with fix patterns of static analysis violations. In: Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, pp. 456-467. http://dx.doi.org/10.1109/SANER.2019.8667970. · DOI
  25. Liu, K., Koyuncu, A., Kim, D., Bissyandé, T.F., 2019c. Tbar: Revisiting template- based automated program repair. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 31-42. http: //dx.doi.org/10.1145/3293882.3330577. · DOI
  26. Liu, K., Koyuncu, A., Kim, K., Kim, D., Bissyandé, T.F., 2018c. Lsrepair: Live search of fix ingredients for automated program repair. In: Proceedings of the 25th Asia-Pacific Software Engineering Conference ERA Track. pp. 658-662. http://dx.doi.org/10.1109/APSEC.2018.00085. · DOI
  27. Liu, K., Wang, S., Koyuncu, A., Bissyandé, T.F., Kim, D., Wu, P., Klein, J., Mao, X., Le Traon, Y., 2020. On the efficiency of test suite based program repair: A systematic assessment of 16 automated repair systems for java programs. In: Proceedings of the 42nd International Conference on Software Engineering. ACM, pp. 615-627. http://dx.doi.org/10.1145/3377811.3380338. · DOI
  28. Liu, X., Zhong, H., 2018. Mining stackoverflow for program repair. In: Proceedings of the 25th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, pp. 118-129. http://dx.doi.org/10.1109/SANER.2018. 8330202. · DOI
  29. Long, F., Amidon, P., Rinard, M., 2017. Automatic inference of code transforms for patch generation. In: Proceedings of the 11th Joint Meeting on Founda- tions of Software Engineering. ACM, pp. 727-739. http://dx.doi.org/10.1145/ 3106237.3106253. · DOI
  30. Lutellier, T., Pang, L., Pham, H.V., Wei, M., Tan, L., ENCORE: ensemble learning using convolution neural machine translation for automatic program repair, arXiv preprint arXiv:1906.08691. · arXiv
  31. Lutellier, T., Pham, H.V., Pang, L., Li, Y., Wei, M., Tan, L., 2020. CoCoNuT: Combining context-aware neural translation models using ensemble for program repair. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, pp. 101-114. http: //dx.doi.org/10.1145/3395363.3397369. · DOI
  32. Madeiral, F., Urli, S., Maia, M., Monperrus, M., 2019. BEARS: an extensible java bug benchmark for automatic program repair studies. In: Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, pp. 468-478. http://dx.doi.org/10.1109/SANER.2019. 8667991. · DOI
  33. Martinez, M., Monperrus, M., 2016. ASTOR: a program repair library for java (demo). In: Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, pp. 441-444. http://dx.doi.org/10.1145/2931037. 2948705. · DOI
  34. Mechtaev, S., Nguyen, M.-D., Noller, Y., Grunske, L., Roychoudhury, A., 2018. Semantic program repair using a reference implementation. In: Proceedings of the 40th International Conference on Software Engineering. ACM, pp. 298-309. http://dx.doi.org/10.1145/3180155.3180247. · DOI
  35. Monperrus, M., 2009. The living review on automated program repair. In: Proceedings of the Symposium on the Foundations of Software Engineering. HAL/archives-ouvertes. fr, pp. 315-324.
  36. Monperrus, M., 2014. A critical review of automatic patch generation learned from human-written patches: essay on the problem statement and the evaluation of automatic software repair. In: Proceedings of the 36th In- ternational Conference on Software Engineering. ACM, pp. 234-242. http: //dx.doi.org/10.1145/2568225.2568324. · DOI
  37. Monperrus, M., 2018. Automatic software repair: A bibliography. ACM Comput. Surv. 51 (1), 17:1-17:24. http://dx.doi.org/10.1145/3105906. · DOI
  38. Motwani, M., Sankaranarayanan, S., Just, R., Brun, Y., 2018. Do automated program repair techniques repair hard and important bugs? Empir. Softw. Eng. 23 (5), 2901-2947. http://dx.doi.org/10.1007/s10664-017-9550-0. · DOI
  39. Nguyen, H.D.T., Qi, D., Roychoudhury, A., Chandra, S., 2013. Semfix: Program repair via semantic analysis. In: Proceedings of the 35th International Conference on Software Engineering. IEEE, pp. 772-781. http://dx.doi.org/ 10.1109/ICSE.2013.6606623. · DOI
  40. Qi, Z., Long, F., Achour, S., Rinard, M., 2015. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In: Proceed- ings of the International Symposium on Software Testing and Analysis. ACM, pp. 24-36. http://dx.doi.org/10.1145/2771783.2771791. · DOI · Paper page
  41. Qi, Y., Mao, X., Lei, Y., Wang, C., 2013. Using automated program repair for evaluating the effectiveness of fault localization techniques. In: Proceedings of the 22nd International Symposium on Software Testing and Analysis. pp. 191-201. http://dx.doi.org/10.1145/2483760.2483785. · DOI
  42. Razzaq, A., Wasala, A., Exton, C., Buckley, J., 2018. The state of empirical evaluation in static feature location. ACM Trans. Softw. Eng. Methodol. 28 (1), 2:1-2:58. http://dx.doi.org/10.1145/3280988. · DOI
  43. Rolim, R., Soares, G., Gheyi, R., D'Antoni, L., Learning quick fixes from code repositories, arXiv preprint arXiv:1803.03806. · arXiv
  44. Saha, R., Lyu, Y., Lam, W., Yoshida, H., Prasad, M., 2018. Bugs.jar: A large-scale, diverse dataset of real-world java bugs. In: Proceedings of the 15th IEEE/ACM International Conference on Mining Software Repositories. IEEE, pp. 10-13. http://dx.doi.org/10.1145/3196398.3196473. · DOI
  45. Saha, R.K., Lyu, Y., Yoshida, H., Prasad, M.R., 2017. ELIXIR: effective object- oriented program repair. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE, pp. 648-659. http: //dx.doi.org/10.1109/ASE.2017.8115675. · DOI
  46. Saha, S., Saha, R.K., Prasad, M.R., 2019. Harnessing evolution for multi-hunk program repair. In: Proceedings of the 41st IEEE/ACM International Con- ference on Software Engineering. IEEE, pp. 13-24. http://dx.doi.org/10.1109/ ICSE.2019.00020. · DOI
  47. Scott, A., Bader, J., Chandra, S., 2019. Getafix: Learning to fix bugs automati- cally. In: Proceedings of the ACM Programming Languages 3 (OOPSLA). pp. 159:1-159:27. http://dx.doi.org/10.1145/3360585. · DOI
  48. Sobreira, V., Durieux, T., Madeiral, F., Monperrus, M., de Almeida Maia, M., 2018. Dissection of a bug dataset: Anatomy of 395 patches from defects4j. In: Proceedings of the IEEE 25th International Conference on Software Analysis, Evolution and Reengineering. IEEE, pp. 130-140. http://dx.doi.org/10.1109/ SANER.2018.8330203. · DOI
  49. Tan, S.H., Yi, J., Yulis, Mechtaev, S., Roychoudhury, A., 2017. Codeflaws: a programming competition benchmark for evaluating automated program repair tools. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, pp. 180-182. http://dx.doi.org/10.1109/ ICSE-C.2017. · DOI
  50. Tian, H., Liu, K., Kaboré, A.K., Koyuncu, A., Li, L., Klein, J., Bissyandé, T.F., 2020. Evaluating representation learning of code changes for predicting patch correctness in program repair. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. ACM, http: //dx.doi.org/10.1145/3324884.3416532. · DOI
  51. Urli, S., Yu, Z., Seinturier, L., Monperrus, M., 2018. How to design a program repair bot? Insights from the repairnator project. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. ACM, pp. 95-104. http://dx.doi.org/10.1145/3183519.3183540. · DOI
  52. Wang, S., Wen, M., Mao, X., Yang, D., 2019. Attention please: Consider mockito when evaluating newly proposed automated program repair techniques. In: Proceedings of the Evaluation and Assessment on Software Engineering, EASE '19. ACM, New York, NY, USA, pp. 260-266. http://dx.doi.org/10.1145/ 3319008.3319349. · DOI
  53. Weimer, W., Nguyen, T., Le Goues, C., Forrest, S., 2009. Automatically finding patches using genetic programming. In: Proceedings of the 31st International Conference on Software Engineering. IEEE, pp. 364-374. http://dx.doi.org/10. 1109/ICSE.2009.5070536. · DOI
  54. Wen, M., Chen, J., Wu, R., Hao, D., Cheung, S.-C., 2018. Context-aware patch generation for better automated program repair. In: Proceedings of the 40th International Conference on Software Engineering. ACM, pp. 1-11. http://dx.doi.org/10.1145/3180155.3180233. · DOI
  55. Xin, Q., Reiss, S.P., 2017. Leveraging syntax-related code for automated program repair. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE, pp. 660-670. http://dx.doi.org/10. 1109/ASE.2017.8115676. · DOI
  56. Xiong, Y., Liu, X., Zeng, M., Zhang, L., Huang, G., 2018. Identifying patch correct- ness in test-based program repair. In: Proceedings of the 40th International Conference on Software Engineering. ACM, pp. 789-799. http://dx.doi.org/10. 1145/3180155.3180182. · DOI · Paper page
  57. Xiong, Y., Wang, J., Yan, R., Zhang, J., Han, S., Huang, G., Zhang, L., 2017. Precise condition synthesis for program repair. In: Proceedings of the 39th IEEE/ACM International Conference on Software Engineering. IEEE, pp. 416-426. http: //dx.doi.org/10.1109/ICSE.2017. · DOI
  58. Xu, X., Sui, Y., Yan, H., Xue, J., 2019. VFix: value-flow-guided precise program repair for null pointer dereferences. In: Proceedings of the 41st International Conference on Software Engineering. IEEE, pp. 512-523. · DOI
  59. Xuan, J., Martinez, M., Demarco, F., Clement, M., Marcote, S.L., Durieux, T., Le Berre, D., Monperrus, M., 2017. Nopol: Automatic repair of conditional statement bugs in java programs. IEEE Trans. Softw. Eng. 43 (1), 34-55. http://dx.doi.org/10.1109/TSE.2016.2560811. · DOI
  60. Xuan, J., Monperrus, M., 2014. Test case purification for improving fault local- ization. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp. 52-63. http://dx.doi.org/ 10.1145/2635868.2635906. · DOI
  61. Yuan, Y., Banzhaf, W., 2018. ARJA: Automated repair of java programs via multi- objective genetic programming. IEEE Trans. Softw. Eng. 1. http://dx.doi.org/ 10.1109/TSE.2018.2874648. · DOI
  62. Zhang, X., Gupta, N., Gupta, R., 2006. Locating faults through automated predicate switching. In: Proceedings of the 28th International Conference on Software Engineering. ACM, pp. 272-281. http://dx.doi.org/10.1145/1134285.1134324. · DOI

Cited by

Sorted by citations

Authors
Liu, K., Li, L., Koyuncu, A., Kim, D., Liu, Z., Klein, J., & Bissyandé, T. F.
Affiliations
  • Nanjing University of Aeronautics and Astronautics (CN)
  • Monash University (AU)
  • University of Luxembourg (LU)
  • Kyungpook National University (KR)
Corresponding author
  • Li, Li
  • Kim, Dongsun
ORCID
Venue
Journal of Systems and Software
Publisher
Elsevier BV
Published
2021-01
Volume
171
Pages
110817
ISSN
0164-1212 (ISSN-L)
DOI
10.1016/j.jss.2020.110817
OpenAlex
W3084918652
Cited by
79
References
5
Citations by year
2026: 5 · 2025: 10 · 2024: 20 · 2023: 11 · 2022: 21 · 2021: 11 · 2020: 1
FWCI
19.12
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Systems and Software (publishedVersion)
  • Open Repository and Bibliography (University of Luxembourg) (submittedVersion, cc-by-nc-sa)
Funders
Monash University
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
Keywords
automated program repair, evaluation metrics, patch generation, overfitting, evaluation bias, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review