Is the cure worse than the disease? overfitting in automated program repair
Abstract
Automated program repair has shown promise for reducing the significant manual effort debugging requires. This paper addresses a deficit of earlier evaluations of automated repair techniques caused by repairing programs and evaluating generated patches' correctness using the same set of tests. Since tests are an imperfect metric of program correctness, evaluations of this type do not discriminate between correct patches and patches that overfit the available tests and break untested but desired functionality. This paper evaluates two well-studied repair tools, GenProg and TrpAutoRepair, on a publicly available benchmark of bugs, each with a human-written patch. By evaluating patches using tests independent from those used during repair, we find that the tools are unlikely to improve the proportion of independent tests passed, and that the quality of the patches is proportional to the coverage of the test suite used during repair. For programs that pass most tests, the tools are as likely to break tests as to fix them. However, novice developers also overfit, and automated repair performs no worse than these developers. In addition to overfitting, we measure the effects of test suite coverage, test suite provenance, and starting program quality, as well as the difference in quality between novice-developer-written and tool-generated patches when quality is assessed with a test suite independent from the one used for patch generation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Test suites are rarely exhaustive and often suffer from weaknesses.
“However, due to practical limitations, test suites are rarely exhaustive and often suffer from weaknesses [50].”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [50]
Smith et al. found that using a test suite inaccessible to evaluated tools can help detect plausible but incorrect patches.
“Smith el al. [50] found that using a test suite that is inaccessible to evaluated tools can help detect plausible but incorrect patches.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [50]
The problem of weak test suites has been well studied in test-based automatic program repair.
“The problem of weak test suites has been well studied in the area of test-based automatic program repair (APR) [37, 40, 46, 50].”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [50]
Cites
Genetic Programming: On the Programming of Computers by Means of Natural Selection
Koza, J. R. (1992) · Medical Entomology and Zoology · Cited by 13,225 · see citing works
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011 · see citing works
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs
Cadar, C., Dunbar, D., & Engler, D. (2008) · doi:10.5555/1855741.1855756 · Cited by 2,655 · see citing works
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447 · see citing works
Mechanization in problem solving: The effect of Einstellung.
Luchins, A. S. (1942) · Psychological Monographs · doi:10.1037/h0093502 · Cited by 1,143 · see citing works
GenProg: A Generic Method for Automatic Software Repair
Goues, C. L., Nguyen, T. H., Forrest, S., & Weimer, W. R. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,084 · see citing works
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,084 · see citing works
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM-SIGACT Symposium on Principles of Programming Languages · doi:10.1145/1926385.1926423 · Cited by 1,039 · see citing works
Arcuri, A., & Briand, L. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985795 · Cited by 989 · see citing works
An experimental evaluation of the assumption of independence in multiversion programming
KNIGHT, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 768 · see citing works
Automatically finding patches using genetic programming
Weimer, W. R., Nguyen, T. H., Goues, C. L., & Forrest, S. (2009) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2009.5070536 · Cited by 768 · see citing works
A few billion lines of code later
Bessey, A., Block, K., Chelf, B., Chou, A., Fulton, B., Hallem, S., Henri-Gros, C., Kamsky, A., McPeak, S., & Engler, D. (2010) · Communications of the ACM · doi:10.1145/1646353.1646374 · Cited by 634 · see citing works
The Current State and Future of Search Based Software Engineering
Harman, M. (2007) · Future of Software Engineering (FOSE) · doi:10.1109/fose.2007.29 · Cited by 632 · see citing works
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926423 · Cited by 581 · see citing works
Are automated debugging techniques actually helping programmers?
Parnin, C., & Orso, A. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001445 · Cited by 580 · see citing works
Generating software test data by evolution
Michael, C. C., McGraw, G. E., & Schatz, M. A. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.988709 · Cited by 481 · see citing works
Automatic patch generation learned from human-written patches
Kim, D., Nam, J., Song, J., & Kim, S. (2013) · Cited by 438 · see citing works
SemFix: program repair via semantic analysis
Nguyen, H. D. T., Qi, D., Roychoudhury, A., & Chandra, S. (2013) · doi:10.5555/2486788.2486890 · Cited by 423 · see citing works
An analysis of patch plausibility and correctness for generate-and-validate patch generation systems
Qi, Z., Long, F., Achour, S., & Rinard, M. (2015) · Proceedings of the 2015 International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771791 · Cited by 420 · see citing works
Automatically patching errors in deployed software
Perkins, J. H., Kim, S., Larsen, S., Amarasinghe, S. P., Bachrach, J. R., Carbin, M., Pacheco, C., Sherwood, F., Sidiroglou, S., Sullivan, G., et al. (2009) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/1629575.1629585 · Cited by 409 · see citing works
N-VERSION PROGRAMMINC: A FAULT-TOLERANCE APPROACH TO RELlABlLlTY OF SOFTWARE OPERATlON
Chen, L., & Avižienis, A. (2005) · IEEE International Symposium on Fault-Tolerant Computing · doi:10.1109/ftcsh.1995.532621 · Cited by 398 · see citing works
Fault-scalable Byzantine fault-tolerant services
Abd-El-Malek, M., Ganger, G. R., Goodson, G. R., Reiter, M. K., & Wylie, J. J. (2005) · ACM SIGOPS Operating Systems Review · doi:10.1145/1095809.1095817 · Cited by 369 · see citing works
TimeAware test suite prioritization
Walcott, K. R., Soffa, M. L., Kapfhammer, G. M., & Roos, R. S. (2006) · International Symposium on Software Testing and Analysis · doi:10.1145/1146238.1146240 · Cited by 345 · see citing works
Bird, C., Bachmann, A., Aune, E., Duffy, J., Bernstein, A., Filkov, V., & Dévanbu, P. (2009) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1595696.1595716 · Cited by 335 · see citing works
The ManyBugs and IntroClass benchmarks for automated repair of C programs
Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2454513 · Cited by 326 · see citing works
Self-citationLeveraging program equivalence for adaptive program repair: Models and first results
Weimer, W. R., Fry, Z. P., & Forrest, S. (2013) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693094 · Cited by 271 · see citing works
A novel co-evolutionary approach to automatic software bug fixing
Arcuri, A., & Yao, X. (2008) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2008.4630793 · Cited by 251 · see citing works
Functional fixedness as related to problem solving: a repetition of three experiments.
Adamson, R. E. (1952) · Journal of Experimental Psychology · doi:10.1037/h0062487 · Cited by 250 · see citing works
Seng, O., Stammel, J., & Burkhart, D. (2006) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1143997.1144315 · Cited by 245 · see citing works
Automated fixing of programs with contracts
Wei, Y., Pei, Y., Furia, C. A., Silva, L. S., Buchholz, S., Meyer, B., & Zeller, A. (2010) · International Symposium on Software Testing and Analysis · doi:10.1145/1831708.1831716 · Cited by 236 · see citing works
Using Mutation to Automatically Suggest Fixes for Faulty Programs
Debroy, V., & Wong, W. E. (2010) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2010.66 · Cited by 228 · see citing works
The plastic surgery hypothesis
Barr, E. T., Brun, Y., Dévanbu, P., Harman, M., & Sarro, F. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635898 · Cited by 217 · see citing works
Self-citationRepairing Programs with Semantic Code Search (T)
Ke, Y., Stolee, K. T., Goues, C. L., & Brun, Y. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.60 · Cited by 196 · see citing works
Self-citationDirectFix: Looking for Simple Program Repairs
Mechtaev, S., Yi, J., & Roychoudhury, A. (2015) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2015.63 · Cited by 181 · see citing works
Monperrus, M. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2568225.2568324 · Cited by 175 · see citing works
Silva, S., & Costa, E. J. F. (2009) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-008-9075-9 · Cited by 163 · see citing works
Automated Fixing of Programs with Contracts
Pei, Y., Furia, C. A., Nordio, M., Wei, Y., Meyer, B., & Zeller, A. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2312918 · Cited by 151 · see citing works
Automated atomicity-violation fixing
Jin, G., Song, L., Zhang, W., Lu, S., & Liblit, B. (2011) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1993498.1993544 · Cited by 151 · see citing works
An Empirical Study on Real Bug Fixes
Zhong, H., & Su, Z. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.101 · Cited by 151 · see citing works
Staffing a software project: A constraint satisfaction and optimization-based approach
Barreto, A. S., Barros, M. D. O., & Werner, C. M. L. (2007) · Computers & Operations Research · doi:10.1016/j.cor.2007.01.010 · Cited by 147 · see citing works
Countering Network Worms Through Automatic Patch Generation
Sidiroglou, S., & Keromytis, A. D. (2005) · IEEE Security & Privacy · doi:10.1109/msp.2005.144 · Cited by 136 · see citing works
Automated atomicity-violation fixing
Jin, G., Song, L., Zhang, W., Lu, S., & Liblit, B. (2011) · ACM SIGPLAN Notices · doi:10.1145/1993316.1993544 · Cited by 135 · see citing works
Fault-scalable Byzantine fault-tolerant services
Abd-El-Malek, M., Ganger, G. R., Goodson, G. R., Reiter, M. K., & Wylie, J. J. (2005) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/1095810.1095817 · Cited by 133 · see citing works
Automatic error elimination by horizontal code transfer across multiple applications
Sidiroglou, S., Lahtinen, E., Long, F., & Rinard, M. (2015) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2737924.2737988 · Cited by 114 · see citing works
A human study of patch maintainability
Fry, Z. P., Landau, B., & Weimer, W. R. (2012) · International Symposium on Software Testing and Analysis · doi:10.1145/2338965.2336775 · Cited by 113 · see citing works
DIRA: Automatic Detection, Identification and Repair of Control-Hijacking Attacks.
Смирнов, А. В., & Chiueh, T. (2005) · Cited by 105 · see citing works
An empirical study on real bug fixes
Zhong, H., & Su, Z. (2015) · doi:10.5555/2818754.2818864 · Cited by 103 · see citing works
Efficient Automated Program Repair through Fault-Recorded Testing Prioritization
Qi, Y., Mao, X., & Lei, Y. (2013) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2013.29 · Cited by 94 · see citing works
Flight of the FINCH Through the Java Wilderness
Orlov, M., & Sipper, M. (2011) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2010.2052622 · Cited by 94 · see citing works
Ecological inference in empirical software engineering
Posnett, D., Filkov, V., & Dévanbu, P. (2011) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2011.6100074 · Cited by 87 · see citing works
Automatic workarounds for web applications
Carzaniga, A., Gorla, A., Perino, N., & Pezzè, M. (2010) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1882291.1882327 · Cited by 84 · see citing works
Automatic recovery from runtime failures
Carzaniga, A., Gorla, A., Mattavelli, A., Perino, N., & Pezzè, M. (2013) · doi:10.5555/2486788.2486891 · Cited by 81 · see citing works
Representations and operators for improving evolutionary software repair
Goues, C. L., Weimer, W. R., & Forrest, S. (2012) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2330163.2330296 · Cited by 76 · see citing works
Finding safety errors with ACO
Alba, E., & Chicano, F. (2007) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1276958.1277171 · Cited by 72 · see citing works
relifix: Automated Repair of Software Regressions
Tan, S. H., & Roychoudhury, A. (2015) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2015.65 · Cited by 69 · see citing works
Problem Difficulty and Code Growth in Genetic Programming
Gustafson, S., Ekárt, A., Burke, E. K., & Kendall, G. (2004) · Genetic Programming and Evolvable Machines · doi:10.1023/b:genp.0000030194.98244.e3 · Cited by 55 · see citing works
Grail: context-aware fixing of concurrency bugs
Liu, P., Tripp, O., & Zhang, C. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635881 · Cited by 53 · see citing works
Program transformations to fix C integers
Coker, Z., & Hafiz, M. (2013) · doi:10.5555/2486788.2486892 · Cited by 52 · see citing works
Automatic Repair of Real Bugs: An Experience Report on the Defects4J Dataset
Martinez, M., Durieux, T., Xuan, J., Sommerard, R., & Monperrus, M. (2015) · arXiv · doi:10.48550/arxiv.1505.07002 · Cited by 47 · see citing works
Semantic differential repair for input validation and sanitization
Alkhalaf, M., Aydin, A., & Bultan, T. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610401 · Cited by 44 · see citing works
Cochran, R. A., D’Antoni, L., Livshits, B., Molnár, D., & Veanes, M. (2014) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/2676726.2676973 · Cited by 37 · see citing works
Automatic repair for multi-threaded programs with Deadlock/Livelock using maximum satisfiability
Lin, Y., & Kulkarni, S. S. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610398 · Cited by 33 · see citing works
Gabel, M., & Su, Z. (2012) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2393596.2393598 · Cited by 31 · see citing works
Estler, H.-C., Furia, C. A., Nordio, M., Piccioni, M., & Meyer, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-06410-9_17 · Cited by 27 · see citing works
Barreto, A. S., Barros, M. D. O., & Werner, C. M. L. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083093 · Cited by 17 · see citing works
Evolution vs. Intelligent Design in Program Patching
Dévanbu, P. (2013) · eScholarship (California Digital Library) · Cited by 11 · see citing works
Flagged first, then by citations
2 of the 68 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Mechtaev, S., Yi, J., & Roychoudhury, A. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884807 · Cited by 515 · see citing works
Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs
Xuan, J., Martínez, M., DeMarco, F., Clement, M., Marcote, S. R. L., Durieux, T., Berre, D. L., & Monperrus, M. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2560811 · Cited by 430 · see citing works
Le Goues, C., Pradel, M., & Roychoudhury, A. (2019) · Communications of the ACM · doi:10.1145/3318162 · Cited by 410 · see citing works
SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z.-M., Kommrusch, S., Tufano, M., Pouchet, L., Poshyvanyk, D., & Martin, M. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 585 · see citing works
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3340544 · Cited by 370 · see citing works
Le, X.-B. D., Lo, D., & Goues, C. L. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 341 · see citing works
Context-aware patch generation for better automated program repair
Wen, M., Chen, J., Wu, R., Hao, D., & Cheung, S.-C. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180233 · Cited by 337 · see citing works
Automatic Software Repair: A Survey
Gazzola, L., Micucci, D., & Mariani, L. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2755013 · Cited by 336 · see citing works
The ManyBugs and IntroClass benchmarks for automated repair of C programs
Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2454513 · Cited by 326 · see citing works
Self-citationTBar: revisiting template-based automated program repair
Liu, K., Koyuncu, A., Kim, D., & Bissyandé, T. F. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330577 · Cited by 326 · see citing works
Monperrus, M. (2018) · ACM Computing Surveys · doi:10.1145/3105906 · Cited by 386 · see citing works
Precise Condition Synthesis for Program Repair
Xiong, Y., Wang, J., Yan, R., Zhang, J., Shi, H., Huang, G., & Zhang, L. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.45 · Cited by 308 · see citing works
Automatic repair of real bugs in java: a large-scale experiment on the defects4j dataset
Martínez, M., Durieux, T., Sommerard, R., Xuan, J., & Monperrus, M. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9470-4 · Cited by 249 · see citing works
Genetic Improvement of Software: A Comprehensive Survey
Petke, J., Haraldsson, S. O., Harman, M., Langdon, W. B., White, D. R., & Woodward, J. R. (2017) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2017.2693219 · Cited by 230 · see citing works
ARJA: Automated Repair of Java Programs via Multi-Objective Genetic Programming
Yuan, Y., & Banzhaf, W. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2874648 · Cited by 225 · see citing works
S3: syntax- and semantic-guided repair synthesis via programming by examples
Le, X.-B. D., Chu, D.-H., Lo, D., Goues, C. L., & Visser, W. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106309 · Cited by 219 · see citing works
Li, Y., Wang, S., & Nguyen, T. N. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380345 · Cited by 211 · see citing works
Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT
Xia, C. S., & Zhang, L. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680323 · Cited by 279 · see citing works
AutoCodeRover: Autonomous Program Improvement
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680384 · Cited by 202 · see citing works
AutoCodeRover: Autonomous Program Improvement
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2404.05427 · Cited by 202 · see citing works
Repairing Programs with Semantic Code Search (T)
Ke, Y., Stolee, K. T., Goues, C. L., & Brun, Y. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.60 · Cited by 196 · see citing works
Self-citationAVATAR: Fixing Semantic Bugs with Fix Patterns of Static Analysis Violations
Liu, K., Koyuncu, A., Kim, D., & Bissyandé, T. F. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8667970 · Cited by 175 · see citing works
An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems
Long, F., & Rinard, M. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884872 · Cited by 167 · see citing works
Gazzola, L., Micucci, D., & Mariani, L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182526 · Cited by 167 · see citing works
Identifying patch correctness in test-based program repair
Xiong, Y., Liu, X., Zeng, M., Zhang, L., & Huang, G. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180182 · Cited by 162 · see citing works
Neural program repair with execution-based backpropagation
Ye, H., Martínez, M., & Monperrus, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510222 · Cited by 161 · see citing works
Liu, K., Koyuncu, A., Bissyandé, T. F., Kim, D., Klein, J., & Traon, Y. L. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00020 · Cited by 159 · see citing works
Towards practical program repair with on-demand candidate generation
Hua, J., Zhang, M., Wang, K., & Sarfraz Khurshid, D. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180245 · Cited by 138 · see citing works
A Survey of Learning-based Automated Program Repair
Zhang, Q., Fang, C., Ma, Y., Sun, W., & Chen, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3631974 · Cited by 153 · see citing works
Anti-patterns in search-based program repair
Tan, S. H., Yoshida, H., Prasad, M. R., & Roychoudhury, A. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2950295 · Cited by 135 · see citing works
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3240732 · Cited by 133 · see citing works
Mining stackoverflow for program repair
Liu, X., & Zhong, H. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330202 · Cited by 132 · see citing works
Durieux, T., Madeiral, F., Martínez, M., & Abreu, R. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338911 · Cited by 130 · see citing works
On the efficiency of test suite based program repair
Liu, K., Wang, S., Koyuncu, A., Kim, K., Bissyandé, T. F., Kim, D., Wu, P., Klein, J., Mao, X., & Traon, Y. L. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380338 · Cited by 125 · see citing works
Better test cases for better automated program repair
Yang, J., Zhikhartsev, A., Liu, Y., & Tan, L. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106274 · Cited by 124 · see citing works
Mining Fix Patterns for FindBugs Violations
Liu, K., Kim, D., Bissyandé, T. F., Yoo, S., & Traon, Y. L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2884955 · Cited by 123 · see citing works
Contract-based program repair without the contracts
Chen, L., Pei, Y., & Furia, C. A. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115674 · Cited by 119 · see citing works
Identifying test-suite-overfitted patches through test case generation
Qi, X., & Reiss, S. P. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092718 · Cited by 117 · see citing works
A feasibility study of using automated program repair for introductory programming assignments
Yi, J., Ahmed, U. Z., Karkare, A., Tan, S. H., & Roychoudhury, A. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106262 · Cited by 113 · see citing works
Tian, H., Liu, K., Kaboré, A. K., Koyuncu, A., Li, L., Klein, J., & Bissyandé, T. F. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416532 · Cited by 101 · see citing works
Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting
Li, H., Rajbahadur, G. K., Lin, D., Bezemer, C., & Jiang, Z. M. (2024) · IEEE Access · doi:10.1109/access.2024.3402543 · Cited by 99 · see citing works
Static automated program repair for heap properties
van Tonder, R., & Goues, C. L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180250 · Cited by 96 · see citing works
Ultra-Large Repair Search Space with Automatically Mined Templates: The Cardumen Mode of Astor
Martínez, M., & Monperrus, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_3 · Cited by 96 · see citing works
Baldur: Whole-Proof Generation and Repair with Large Language Models
First, E., Rabe, M. N., Ringer, T., & Brun, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616243 · Cited by 94 · see citing works
Self-citationiFixR: bug report driven program repair
Koyuncu, A., Liu, K., Bissyandé, T. F., Kim, D., Monperrus, M., Klein, J., & Traon, Y. L. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338935 · Cited by 94 · see citing works
On Reliability of Patch Correctness Assessment
Le, X.-B. D., Bao, L., Lo, D., Xia, X., Li, S., & Păsăreanu, C. S. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00064 · Cited by 84 · see citing works
Hassan, F., & Wang, X. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180181 · Cited by 84 · see citing works
Automated patch correctness assessment
Wang, S., Wen, M., Lin, B., Wu, H., Qin, Y., Zou, D., Mao, X., & Jin, H. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416590 · Cited by 79 · see citing works
A critical review on the evaluation of automated program repair systems
Liu, K., Li, L., Koyuncu, A., Kim, D., Liu, Z., Klein, J., & Bissyandé, T. F. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110817 · Cited by 79 · see citing works
Automated patch assessment for program repair at scale
Ye, H., Martínez, M., & Monperrus, M. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09920-w · Cited by 75 · see citing works
Gao, X. Y., Wang, B., Duck, G. J., Ji, R., Xiong, Y., & Roychoudhury, A. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3418461 · Cited by 72 · see citing works
Overfitting in semantics-based automated program repair
Le, X.-B. D., Thung, F., Lo, D., & Goues, C. L. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9577-2 · Cited by 75 · see citing works
JFIX: semantics-based repair of Java programs via symbolic PathFinder
Le, X.-B. D., Chu, D.-H., Lo, D., Goues, C. L., & Visser, W. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3098225 · Cited by 71 · see citing works
Repairing crashes in Android apps
Tan, S. H., Dong, Z., Gao, X., & Roychoudhury, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180243 · Cited by 71 · see citing works
How to design a program repair bot?
Urli, S., Yu, Z., Seinturier, L., & Monperrus, M. (2018) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3183519.3183540 · Cited by 70 · see citing works
LSRepair: Live Search of Fix Ingredients for Automated Program Repair
Liu, K., Koyuncu, A., Kim, K., Kim, D., & Bissyandé, T. F. (2018) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2018.00085 · Cited by 70 · see citing works
Automated Classification of Overfitting Patches With Statically Extracted Code Features
Ye, H., Gu, J. N., Martínez, M., Durieux, T., & Monperrus, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3071750 · Cited by 69 · see citing works
A comprehensive study of automatic program repair on the QuixBugs benchmark
Ye, H., Martínez, M., Durieux, T., & Monperrus, M. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110825 · Cited by 67 · see citing works
Automatic Software Repair: a BibliographyPreprint
Monperrus, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00515 · Cited by 66 · see citing works
Context-Aware Code Change Embedding for Better Patch Correctness Assessment
Lin, B., Wang, S., Wen, M., & Mao, X. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3505247 · Cited by 63 · see citing works
Gao, X. Y., Mechtaev, S., & Roychoudhury, A. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330558 · Cited by 61 · see citing works
Phoenix: automated data-driven synthesis of repairs for static analysis violations
Bavishi, R., Yoshida, H., & Prasad, M. R. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338952 · Cited by 61 · see citing works
Yu, Z., Martínez, M., Danglot, B., Durieux, T., & Monperrus, M. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9619-4 · Cited by 60 · see citing works
How to Design a Program Repair Bot? Insights from the Repairnator ProjectPreprint
Urli, S., Yu, Z., Seinturier, L., & Monperrus, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.09852 · Cited by 59 · see citing works
SOSRepair: Expressive Semantic Search for Real-World Program Repair
Afzal, A., Motwani, M., Stolee, K. T., Brun, Y., & Goues, C. L. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2944914 · Cited by 56 · see citing works
Self-citationTrautsch, F., Herbold, S., Makedonski, P., & Grabowski, J. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9537-x · Cited by 55 · see citing works
Do automated program repair techniques repair hard and important bugs?
Motwani, M., Sankaranarayanan, S., Just, R., & Brun, Y. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9550-0 · Cited by 54 · see citing works
Self-citationLearning to Fix Build Errors with Graph2Diff Neural Networks
Tarlow, D., Moitra, S., Rice, A. C., Chen, Z., Manzagol, P.-A., Sutton, C. A., & Aftandilian, E. E. (2020) · International Conference on Software Engineering · doi:10.1145/3387940.3392181 · Cited by 55 · see citing works
Toward Better Evolutionary Program Repair
Yuan, Y., & Banzhaf, W. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3360004 · Cited by 50 · see citing works
MemFix: static analysis-based repair of memory deallocation errors for C
Lee, J., Hong, S., & Oh, H. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236079 · Cited by 50 · see citing works
Quality of Automated Program Repair on Real-World Defects
Motwani, M., Soto, M., Brun, Y., Just, R., & Goues, C. L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2998785 · Cited by 50 · see citing works
Self-citationVarFix: balancing edit expressiveness and search effectiveness in automated program repair
Wong, C.-P., Santiesteban, P., Kästner, C., & Goues, C. L. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468600 · Cited by 48 · see citing works
Shariffdeen, R., Noller, Y., Grunske, L., & Roychoudhury, A. (2021) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3453483.3454051 · Cited by 48 · see citing works
A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark
Ye, H., Martínez, M., & Monperrus, M. (n.d.) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 48 · see citing works
Automatically Generating Precise Oracles from Structured Natural Language Specifications
Motwani, M., & Brun, Y. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00035 · Cited by 46 · see citing works
Self-citationA metamorphic testing approach for supporting program repair without the need for a test oracle
Jiang, M., Chen, T. Y., Kuo, F., Towey, D., & Ding, Z. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.04.002 · Cited by 46 · see citing works
Automated Transplantation and Differential Testing for Clones
Zhang, T., & Kim, M. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.67 · Cited by 46 · see citing works
Brownlee, A. E. I., Petke, J., Alexander, B., Barr, E. T., Wagner, M., & White, D. R. (2019) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3321707.3321841 · Cited by 44 · see citing works
Self-citationHistory-driven build failure fixing: how far are we?
Lou, Y., Chen, J., Zhang, L., Hao, D., & Zhang, L. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330578 · Cited by 44 · see citing works
Common Bug-Fix Patterns: A Large-Scale Observational Study
Campos, E. C., & Maia, M. D. A. (2017) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2017.55 · Cited by 42 · see citing works
Test-Equivalence Analysis for Automatic Patch Generation
Mechtaev, S., Gao, X., Tan, S. H., & Roychoudhury, A. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3241980 · Cited by 42 · see citing works
TacTok: semantics-aware proof synthesis
First, E., Brun, Y., & Guha, A. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428299 · Cited by 43 · see citing works
Self-citationA correlation study between automated program repair and test-suite metrics
Yi, J., Tan, S. H., Mechtaev, S., Böhme, M., & Roychoudhury, A. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9552-y · Cited by 38 · see citing works
Automated Patch Transplantation
Shariffdeen, R., Tan, S. H., Gao, M., & Roychoudhury, A. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3412376 · Cited by 38 · see citing works
Exploring True Test Overfitting in Dynamic Automated Program Repair using Formal Methods
Nilizadeh, A., Leavens, G. T., Le, X.-B. D., Păsăreanu, C. S., & Cok, D. R. (2021) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst49551.2021.00033 · Cited by 37 · see citing works
SketchFix: a tool for automated program repair approach using lazy candidate generation
Hua, J., Zhang, M., Wang, K., & Sarfraz Khurshid, D. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264600 · Cited by 37 · see citing works
Experience report: How do techniques, programs, and tests impact automated program repair?
Kong, X., Zhang, L., Wong, W. E., & Li, B. (2015) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2015.7381813 · Cited by 36 · see citing works
A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark
Ye, H., Martínez, M., Durieux, T., & Monperrus, M. (2019) · IEEE International Workshop on Intelligent Bug Fixing (IBF) · doi:10.1109/ibf.2019.8665475 · Cited by 36 · see citing works
Yu, Z. (., Martínez, M., Danglot, B. (., Durieux, T., & Monperrus, M. (2018) · LillOA (Université de Lille (University Of Lille)) · Cited by 36 · see citing works
Learning to Generate Corrective Patches using Neural Machine TranslationPreprint
Hata, H., Shihab, E., & Neubig, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07170 · Cited by 36 · see citing works
Tian, H., Liu, K., Li, Y., Kaboré, A. K., Koyuncu, A., Habib, A., Li, L., Wen, J., Klein, J., & Bissyandé, T. F. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3576039 · Cited by 35 · see citing works
Wang, S., Wen, M., Chen, L., Yi, X., & Mao, X. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870172 · Cited by 34 · see citing works
Astor: Exploring the design space of generate-and-validate program repair beyond GenProg
Martínez, M., & Monperrus, M. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.01.069 · Cited by 33 · see citing works
Evaluating Automatic Program Repair Capabilities to Repair API Misuses
Kechagia, M., Mechtaev, S., Sarro, F., & Harman, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3067156 · Cited by 33 · see citing works
Predicting Patch Correctness Based on the Similarity of Failing Test Cases
Tian, H., Li, Y., Pian, W., Kaboré, A. K., Liu, K., Habib, A., Klein, J., & Bissyandé, T. F. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3511096 · Cited by 33 · see citing works
Diversity-driven automated formal verification
First, E., & Brun, Y. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510138 · Cited by 35 · see citing works
Self-citationSymbolic execution with existential second-order constraints
Mechtaev, S., Griggio, A., Cimatti, A., & Roychoudhury, A. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236049 · Cited by 32 · see citing works
Hong, S., Lee, J., Lee, J. S., & Oh, H. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380323 · Cited by 32 · see citing works
Automatic patch generation with context-based change application
Kim, J., & Kim, S. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09742-5 · Cited by 31 · see citing works
A manual inspection of Defects4J bugs and its implications for automatic program repair
Jiang, J., Xiong, Y., & Xia, X. (2019) · Science China Information Sciences · doi:10.1007/s11432-018-1465-6 · Cited by 31 · see citing works
- Authors
- Smith, E. K., Barr, E. T., Le Goues, C., & Brun, Y.
- Affiliations
- University of Massachusetts Amherst (US)
- University College London (GB)
- Carnegie Mellon University, USA
- University of Massachusetts, USA
- Venue
- Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2015-08-30
- Pages
- 532-543
- OpenAlex
- W2061575154
- Cited by
- 323
- References
- 68
- Citations by year
- 2026: 18 · 2025: 25 · 2024: 33 · 2023: 22 · 2022: 29 · 2021: 26 · 2020: 37 · 2019: 40 · 2018: 43 · 2017: 27 · 2016: 19 · 2015: 4
- FWCI
- 23.83
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- Funders
- National Science Foundation (CCF-1446683, CCF-1446966, CCF-1453508)
- Microsoft Research (Software Engineering Innovation Foundation Award)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Keywords
- automated program repair, overfitting, patch evaluation, test suite coverage, novice programmers, program correctness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
