The DOI matches its Crossref record; 66 of the 68 references it cites were found in a registry or an index.
Is the cure worse than the disease? overfitting in automated program repair
Abstract
Automated program repair has shown promise for reducing the significant manual effort debugging requires. This paper addresses a deficit of earlier evaluations of automated repair techniques caused by repairing programs and evaluating generated patches' correctness using the same set of tests. Since tests are an imperfect metric of program correctness, evaluations of this type do not discriminate between correct patches and patches that overfit the available tests and break untested but desired functionality. This paper evaluates two well-studied repair tools, GenProg and TrpAutoRepair, on a publicly available benchmark of bugs, each with a human-written patch. By evaluating patches using tests independent from those used during repair, we find that the tools are unlikely to improve the proportion of independent tests passed, and that the quality of the patches is proportional to the coverage of the test suite used during repair. For programs that pass most tests, the tools are as likely to break tests as to fix them. However, novice developers also overfit, and automated repair performs no worse than these developers. In addition to overfitting, we measure the effects of test suite coverage, test suite provenance, and starting program quality, as well as the difference in quality between novice-developer-written and tool-generated patches when quality is assessed with a test suite independent from the one used for patch generation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, due to practical limitations, test suites are rarely exhaustive and often suffer from weaknesses [50].”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [50]
In the citing paper
“Smith el al. [50] found that using a test suite that is inaccessible to evaluated tools can help detect plausible but incorrect patches.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [50]
In the citing paper
“The problem of weak test suites has been well studied in the area of test-based automatic program repair (APR) [37, 40, 46, 50].”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [50]
Citations
Cites
Genetic Programming: On the Programming of Computers by Means of Natural Selection
Koza, J. R. (1992) · Medical Entomology and Zoology · Cited by 13,082
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs
Cadar, C., Dunbar, D., & Engler, D. (2008) · doi:10.5555/1855741.1855756 · Cited by 2,655
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Mechanization in problem solving: The effect of Einstellung.
Luchins, A. S. (1942) · Psychological Monographs · doi:10.1037/h0093502 · Cited by 1,143
GenProg: A Generic Method for Automatic Software Repair
Goues, C. L., Nguyen, T. H., Forrest, S., & Weimer, W. R. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,084
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,084
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM-SIGACT Symposium on Principles of Programming Languages · doi:10.1145/1926385.1926423 · Cited by 1,039
Arcuri, A., & Briand, L. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985795 · Cited by 989
An experimental evaluation of the assumption of independence in multiversion programming
KNIGHT, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 768
Automatically finding patches using genetic programming
Weimer, W. R., Nguyen, T. H., Goues, C. L., & Forrest, S. (2009) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2009.5070536 · Cited by 768
The Current State and Future of Search Based Software Engineering
Harman, M. (2007) · Future of Software Engineering (FOSE) · doi:10.1109/fose.2007.29 · Cited by 636
A few billion lines of code later
Bessey, A., Block, K., Chelf, B., Chou, A., Fulton, B., Hallem, S., Henri-Gros, C., Kamsky, A., McPeak, S., & Engler, D. (2010) · Communications of the ACM · doi:10.1145/1646353.1646374 · Cited by 634
Are automated debugging techniques actually helping programmers?
Parnin, C., & Orso, A. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001445 · Cited by 593
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926423 · Cited by 581
Generating software test data by evolution
Michael, C. C., McGraw, G. E., & Schatz, M. A. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.988709 · Cited by 481
Automatic patch generation learned from human-written patches
Kim, D., Nam, J., Song, J., & Kim, S. (2013) · Cited by 438
SemFix: program repair via semantic analysis
Nguyen, H. D. T., Qi, D., Roychoudhury, A., & Chandra, S. (2013) · doi:10.5555/2486788.2486890 · Cited by 423
An analysis of patch plausibility and correctness for generate-and-validate patch generation systems
Qi, Z., Long, F., Achour, S., & Rinard, M. (2015) · Proceedings of the 2015 International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771791 · Cited by 420
Automatically patching errors in deployed software
Perkins, J. H., Kim, S., Larsen, S., Amarasinghe, S. P., Bachrach, J. R., Carbin, M., Pacheco, C., Sherwood, F., Sidiroglou, S., Sullivan, G., et al. (2009) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/1629575.1629585 · Cited by 409
N-VERSION PROGRAMMINC: A FAULT-TOLERANCE APPROACH TO RELlABlLlTY OF SOFTWARE OPERATlON
Chen, L., & Avižienis, A. (2005) · IEEE International Symposium on Fault-Tolerant Computing · doi:10.1109/ftcsh.1995.532621 · Cited by 398
Fault-scalable Byzantine fault-tolerant services
Abd-El-Malek, M., Ganger, G. R., Goodson, G. R., Reiter, M. K., & Wylie, J. J. (2005) · ACM SIGOPS Operating Systems Review · doi:10.1145/1095809.1095817 · Cited by 370
TimeAware test suite prioritization
Walcott, K. R., Soffa, M. L., Kapfhammer, G. M., & Roos, R. S. (2006) · International Symposium on Software Testing and Analysis · doi:10.1145/1146238.1146240 · Cited by 345
Bird, C., Bachmann, A., Aune, E., Duffy, J., Bernstein, A., Filkov, V., & Dévanbu, P. (2009) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1595696.1595716 · Cited by 339
The ManyBugs and IntroClass benchmarks for automated repair of C programs
Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2454513 · Cited by 326
Self-citationLeveraging program equivalence for adaptive program repair: Models and first results
Weimer, W. R., Fry, Z. P., & Forrest, S. (2013) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693094 · Cited by 271
A novel co-evolutionary approach to automatic software bug fixing
Arcuri, A., & Yao, X. (2008) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2008.4630793 · Cited by 251
Functional fixedness as related to problem solving: a repetition of three experiments.
Adamson, R. E. (1952) · Journal of Experimental Psychology · doi:10.1037/h0062487 · Cited by 250
Seng, O., Stammel, J., & Burkhart, D. (2006) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1143997.1144315 · Cited by 245
Automated fixing of programs with contracts
Wei, Y., Pei, Y., Furia, C. A., Silva, L. S., Buchholz, S., Meyer, B., & Zeller, A. (2010) · International Symposium on Software Testing and Analysis · doi:10.1145/1831708.1831716 · Cited by 236
Using Mutation to Automatically Suggest Fixes for Faulty Programs
Debroy, V., & Wong, W. E. (2010) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2010.66 · Cited by 228
The plastic surgery hypothesis
Barr, E. T., Brun, Y., Dévanbu, P., Harman, M., & Sarro, F. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635898 · Cited by 217
Self-citationRepairing Programs with Semantic Code Search (T)
Ke, Y., Stolee, K. T., Goues, C. L., & Brun, Y. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.60 · Cited by 196
Self-citationDirectFix: Looking for Simple Program Repairs
Mechtaev, S., Yi, J., & Roychoudhury, A. (2015) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2015.63 · Cited by 181
Monperrus, M. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568324 · Cited by 173
Silva, S., & Costa, E. J. F. (2009) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-008-9075-9 · Cited by 157
An Empirical Study on Real Bug Fixes
Zhong, H., & Su, Z. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.101 · Cited by 154
Automated Fixing of Programs with Contracts
Pei, Y., Furia, C. A., Nordio, M., Wei, Y., Meyer, B., & Zeller, A. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2312918 · Cited by 151
Automated atomicity-violation fixing
Jin, G., Song, L., Zhang, W., Lu, S., & Liblit, B. (2011) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1993498.1993544 · Cited by 151
Staffing a software project: A constraint satisfaction and optimization-based approach
Barreto, A. S., Barros, M. D. O., & Werner, C. M. L. (2007) · Computers & Operations Research · doi:10.1016/j.cor.2007.01.010 · Cited by 147
Countering Network Worms Through Automatic Patch Generation
Sidiroglou, S., & Keromytis, A. D. (2005) · IEEE Security & Privacy · doi:10.1109/msp.2005.144 · Cited by 136
Automated atomicity-violation fixing
Jin, G., Song, L., Zhang, W., Lu, S., & Liblit, B. (2011) · ACM SIGPLAN Notices · doi:10.1145/1993316.1993544 · Cited by 135
Fault-scalable Byzantine fault-tolerant services
Abd-El-Malek, M., Ganger, G. R., Goodson, G. R., Reiter, M. K., & Wylie, J. J. (2005) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/1095810.1095817 · Cited by 134
Automatic error elimination by horizontal code transfer across multiple applications
Sidiroglou, S., Lahtinen, E., Long, F., & Rinard, M. (2015) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2737924.2737988 · Cited by 114
A human study of patch maintainability
Fry, Z. P., Landau, B., & Weimer, W. R. (2012) · International Symposium on Software Testing and Analysis · doi:10.1145/2338965.2336775 · Cited by 113
DIRA: Automatic Detection, Identification and Repair of Control-Hijacking Attacks.
Смирнов, А. В., & Chiueh, T. (2005) · Cited by 105
An empirical study on real bug fixes
Zhong, H., & Su, Z. (2015) · doi:10.5555/2818754.2818864 · Cited by 103
Efficient Automated Program Repair through Fault-Recorded Testing Prioritization
Qi, Y., Mao, X., & Lei, Y. (2013) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2013.29 · Cited by 94
Flight of the FINCH Through the Java Wilderness
Orlov, M., & Sipper, M. (2011) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2010.2052622 · Cited by 94
Ecological inference in empirical software engineering
Posnett, D., Filkov, V., & Dévanbu, P. (2011) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2011.6100074 · Cited by 87
Automatic workarounds for web applications
Carzaniga, A., Gorla, A., Perino, N., & Pezzè, M. (2010) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1882291.1882327 · Cited by 84
Automatic recovery from runtime failures
Carzaniga, A., Gorla, A., Mattavelli, A., Perino, N., & Pezzè, M. (2013) · doi:10.5555/2486788.2486891 · Cited by 81
Representations and operators for improving evolutionary software repair
Goues, C. L., Weimer, W. R., & Forrest, S. (2012) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2330163.2330296 · Cited by 76
Finding safety errors with ACO
Alba, E., & Chicano, F. (2007) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1276958.1277171 · Cited by 72
relifix: Automated Repair of Software Regressions
Tan, S. H., & Roychoudhury, A. (2015) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2015.65 · Cited by 69
Problem Difficulty and Code Growth in Genetic Programming
Gustafson, S., Ekárt, A., Burke, E. K., & Kendall, G. (2004) · Genetic Programming and Evolvable Machines · doi:10.1023/b:genp.0000030194.98244.e3 · Cited by 55
Grail: context-aware fixing of concurrency bugs
Liu, P., Tripp, O., & Zhang, C. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635881 · Cited by 53
Program transformations to fix C integers
Coker, Z., & Hafiz, M. (2013) · doi:10.5555/2486788.2486892 · Cited by 52
Automatic Repair of Real Bugs: An Experience Report on the Defects4J Dataset
Martinez, M., Durieux, T., Xuan, J., Sommerard, R., & Monperrus, M. (2015) · arXiv · doi:10.48550/arxiv.1505.07002 · Cited by 47
Semantic differential repair for input validation and sanitization
Alkhalaf, M., Aydin, A., & Bultan, T. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610401 · Cited by 44
Cochran, R. A., D’Antoni, L., Livshits, B., Molnár, D., & Veanes, M. (2014) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/2676726.2676973 · Cited by 37
Automatic repair for multi-threaded programs with Deadlock/Livelock using maximum satisfiability
Lin, Y., & Kulkarni, S. S. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610398 · Cited by 33
Gabel, M., & Su, Z. (2012) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2393596.2393598 · Cited by 31
Estler, H.-C., Furia, C. A., Nordio, M., Piccioni, M., & Meyer, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-06410-9_17 · Cited by 27
Barreto, A. S., Barros, M. D. O., & Werner, C. M. L. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083093 · Cited by 17
Evolution vs. Intelligent Design in Program Patching
Dévanbu, P. (2013) · eScholarship (California Digital Library) · Cited by 11
Research works printed but not linked: 6
- L. Chen and A. Avižienis . N-version programming: A faulttolerance approach to reliability of software operation . In IEEE International Symposium on Fault-Tolerant Computing (FTCS) , pages 3– 9 , 1978 . L. Chen and A. Avižienis. N-version programming: A faulttolerance approach to reliability of software operation. In IEEE International Symposium on Fault-Tolerant Computing (FTCS), pages 3–9, 1978.
- 10.5555/2486788.2486893 · DOI
- 10.5555/2337223.2337225 · DOI
- 10.5555/2818754.2818811 · DOI
- 10.5555/541177 · DOI
- Research Triangle Institute . The economic impacts of inadequate infrastructure for software testing. NIST Planning Report 02-3, May 2002 . Research Triangle Institute. The economic impacts of inadequate infrastructure for software testing. NIST Planning Report 02-3, May 2002.
Cited by
Sorted by citations
Mechtaev, S., Yi, J., & Roychoudhury, A. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884807 · Cited by 515
Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs
Xuan, J., Martínez, M., DeMarco, F., Clement, M., Marcote, S. R. L., Durieux, T., Berre, D. L., & Monperrus, M. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2560811 · Cited by 430
Le Goues, C., Pradel, M., & Roychoudhury, A. (2019) · Communications of the ACM · doi:10.1145/3318162 · Cited by 410
SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 448
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3340544 · Cited by 370
Metadata
- Authors
- Smith, E. K., Barr, E. T., Le Goues, C., & Brun, Y.
- Affiliations
- University of Massachusetts Amherst (US)
- University College London (GB)
- Carnegie Mellon University, USA
- University of Massachusetts, USA
- Venue
- Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2015-08-30
- Pages
- 532-543
- OpenAlex
- W2061575154
- Cited by
- 323
- References
- 68
- Citations by year
- 2026: 18 · 2025: 25 · 2024: 33 · 2023: 22 · 2022: 29 · 2021: 26 · 2020: 37 · 2019: 40 · 2018: 43 · 2017: 27 · 2016: 19 · 2015: 4
- FWCI
- 23.83
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- Funders
- National Science Foundation (CCF-1446683, CCF-1446966, CCF-1453508)
- Microsoft Research (Software Engineering Innovation Foundation Award)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Keywords
- automated program repair, overfitting, patch evaluation, test suite coverage, novice programmers, program correctness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
