CoChat
Sign Up
Cited by 328Software Testing and Debugging TechniquesSoftware Engineering ResearchEvolutionary Algorithms and Applications

The DOI matches its Crossref record; 66 of the 68 references it cites were found in a registry or an index.

Is the cure worse than the disease? overfitting in automated program repair

Smith, Edward K., Barr, Earl T., Le Goues, Claire, Brun, Yuriy·2015·Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering
DOI 10.1145/2786805.2786825
Open this paper in CoChat — free account

Abstract

Automated program repair has shown promise for reducing the significant manual effort debugging requires. This paper addresses a deficit of earlier evaluations of automated repair techniques caused by repairing programs and evaluating generated patches' correctness using the same set of tests. Since tests are an imperfect metric of program correctness, evaluations of this type do not discriminate between correct patches and patches that overfit the available tests and break untested but desired functionality. This paper evaluates two well-studied repair tools, GenProg and TrpAutoRepair, on a publicly available benchmark of bugs, each with a human-written patch. By evaluating patches using tests independent from those used during repair, we find that the tools are unlikely to improve the proportion of independent tests passed, and that the quality of the patches is proportional to the coverage of the test suite used during repair. For programs that pass most tests, the tools are as likely to break tests as to fix them. However, novice developers also overfit, and automated repair performs no worse than these developers. In addition to overfitting, we measure the effects of test suite coverage, test suite provenance, and starting program quality, as well as the difference in quality between novice-developer-written and tool-generated patches when quality is assessed with a test suite independent from the one used for patch generation.

Overview

Integrity signals

Registry

Crossref · 10.1145/2786805.2786825

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 68 references

Research works printed but not linked: 6

  1. L. Chen and A. Avižienis . N-version programming: A faulttolerance approach to reliability of software operation . In IEEE International Symposium on Fault-Tolerant Computing (FTCS) , pages 3– 9 , 1978 . L. Chen and A. Avižienis. N-version programming: A faulttolerance approach to reliability of software operation. In IEEE International Symposium on Fault-Tolerant Computing (FTCS), pages 3–9, 1978.
  2. 10.5555/2486788.2486893 · DOI
  3. 10.5555/2337223.2337225 · DOI
  4. 10.5555/2818754.2818811 · DOI
  5. 10.5555/541177 · DOI
  6. Research Triangle Institute . The economic impacts of inadequate infrastructure for software testing. NIST Planning Report 02-3, May 2002 . Research Triangle Institute. The economic impacts of inadequate infrastructure for software testing. NIST Planning Report 02-3, May 2002.

Cited by

Sorted by citations

See all 328

Related

Metadata

Authors
Smith, E. K., Barr, E. T., Le Goues, C., & Brun, Y.
Affiliations
  • University of Massachusetts Amherst (US)
  • University College London (GB)
  • Carnegie Mellon University, USA
  • University of Massachusetts, USA
ORCID
Venue
Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering
Publisher
ACM
Published
2015-08-30
Pages
532-543
DOI
10.1145/2786805.2786825
OpenAlex
W2061575154
Cited by
323
References
68
Citations by year
2026: 18 · 2025: 25 · 2024: 33 · 2023: 22 · 2022: 29 · 2021: 26 · 2020: 37 · 2019: 40 · 2018: 43 · 2017: 27 · 2016: 19 · 2015: 4
FWCI
23.83
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
Funders
  • National Science Foundation (CCF-1446683, CCF-1446966, CCF-1453508)
  • Microsoft Research (Software Engineering Innovation Foundation Award)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
automated program repair, overfitting, patch evaluation, test suite coverage, novice programmers, program correctness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review