Better test cases for better automated program repair
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Automated generate-and-validate program repair techniques (G&V techniques) suffer from generating many overfitted patches due to in-capabilities of test cases. Such overfitted patches are incor- rect patches, which only make all given test cases pass, but fail to fix the bugs. In this work, we propose an overfitted patch detec- tion framework named Opad (Overfitted PAtch Detection). Opad helps improve G&V techniques by enhancing existing test cases to filter out overfitted patches. To enhance test cases, Opad uses fuzz testing to generate new test cases, and employs two test or- acles (crash and memory-safety) to enhance validity checking of automatically-generated patches. Opad also uses a novel metric (named O-measure) for deciding whether automatically-generated patches overfit. Evaluated on 45 bugs from 7 large systems (the same benchmark used by GenProg and SPR), Opad filters out 75.2% (321/427) over- fitted patches generated by GenProg/AE, Kali, and SPR. In addition, Opad guides SPR to generate correct patches for one more bug (the original SPR generates correct patches for 11 bugs). Our analysis also shows that up to 40% of such automatically-generated test cases may further improve G&V techniques if empowered with better test oracles (in addition to crash and memory-safety oracles employed by Opad).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
JCrasher: an automatic robustness tester for Java
Csallner, C., & Smaragdakis, Y. (2004) · Software: Practice and Experience · doi:10.1002/spe.602 · Cited by 426 · cited 1×
“Random test generation techniques [7, 27] and fuzz testing tools scale to large systems, but lack of direction.”
BackgroundAn analysis of patch plausibility and correctness for generate-and-validate patch generation systems
Qi, Z., Long, F., Achour, S., & Rinard, M. (2015) · Proceedings of the 2015 International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771791 · Cited by 420
Is the cure worse than the disease? overfitting in automated program repair
Smith, E. K., Barr, E. T., Le Goues, C., & Brun, Y. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786825 · Cited by 323
Identifying patch correctness in test-based program repair
Xiong, Y., Liu, X., Zeng, M., Zhang, L., & Huang, G. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180182 · Cited by 162
Cited by
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3340544 · Cited by 370
CoCoNuT: combining context-aware neural translation models using ensemble for program repair
Lutellier, T., Pham, H. V., Pang, L., Li, Y., Wei, M., & Tan, L. (2020) · Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3395363.3397369 · Cited by 344
Self-citationShaping program repair space with existing patches and similar code
Jiang, J., Xiong, Y., Zhang, H., Gao, Q., & Chen, X. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213871 · Cited by 329
“In addition, several approaches [63, 66, 70] have been proposed to classify patches to increase precision, and SimFix can potentially be combined with these approaches to gain a better precision.”
Context-aware patch generation for better automated program repair
Wen, M., Chen, J., Wu, R., Hao, D., & Cheung, S.-C. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180233 · Cited by 328
CURE: Code-Aware Neural Machine Translation for Automatic Program Repair
Jiang, N., Lutellier, T., & Tan, L. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00107 · Cited by 290
Self-citation
Metadata
- Authors
- Yang, J., Zhikhartsev, A., Liu, Y., & Tan, L.
- Venue
- Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2017-08-21
- Pages
- 831-841
- OpenAlex
- W2741328617
- Cited by
- 123
- References
- 4
- FWCI
- 10.64
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Better test cases for better automated program repair. https://app.cochat.ai/papers/10.1145/3106237.3106274
- The paper
- Yang, J., Zhikhartsev, A., Liu, Y., & Tan, L. (2017). Better test cases for better automated program repair. Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. https://doi.org/10.1145/3106237.3106274
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.