Identifying test-suite-overfitted patches through test case generation
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
A typical automatic program repair technique that uses a test suite as the correct criterion can produce a patched program that is test-suite-overfitted, or overfitting, which passes the test suite but does not actually repair the bug. In this paper, we propose DiffTGen which identifies a patched program to be overfitting by first generating new test inputs that uncover semantic differences between the original faulty program and the patched program, then testing the patched program based on the semantic differences, and finally generating test cases. Such a test case could be added to the original test suite to make it stronger and could prevent the repair technique from generating a similar overfitting patch again. We evaluated DiffTGen on 89 patches generated by four automatic repair techniques for Java with 79 of them being likely to be overfitting and incorrect. DiffTGen identifies in total 39 (49.4%) overfitting patches and yields the corresponding test cases. We further show that an automatic repair technique, if configured with DiffTGen, could avoid yielding overfitting patches and potentially produce correct ones.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
An analysis of patch plausibility and correctness for generate-and-validate patch generation systems
Qi, Z., Long, F., Achour, S., & Rinard, M. (2015) · Proceedings of the 2015 International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771791 · Cited by 420
Is the cure worse than the disease? overfitting in automated program repair
Smith, E. K., Barr, E. T., Le Goues, C., & Brun, Y. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786825 · Cited by 323 · cited 1×
“Studies [30,33] have shown that early repair techniques suffer from severe overfitting problems.”
BackgroundAutomatic Repair of Real Bugs: An Experience Report on the Defects4J Dataset
Martinez, M., Durieux, T., Xuan, J., Sommerard, R., & Monperrus, M. (2015) · arXiv · doi:10.48550/arxiv.1505.07002 · Cited by 47
Cited by
Shaping program repair space with existing patches and similar code
Jiang, J., Xiong, Y., Zhang, H., Gao, Q., & Chen, X. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213871 · Cited by 329
“In addition, several approaches [63, 66, 70] have been proposed to classify patches to increase precision, and SimFix can potentially be combined with these approaches to gain a better precision.”
Context-aware patch generation for better automated program repair
Wen, M., Chen, J., Wu, R., Hao, D., & Cheung, S.-C. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180233 · Cited by 328
Identifying patch correctness in test-based program repair
Xiong, Y., Liu, X., Zeng, M., Zhang, L., & Huang, G. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180182 · Cited by 162
Baldur: Whole-Proof Generation and Repair with Large Language Models
First, E., Rabe, M. N., Ringer, T., & Brun, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616243 · Cited by 94
Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study
Wang, Y., Pradel, M., & Liu, Z. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764576 · Cited by 98
“Another option is to generate more regression tests based on the fixed repository version to strengthen the validation [57, 62].”
Metadata
- Authors
- Xin, Q., & Reiss, S. P.
- Venue
- Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2017-07-10
- Pages
- 226-236
- OpenAlex
- W2735571786
- Cited by
- 115
- References
- 4
- FWCI
- 9.82
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Identifying test-suite-overfitted patches through test case generation. https://app.cochat.ai/papers/10.1145/3092703.3092718
- The paper
- Xin, Q., & Reiss, S. P. (2017). Identifying test-suite-overfitted patches through test case generation. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. https://doi.org/10.1145/3092703.3092718
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.