CoChat
Sign Up
Cited by 162Software Testing and Debugging TechniquesSoftware Engineering ResearchEvolutionary Algorithms and Applications

Identifying patch correctness in test-based program repair

Open access
Xiong, Yingfei, Liu, Xinyuan, Zeng, Muhan, Zhang, Lu, Huang, Gang·2018·Proceedings of the 40th International Conference on Software Engineering
DOI 10.1145/3180155.3180182
Open this paper in CoChat — free accountOpen PDF

Abstract

Test-based automatic program repair has attracted a lot of attention in recent years. However, the test suites in practice are often too weak to guarantee correctness and existing approaches often generate a large number of incorrect patches.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3180155.3180182

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Xiong et al. proposed to incorporate execution behavior similarity under generated test inputs to detect incorrect patches that behave significantly differently from their oracle patches.

    “Xiong et al. [58] proposed to incorporate execution behavior similarity under generated test inputs to detect incorrect patches that behave significantly differently from their oracle patches.”

    From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [58]

    The Conclusion (p. 11) says the approach identifies incorrect patches using similarity between program executions, and Section 3 (p. 3) explains how it uses new test inputs. But Section 3 defines the comparison as executions of the original buggy and patched versions, not executions of candidate patches against “oracle patches.” (LLM, conf=0.89)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 38

  1. [1][1] George K. Baah, Alexander G. Gray, and Mary Jean Harrold. 2006. On-line anomaly detection of deployed software: a statistical machine learning approach. In Third International Workshop on Software Quality Assurance, SOQUA 2006, Portland, Oregon, USA, November 6, 2006 , Johannes Mayer, Neelam Gupta, and Yves Ledru (Eds.). ACM, 70-77. https://doi.org/10.1145/1188895.1188911 - · DOI
  2. [2][2] Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Trans. Software Eng. 41, 5 (2015), 507-525. https://doi.org/10.1109/TSE.2014.2372785 - · DOI · Paper page
  3. [3][3] Loris DâĂŹAntoni, Roopsha Samanta, and Rishabh Singh. 2016. Qlose: Program repair with quantitative objectives. In International Conference on Computer Aided Verification . Springer, 383-401. -
  4. [4][4] William Dickinson, David Leon, and Andy Podgurski. 2001. Pursuing failure: the distribution of program failures in a profile space. In Proceedings of the 8th European Software Engineering Conference held jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering 2001, Vienna, Austria, September 10-14, 2001 , A. Min Tjoa and Volker Gruhn (Eds.). ACM, 246255. https://doi.org/10.1145/503209.503243 - · DOI
  5. [5][5] Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. 2001. Dynamically Discovering Likely Program Invariants to Support Program Evolution. IEEE Trans. Software Eng. 27, 2 (2001), 99-123. https://doi.org/10.1109/32.908957 - · DOI
  6. [6][6] Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. 2007. The Daikon system for dynamic detection of likely invariants. Sci. Comput. Program. 69, 1-3 (2007), 35-45. https://doi.org/10.1016/j.scico.2007.01.015 - · DOI
  7. [7][7] Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In ESEC/FSE . ACM, 416-419. -
  8. [8][8] Gordon Fraser and Andreas Zeller. 2012. Mutation-Driven Generation of Unit Tests and Oracles. IEEE Trans. Software Eng. 38, 2 (2012), 278-292. https: //doi.org/10.1109/TSE.2011.93 - · DOI
  9. [9][9] Qing Gao, Yingfei Xiong, Yaqing Mi, Lu Zhang, Weikun Yang, Zhaoping Zhou, Bing Xie, and Hong Mei. 2015. Safe Memory-Leak Fixing for C Programs. In Proceedings of the 37th International Conference on Software Engineering-Volume 1 . IEEE Press, 459-470. -
  10. [11][11] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. DeepFix: Fixing Common C Language Errors by Deep Learning. In AAAI . 1345-1351. -
  11. [12][12] Murali Haran, Alan F. Karr, Alessandro Orso, Adam A. Porter, and Ashish P. Sanil. 2005. Applying classification techniques to remotely-collected program execution data. In Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2005, Lisbon, Portugal, September 5-9, 2005 , Michel Wermelinger and Harald C. Gall (Eds.). ACM, 146-155. https: //doi.org/10.1145/1081706.1081732 - · DOI
  12. [13][13] Mary Jean Harrold, Gregg Rothermel, Rui Wu, and Liu Yi. 1998. An Empirical Investigation of Program Spectra. In Proceedings of the SIGPLAN/SIGSOFT Workshop on Program Analysis For Software Tools and Engineering, PASTE '98, Montreal, Canada, June 16, 1998 , Thomas Ball, Frank Tip, and A. Michael Berman (Eds.). ACM, 83-90. https://doi.org/10.1145/277631.277647 - · DOI
  13. [16][16] Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. 2013. Automatic patch generation learned from human-written patches. In ICSE '13 . 802-811. -
  14. [17][17] Xuan-Bach D Le, David Lo, and Claire Le Goues. 2016. History Driven Program Repair. In Software Analysis, Evolution, and Reengineering (SANER), 2016 IEEE 23rd International Conference on , Vol. 1. IEEE, 213-224. -
  15. [18][18] Xuan-Bach D. Le, Duc-Hiep Chu, David Lo, Claire Le Goues, and Willem Visser. 2017. S3: syntax- and semantic-guided repair synthesis via programming by examples. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017 . 593-604. https://doi.org/10.1145/3106237.3106309 - · DOI
  16. [19][19] Claire Le Goues, Michael Dewey-Vogt, Stephanie Forrest, and Westley Weimer. 2012. A Systematic Study of Automated Program Repair: Fixing 55 out of 105 Bugs for $8 Each. In ICSE . 3-13. -
  17. [20][20] C. Le Goues, ThanhVu Nguyen, S. Forrest, and W. Weimer. 2012. GenProg: A Generic Method for Automatic Software Repair. Software Engineering, IEEE Transactions on 38, 1 (Jan 2012), 54-72. -
  18. [22][22] Fan Long and Martin Rinard. 2015. Staged program repair with condition synthesis. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015 . 166-178. https://doi.org/10.1145/2786805.2786811 - · DOI
  19. [23][23] Fan Long and Martin Rinard. 2016. Automatic patch generation by learning correct code. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016 . 298-312. https://doi.org/10.1145/2837614.2837617 - · DOI
  20. [24][24] Fan Long and Martin C. Rinard. 2016. An analysis of the search spaces for generate and validate patch generation systems. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016 . 702-713. https://doi.org/10.1145/2884781.2884872 - · DOI · Paper page
  21. [26][26] Paul Dan Marinescu and Cristian Cadar. 2013. KATCH: High-coverage Testing of Software Patches. In ESEC/FSE . 235-245. -
  22. [28][28] Matias Martinez and Martin Monperrus. 2016. ASTOR: A Program Repair Library for Java. In Proceedings of ISSTA, Demonstration Track . 441-444. https://doi.org/ 10.1145/2931037.2948705 - · DOI
  23. [29][29] Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2015. DirectFix: Looking for Simple Program Repairs. In ICSE . 448-458. -
  24. [30][30] Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis. In ICSE . 691-701. -
  25. [31][31] Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra. 2013. SemFix: Program Repair via Semantic Analysis. In ICSE . 772-781. -
  26. [32][32] Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-directed Random Testing for Java. In OOPSLA . 815-816. -
  27. [33][33] Mauro Pezze and Cheng Zhang. 2015. Automated Test Oracles: A Survey. Advances in Computers 95 (2015), 1-48. -
  28. [35][35] Yuhua Qi, Xiaoguang Mao, Yan Lei, Ziying Dai, and Chengsong Wang. 2014. The Strength of Random Search on Automated Program Repair. In ICSE . 254-265. -
  29. [36][36] Zichao Qi, Fan Long, Sara Achour, and Martin C. Rinard. 2015. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In ISSTA . 24-36. -
  30. [37][37] Thomas Reps, Thomas Ball, Manuvir Das, and James Larus. 1997. The use of program profiling for software maintenance with applications to the year 2000 problem. In ESEC/FSE . Springer, 432-449. -
  31. [38][38] Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. 2013. Automated Feedback Generation for Introductory Programming Assignments. In PLDI . 1526. -
  32. [42][42] Westley Weimer. 2006. Patches as better bug reports. In Generative Programming and Component Engineering, 5th International Conference, GPCE 2006, Portland, Oregon, USA, October 22-26, 2006, Proceedings , Stan Jarzabek, Douglas C. Schmidt, and Todd L. Veldhuizen (Eds.). ACM, 181-190. https://doi.org/10.1145/1173706. 1173734 - · DOI
  33. [44][44] Westley Weimer, ThanhVu Nguyen, Claire Le Goues, and Stephanie Forrest. 2009. Automatically finding patches using genetic programming. In ICSE '09 . 364-374. -
  34. [45][45] Qi Xin and Steven Reiss. 2017. Identifying Test-Suite-Overfitted Patches through Test Case Generation. In ISSTA . -
  35. [46][46] Yingfei Xiong, Jie Wang, Runfa Yan, Jiachen Zhang, Shi Han, Gang Huang, and Lu Zhang. 2017. Precise Condition Synthesis for Program Repair. In ICSE . -
  36. [47][47] Yingfei Xiong, Hansheng Zhang, Arnaud Hubaux, Steven She, Jie Wang, and Krzysztof Czarnecki. 2015. Range fixes: Interactive error resolution for software configuration. Software Engineering, IEEE Transactions on 41, 6 (2015), 603-619. -
  37. [48][48] Jifeng Xuan, Matias Martinez, Favio Demarco, Maxime Clément, Sebastian Lamelas, Thomas Durieux, Daniel Le Berre, and Martin Monperrus. 2016. Nopoly: Automatic Repair of Conditional Statement Bugs in Java Programs. IEEE Transactions on Software Engineering (2016). -
  38. [50][50] Zhongxing Yu, Matias Martinez, Benjamin Danglot, Thomas Durieux, and Martin Monperrus. 2017. Test Case Generation for Program Repair: A Study of Feasibility and Effectiveness. CoRR abs/1703.00198 (2017). · arXiv

Cited by

Sorted by citations

Authors
Xiong, Y., Liu, X., Zeng, M., Zhang, L., & Huang, G.
Affiliations
Peking University (CN)
ORCID
Venue
Proceedings of the 40th International Conference on Software Engineering
Publisher
ACM
Published
2018-05-27
Pages
789-799
ISSN
0270-5257 (ISSN-L)
DOI
10.1145/3180155.3180182
OpenAlex
W2952920225
Cited by
162
References
2
Citations by year
2026: 10 · 2025: 12 · 2024: 24 · 2023: 11 · 2022: 23 · 2021: 20 · 2020: 24 · 2019: 27 · 2018: 9 · 2017: 2
FWCI
13.56
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
automated program repair

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review