CoChat
Sign Up
PreprintCited by 47Software Testing and Debugging TechniquesSoftware Engineering ResearchEvolutionary Algorithms and Applications

Automatic Repair of Real Bugs: An Experience Report on the Defects4J Dataset

Open access
Martinez, Matias, Durieux, Thomas, Xuan, Jifeng, Sommerard, Romain, Monperrus, Martin·2015·arXiv
DOI 10.48550/arxiv.1505.07002arXiv 1505.07002
Open this paper in CoChat — free accountOpen PDF

Abstract

Defects4J is a large, peer-reviewed, structured dataset of real-world Java bugs. Each bug in Defects4J is provided with a test suite and at least one failing test case that triggers the bug. In this paper, we report on an experiment to explore the effectiveness of automatic repair on Defects4J. The result of our experiment shows that 47 bugs of the Defects4J dataset can be automatically repaired by state-of- the-art repair. This sets a baseline for future research on automatic repair for Java. We have manually analyzed 84 different patches to assess their real correctness. In total, 9 real Java bugs can be correctly fixed with test-suite based repair. This analysis shows that test-suite based repair suffers from under-specified bugs, for which trivial and incorrect patches still pass the test suite. With respect to practical applicability, it takes in average 14.8 minutes to find a patch. The experiment was done on a scientific grid, totaling 17.6 days of computation time. All their systems and experimental results are publicly available on Github in order to facilitate future research on automatic repair.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1505.07002

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. [3][3] R. Abreu, P. Zoeteweij, and A. J. Van Gemund. On the accuracy of spectrum-based fault localization. In Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION, 2007. TAICPART-MUTATION 2007 , pages 89-98. IEEE, 2007. -
  2. [4][4] A. Arcuri and X. Yao. A novel co-evolutionary approach to automatic software bug fixing. In Proceedings of the IEEE Congress on Evolutionary Computation , pages 162-168, 2008. -
  3. [5][5] R. Bolze, F. Cappello, E. Caron, M. Dayd´ e, F. Desprez, E. Jeannot, Y. J´ egou, S. Lanteri, J. Leduc, N. Melab, et al. Grid'5000: a large scale and highly reconfigurable experimental grid testbed. volume 20, pages 481-494. SAGE Publications, 2006. -
  4. [6][6] C. Cifuentes, C. Hoermann, N. Keynes, L. Li, S. Long, E. Mealy, M. Mounteney, and B. Scholz. Begbunch: Benchmarking for c bug detection tools. In Proceedings of ISSTA , pages 16-20, New York, USA, 2009. ACM. -
  5. [7][7] V. Dallmeier and T. Zimmermann. Extraction of Bug Localization Benchmarks from History. In Proceedings of the Twenty-second IEEE/ACM International Conference on Automated Software Engineering , pages 433-436, 2007. -
  6. [8][8] V. Debroy and W. E. Wong. Using mutation to automatically suggest fixes for faulty programs. In Third International Conference on Software Testing, Verification and Validation, ICST 2010, Paris, France, April 7-9, 2010 , pages 65-74, 2010. -
  7. [9][9] F. Demarco, J. Xuan, D. L. Berre, and M. Monperrus. Automatic repair of buggy if conditions and missing preconditions with smt. In Proceedings of the 6th International Workshop on Constraints in Software Testing, Verification, and Analysis , pages 30-39. ACM, 2014. -
  8. [10][10] H. Do, S. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering , 10(4):405-435, 2005. -
  9. [12][12] Z. P. Fry, B. Landau, and W. Weimer. A human study of patch maintainability. In Proceedings of the 2012 International Symposium on Software Testing and Analysis , ISSTA 2012, pages 177-187, New York, NY, USA, 2012. ACM. -
  10. [13][13] Z. P. Fry, B. Landau, and W. Weimer. A human study of patch maintainability. In Proceedings of the International Symposium on Software Testing and Analysis , pages 177-187, 2012. -
  11. [14][14] D. Gopinath, S. Khurshid, D. Saha, and S. Chandra. Data-guided repair of selection statements. In Proceedings of the 36th International Conference on Software Engineering , pages 243-253. ACM, 2014. -
  12. [15][15] Z. Gu, E. Barr, D. Hamilton, and Z. Su. Has the bug really been fixed? In Software Engineering, 2010 ACM/IEEE 32nd International Conference on , volume 1, pages 55-64, May 2010. -
  13. [16][16] S. Jha, S. Gulwani, S. A. Seshia, and A. Tiwari. Oracle-guided component-based program synthesis. In Proceedings of the International Conference on Software Engineering , volume 1, pages 215-224. IEEE, 2010. -
  14. [17][17] G. Jin, L. Song, W. Zhang, S. Lu, and B. Liblit. Automated atomicity-violation fixing. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation , pages 389-400, 2011. -
  15. [18][18] J. A. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proceedings of the 24th international conference on Software engineering , pages 467-477. ACM, 2002. -
  16. [20][20] R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing. In 22nd International Symposium on the Foundations of Software Engineering (FSE 2014) , 2014. -
  17. [21][21] D. Kim, J. Nam, J. Song, and S. Kim. Automatic patch generation learned from human-written patches. In Proceedings of the 35th International Conference on Software Engineering , pages 802-811, 2013. -
  18. [22][22] C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In Software Engineering (ICSE), 2012 34th International Conference on , pages 3-13. IEEE, 2012. -
  19. [23][23] C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In Proceedings of the 34th International Conference on Software Engineering , pages 3-13, 2012. -
  20. [25][25] C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer. Genprog: A generic method for automatic software repair. Software Engineering, IEEE Transactions on , 38(1):54-72, 2012. -
  21. [26][26] F. Long and M. C. Rinard. Staged program repair with condition synthesis. In Proceedings of ESE/FSE , 2015. -
  22. [27][27] F. Long, S. Sidiroglou-Douskos, and M. C. Rinard. Automatic runtime error repair and containment via recovery shepherding. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '14, Edinburgh, United Kingdom - June 09 - 11, 2014 , page 26, 2014. -
  23. [28][28] S. Lu, Z. Li, F. Qin, L. Tan, P. Zhou, and Y. Zhou. Bugbench: Benchmarks for evaluating bug detection tools. In Workshop on the Evaluation of Software Defect Detection Tools , 2005. -
  24. [30][30] M. Martinez and M. Monperrus. Mining software repair models for reasoning on the search space of automated program fixing. Empirical Software Engineering , 20(1):176-205, 2015. -
  25. [31][31] M. Martinez, W. Weimer, and M. Monperrus. Do the fix ingredients already exist? an empirical inquiry into the redundancy assumptions of program repair approaches. In Proceedings of the 36th International Conference on Software Engineering , pages 492-495, 2014. -
  26. [32][32] S. Mechtaev, J. Yi, and A. Roychoudhury. Directfix: Looking for simple program repairs. In Proceedings of the 37th International Conference on Software Engineering . IEEE, 2015. -
  27. [33][33] M. Monperrus. A critical review of automatic patch generation learned from human-written patches: essay on the problem statement and the evaluation of automatic software repair. In Proceedings of the 36th International Conference on Software Engineering , pages 234-242. ACM, 2014. -
  28. [34][34] H. D. T. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra. Semfix: Program repair via semantic analysis. In Proceedings of the 2013 International Conference on Software Engineering , pages 772-781. IEEE Press, 2013. -
  29. [35][35] T. Noor and H. Hemmati. Test case analytics: Mining test case traces to improve risk-driven testing. In Proceedings of the IEEE 1st International Workshop on Software Analytics , pages 13-16. IEEE, 2015. -
  30. [36][36] Y. Pei, C. A. Furia, M. Nordio, Y. Wei, B. Meyer, and A. Zeller. Automated fixing of programs with contracts. IEEE Trans. Software Eng. , 40(5):427-449, 2014. -
  31. [37][37] Y. Qi, X. Mao, and Y. Lei. Efficient automated program repair through fault-recorded testing prioritization. In 2013 IEEE International Conference on Software Maintenance , pages 180-189, 2013. -
  32. [38][38] Y. Qi, X. Mao, Y. Lei, Z. Dai, and C. Wang. The strength of random search on automated program repair. In Proceedings of the 36th International 11. Conference on Software Engineering , pages 254-265. ACM, 2014. -
  33. [39][39] Z. Qi, F. Long, S. Achour, and M. Rinard. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In Proceedings of ISSTA . ACM, 2015. -
  34. [40][40] H. Samimi, M. Sch¨ afer, S. Artzi, T. D. Millstein, F. Tip, and L. J. Hendren. Automated repair of HTML generation errors in PHP applications using string constraint solving. In Proceedings of the 34th International Conference on Software Engineering , pages 277-287, 2012. -
  35. [41][41] S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri. Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges. In Proceedings of the International Conference on Automated Software Engineering , 2015. -
  36. [43][43] S. H. Tan and A. Roychoudhury. relifix: Automated repair of software regressions. In Proceedings of ICSE , 2015. -
  37. [44][44] Y. Tao, J. Kim, S. Kim, and C. Xu. Automatically generated patches as debugging aids: a human study. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , pages 64-74, 2014. -
  38. [45][45] J. Xuan, M. Matias, F. DeMarco, L. Sebastian, D. Thomas, D. Le Berre, and M. Monperrus. Nopol: Automatic repair of conditional statement bugs in large-scale object-oriented programs. Under 2nd review at IEEE Transactions on Software Engineering. -
  39. [46][46] J. Xuan and M. Monperrus. Test case purification for improving fault localization. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) . ACM, 2014. -
  40. [47][47] H. Zhong and Z. Su. An empirical study on real bug fixes. In Proceedings of the 37th International Conference on Software Engineering (ICSE) . IEEE, 2015.

Cited by

Sorted by citations

Authors
Martinez, M., Durieux, T., Xuan, J., Sommerard, R., & Monperrus, M.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Université de Lille (FR)
Venue
arXiv
Publisher
arXiv
Published
2015
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1505.07002
arXiv
1505.07002
OpenAlex
W1693912443
Cited by
47
References
3
Citations by year
2025: 1 · 2024: 2 · 2023: 3 · 2022: 2 · 2021: 4 · 2020: 2 · 2019: 5 · 2018: 6 · 2017: 7 · 2016: 9 · 2015: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (acceptedVersion)
  • HAL (submittedVersion)
  • LillOA (Université de Lille (University Of Lille)) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • U.S. Department of Defense
  • Advanced Research Projects Agency
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
Defects4J, automated program repair

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review