The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationEvaluating and Improving Fault Localization
Pearson, S., Campos, J., Just, R., Fraser, G., Abreu, R., Ernst, M. D., Pang, D., & Keller, B. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.62 · Cited by 426
Self-citationAn Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3334955 · Cited by 431
“In experiments on 5 projects from Defects4J [69], AthenaTest generated 158K test cases, achieving similar test coverage as EvoSuite [16], a popular search-based test generation tool, and covering 43% of all focal methods.”
MethodNopol: Automatic Repair of Conditional Statement Bugs in Java Programs
Xuan, J., Martinez, M., DeMarco, F., Clement, M., Marcote, S. L., Durieux, T., Le Berre, D., & Monperrus, M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2560811 · Cited by 430
An analysis of patch plausibility and correctness for generate-and-validate patch generation systems
Qi, Z., Long, F., Achour, S., & Rinard, M. (2015) · Proceedings of the 2015 International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771791 · Cited by 420
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S. J., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 396
DeepMutation: Mutation Testing of Deep Learning Systems
Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., Xie, C., Li, L., Liu, Y., Zhao, J., et al. (2018) · 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2018.00021 · Cited by 366
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3340544 · Cited by 370
CoCoNuT: combining context-aware neural translation models using ensemble for program repair
Lutellier, T., Pham, H. V., Pang, L., Li, Y., Wei, M., & Tan, L. (2020) · Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3395363.3397369 · Cited by 344
Le, X. B. D., Lo, D., & Le Goues, C. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 332
Context-aware patch generation for better automated program repair
Wen, M., Chen, J., Wu, R., Hao, D., & Cheung, S.-C. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180233 · Cited by 328
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336
Automatic Software Repair: A Survey
Gazzola, L., Micucci, D., & Mariani, L. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2755013 · Cited by 327
Shaping program repair space with existing patches and similar code
Jiang, J., Xiong, Y., Zhang, H., Gao, Q., & Chen, X. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213871 · Cited by 329
CURE: Code-Aware Neural Machine Translation for Automatic Program Repair
Jiang, N., Lutellier, T., & Tan, L. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00107 · Cited by 290
TBar: revisiting template-based automated program repair
Liu, K., Koyuncu, A., Kim, D., & Bissyandé, T. F. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330577 · Cited by 321
Is the cure worse than the disease? overfitting in automated program repair
Smith, E. K., Barr, E. T., Le Goues, C., & Brun, Y. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786825 · Cited by 323
Precise Condition Synthesis for Program Repair
Xiong, Y., Wang, J., Yan, R., Zhang, J., Han, S., Huang, G., & Zhang, L. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.45 · Cited by 299
An Empirical Study of Fault Localization Families and Their Combinations
Zou, D., Liang, J., Xiong, Y., Ernst, M. D., & Zhang, L. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2892102 · Cited by 270
Self-citationSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264
Testing machine learning based systems: a systematic mapping
Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09881-0 · Cited by 251
Automatic repair of real bugs in java: a large-scale experiment on the defects4j dataset
Martinez, M., Durieux, T., Sommerard, R., Xuan, J., & Monperrus, M. (2017) · Empirical Software Engineering · doi:10.1007/s10664-016-9470-4 · Cited by 249
FLUCCS: using code and change metrics to improve fault localization
Sohn, J., & Yoo, S. (2017) · Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3092703.3092717 · Cited by 242
ASTOR: a program repair library for Java (demo)
Martinez, M., & Monperrus, M. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2948705 · Cited by 235
Automated Repair of Programs from Large Language Models
Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., & Tan, S. H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00128 · Cited by 256
Shamshiri, S., Just, R., Rojas, J. M., Fraser, G., McMinn, P., & Arcuri, A. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.86 · Cited by 230
Self-citationARJA: Automated Repair of Java Programs via Multi-Objective Genetic Programming
Yuan, Y., & Banzhaf, W. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2874648 · Cited by 224
S3: syntax- and semantic-guided repair synthesis via programming by examples
Le, X.-B. D., Chu, D.-H., Lo, D., Le Goues, C., & Visser, W. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106309 · Cited by 220
A learning-to-rank based fault localization approach using likely invariants
B. Le, T.-D., Lo, D., Le Goues, C., & Grunske, L. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931049 · Cited by 200
Impact of Code Language Models on Automated Program Repair
Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00125 · Cited by 202
FixMiner: Mining relevant fix patterns for automated program repair
Koyuncu, A., Liu, K., Bissyandé, T. F., Kim, D., Klein, J., Monperrus, M., & Le Traon, Y. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09780-z · Cited by 188
Elixir: Effective object-oriented program repair
Saha, R. K., Lyu, Y., Yoshida, H., & Prasad, M. R. (2017) · 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115675 · Cited by 189
Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction
Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00194 · Cited by 204
InferFix: End-to-End Program Repair with LLMs
Jin, M., Shahriar, S., Tufano, M., Shi, X., Lu, S., Sundaresan, N., & Svyatkovskiy, A. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613892 · Cited by 191
Big code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182
Test case selection and prioritization using machine learning: a systematic literature review
Pan, R., Bagherzadeh, M., Ghaleb, T. A., & Briand, L. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10066-6 · Cited by 178
(2025) · doi:10.1109/icse55347.2025.00157 · Cited by 157
“We apply RepairAgent to all bugs in the Defects4J dataset [24], which consists of 835 real-world bugs from 17 Java projects, including 395 bugs from 6 projects in Defects4Jv1.”
AVATAR: Fixing Semantic Bugs with Fix Patterns of Static Analysis Violations
Liu, K., Koyuncu, A., Kim, D., & Bissyande, T. F. (2019) · 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8667970 · Cited by 171
Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities
White, M., Tufano, M., Martinez, M., Monperrus, M., & Poshyvanyk, D. (2019) · 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8668043 · Cited by 163
Bugs.jar: a large-scale, diverse dataset of real-world Java bugs
Saha, R. K., Lyu, Y., Lam, W., Yoshida, H., & Prasad, M. R. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196473 · Cited by 168
Practical program repair via bytecode mutation
Ghanbari, A., Benton, S., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330559 · Cited by 162
iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests
Lam, W., Oei, R., Shi, A., Marinov, D., & Xie, T. (2019) · 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) · doi:10.1109/icst.2019.00038 · Cited by 162
Neural Transfer Learning for Repairing Security Vulnerabilities in C Code
Chen, Z., Kommrusch, S., & Monperrus, M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3147265 · Cited by 160
Automatic software repair: a survey
Gazzola, L., Micucci, D., & Mariani, L. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3182526 · Cited by 163
Boosting coverage-based fault localization via graph-based representation learning
Lou, Y., Zhu, Q., Dong, J., Li, X., Sun, Z., Hao, D., Zhang, L., & Zhang, L. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468580 · Cited by 163
Neural program repair with execution-based backpropagation
Ye, H., Martinez, M., & Monperrus, M. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510222 · Cited by 159
Leveraging syntax-related code for automated program repair
Qi, X., & Reiss, S. P. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115676 · Cited by 171
Liu, K., Koyuncu, A., Bissyandé, T. F., Kim, D., Klein, J., & Traon, Y. L. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00020 · Cited by 167
Transforming programs and tests in tandem for fault localization
Li, X., & Zhang, L. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133916 · Cited by 162
Widyasari, R., Sim, S. Q., Lok, C., Qi, H., Phan, J., Tay, Q., Tan, C., Wee, F., Tan, J. E., Yieh, Y., et al. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417943 · Cited by 150
Code smell detection by deep direct-learning and transfer-learning
Sharma, T., Efstathiou, V., Λουρίδας, Π., & Spinellis, D. D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110936 · Cited by 167
Large Language Models for Test-Free Fault Localization
Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623342 · Cited by 144
“2.0) [21] dataset highlights a null pointer exception that can be triggered in both of these methods, for the same reason.”
BackgroundMethodInfluentialBoosting spectrum-based fault localization using PageRank
Zhang, M., Li, X., Zhang, L., & Sarfraz Khurshid, D. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092731 · Cited by 148
Towards practical program repair with on-demand candidate generation
Hua, J., Zhang, M., Wang, K., & Sarfraz Khurshid, D. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180245 · Cited by 138
A Survey of Learning-based Automated Program Repair
Zhang, Q., Fang, C., Ma, Y., Sun, W., & Chen, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3631974 · Cited by 291
Applying CodeBERT for Automated Program Repair of Java Simple Bugs
Mashhadi, E., & Hemmati, H. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00063 · Cited by 156
Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
Wong, M.-F., Guo, S., Hang, C. N., Ho, S., & Tan, C. W. (2023) · Entropy · doi:10.3390/e25060888 · Cited by 223
“…for code refinement can be evaluated by CodeXGLUE [34] or HumanEval [35] as the abstracted codes or the classical APR benchmarks such as Defects4J [101] and QuixBugs [102] as real-world codes, but the understanding and generation of concrete variable and function names is still mandatory and…”
BackgroundTufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3240732 · Cited by 144
Mining stackoverflow for program repair
Liu, X., & Zhong, H. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330202 · Cited by 134
Identifying patch correctness in test-based program repair
Xiong, Y., Liu, X., Zeng, M., Zhang, L., & Huang, G. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180182 · Cited by 162
Durieux, T., Madeiral, F., Martínez, M., & Abreu, R. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338911 · Cited by 147
A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
Kang, S., An, G., & Yoo, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660771 · Cited by 128
“0 [15] (Java), chosen to allow comparison with traditional techniques, and BugsInPy [42] (Python), to demonstrate that AutoFL can quickly be adapted to other languages as well.”
An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems
Long, F., & Rinard, M. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884872 · Cited by 167
Liu, K., Wang, S., Koyuncu, A., Kim, K., Bissyandé, T. F., Kim, D., Wu, P., Klein, J., Mao, X., & Traon, Y. L. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380338 · Cited by 139
Mining Fix Patterns for FindBugs Violations
Liu, K., Kim, D., Bissyandé, T. F., Yoo, S., & Traon, Y. L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2884955 · Cited by 133
Where is the bug and how is it fixed? an experiment with practitioners
Böhme, M., Soremekun, E., Chattopadhyay, S., Ugherughe, E., & Zeller, A. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106255 · Cited by 136
Contract-based program repair without the contracts
Chen, L., Pei, Y., & Furia, C. A. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115674 · Cited by 122
Harnessing Evolution for Multi-Hunk Program Repair
Saha, S., Saha, R. K., & Prasad, M. R. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00020 · Cited by 117
TOGA: a neural method for test oracle generation
Dinella, E., Ryan, G., Mytkowicz, T., & Lahiri, S. K. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510141 · Cited by 221
Identifying test-suite-overfitted patches through test case generation
Qi, X., & Reiss, S. P. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092718 · Cited by 128
BEARS: An Extensible Java Bug Benchmark for Automatic Program Repair Studies
Madeiral, F., Urli, S., Maia, M., & Monperrus, M. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8667991 · Cited by 116
Karampatsis, R.-M., Sutton, C., Informatics, S. O., Institute for Machine Learning, & Intelligence, D. S. A. A. (2020) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.5281/zenodo.3653444 · Cited by 117
Wei, Y., Xia, C. S., & Zhang, L. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616271 · Cited by 115
“We evaluate Repilot on a subset of the widely studied Defects4J 1.2 and 2.0 datasets [32] and demonstrate state-of-the-art results in both the number of correct fixes and compilation rate — the percentage of the generated patches that can be successfully compiled.”
MethodInfluentialCODIT: Code Editing With Tree-Based Neural Models
Chakraborty, S., Ding, Y., Allamanis, M., & Ray, B. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3020502 · Cited by 149
How does regression test prioritization perform in real-world software evolution?
Lu, Y., Lou, Y., Cheng, S., Zhang, L., Hao, D., Zhou, Y., & Zhang, L. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884874 · Cited by 110
FAST approaches to scalable similarity-based test case prioritization
Miranda, B., Cruciani, E., Verdecchia, R., & Bertolino, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180210 · Cited by 110
An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Meng, X., Zhang, J., Liu, Y., Wang, W., Li, S., & Zhang, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00181 · Cited by 191
“0 [47].”
MethodDissection of a Bug Dataset:Anatomy of 395 Patches from Defects4J
Sobreira, V. (., Durieux, T., Madeiral, F., Monperrus, M., & Maia, M. A. (. A. ). (2018) · LillOA (Université de Lille (University Of Lille)) · Cited by 107
A Survey on Software Defect Prediction Using Deep Learning
Akimova, E. N., Bersenev, A. Y., Deikov, A. A., Kobylkin, K. S., Konygin, A. V., Mezentsev, I. P., & Misilov, V. E. (2021) · Mathematics · doi:10.3390/math9111180 · Cited by 150
Are mutation scores correlated with real fault detection?
Papadakis, M., Shin, D., Yoo, S., & Bae, D. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180183 · Cited by 130
Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies
Madeiral, F., Urli, S., Maia, M., & Monperrus, M. (2019) · HAL · Cited by 106
Breaking the Silence: the Threats of Using LLMs in Software Engineering
Sallou, J., Durieux, T., & Panichella, A. (2024) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3639476.3639764 · Cited by 210
“5 to generate JUnit test cases for two buggy programs in the Defect4j dataset [24], Chart-11 and Math-5 .”
InfluentialXia, X., Bao, L., Lo, D., & Li, S. (2016) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2016.67 · Cited by 104
Dissection of a bug dataset: Anatomy of 395 patches from Defects4J
Sobreira, V., Durieux, T., Madeiral, F., Monperrus, M., & Maia, M. D. A. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330203 · Cited by 140
Can automated program repair refine fault localization? a unified debugging approach
Lou, Y., Ghanbari, A., Li, X., Zhang, L., Zhang, H., Hao, D., & Zhang, L. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397351 · Cited by 110
On the reliability of coverage-based fuzzer benchmarking
Böhme, M., Szekeres, L., & Metzman, J. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510230 · Cited by 104
A similarity-based approach for test case prioritization using historical failure data
Noor, T. B., & Hemmati, H. (2015) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2015.7381799 · Cited by 103
How Effective Are Neural Networks for Fixing Security Vulnerabilities
Wu, Y., Jiang, N., Pham, H. V., Lutellier, T., Davis, J., Tan, L., Babkin, P., & Shah, S. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598135 · Cited by 218
“To extend this benchmark, we collect Java vulnerabilities following prior work [41]: i) The vulnerability should only be related to Java source code, ii) The xing commit should contain at least one test case that passes on +5 8G but fails on +1D6 , iii) The xing patch should only include changes that x the vulnerabili…”
BackgroundMethodInfluentialTian, H., Liu, K., Kaboré, A. K., Koyuncu, A., Li, L., Klein, J., & Bissyandé, T. F. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416532 · Cited by 113
Can OpenAI's codex fix bugs?: an evaluation on QuixBugs
Prenner, J. A., Babii, H., & Robbes, R. (2022) · Proceedings of the Third International Workshop on Automated Program Repair · doi:10.1145/3524459.3527351 · Cited by 99
Static automated program repair for heap properties
van Tonder, R., & Goues, C. L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180250 · Cited by 102
Ultra-Large Repair Search Space with Automatically Mined Templates: The Cardumen Mode of Astor
Martínez, M., & Monperrus, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_3 · Cited by 102
Durieux, T., & Monperrus, M. (2016) · International Workshop on Automation of Software Test (AST) · doi:10.1145/2896921.2896931 · Cited by 95
BugsJS: a Benchmark of JavaScript Bugs
Gyimesi, P., Vancsics, B., Stocco, A., Mazinanian, D., Beszédes, Á., Ferenć, R., & Mesbah, A. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00019 · Cited by 100
Code Generation Using Machine Learning: A Systematic Review
Dehaerne, E., Dey, B., Halder, S., Gendt, S. D., & Meert, W. (2022) · IEEE Access · doi:10.1109/access.2022.3196347 · Cited by 162
iFixR: bug report driven program repair
Koyuncu, A., Liu, K., Bissyandé, T. F., Kim, D., Monperrus, M., Klein, J., & Traon, Y. L. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338935 · Cited by 101
Historical Spectrum Based Fault Localization
Wen, M., Chen, J., Tian, Y., Wu, R., Hao, D., Han, S., & Cheung, S.-C. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2948158 · Cited by 106
Meng, X., Wang, X., Zhang, H., Sun, H., & Liu, X. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510147 · Cited by 104
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.